This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

whois -h whois.radb.net -- '-i origin AS50896'

Device descriptor Remembering that values which are > 1 byte in length are always stored little-endian, we can construct our device descriptor as:

Our device descriptor

Field Name	Value	Bytes
bLength	18	0x12
bDescriptorType	DEVICE	0x01
bcdUSB	USB 2.0	0x00 0x02
bDeviceClass	0	0x00
bDeviceSubClass	0	0x00
bDeviceProtocol	0	0x00
bMaxPacketSize	64	0x40
idVendor	TEST	0xff 0xff
idProduct	TEST	0xff 0xff
bcdDevice	0.0.1	0x01 0x00
iManufacturer	1	0x01
iProduct	2	0x02
iSerialNumber	3	0x03
bNumConfigurations	1	0x01

We're using the vendor ID and product id 0xffff because at this point we don't have any useful values for this (it costs $5,000 to register a vendor ID). This gives us a final byte array of:

0x12 0x01 0x00 0x02 0x00 0x00 0x00 0x40 (Early descriptor) 0xff 0xff 0xff 0xff 0x01 0x00 0x01 0x02 0x03 0x01 (and the rest)

We're reserving string ids 1, 2, and 3, for the manufacturer string, product name string, and serial number string respectively. I'm deliberately including them all so that we can see it all come out later in lsusb. If you feed the above hex sequence into a USB descriptor decoder then you can check my working.

Endpoint Descriptor We want a single configuration, which covers our one interface, with one endpoint in it. Let's start with the endpoint...

Our bulk IN endpoint

Field Name	Value	Bytes
bLength	7	0x07
bDescriptorType	ENDPOINT	0x05
bEndpointAddress	EP2IN	0x82
bmAttributes	BULK	0x02
wMaxPacketSize	64	0x40 0x00
bInterval	IGNORED	0x00

We're giving a single bulk IN endpoint, since that's the simplest thing to describe at this time. This endpoint will never be ready and so nothing will ever be read into the host. All that gives us:

0x07 0x05 0x82 0x02 0x40 0x00 0x00

Interface Descriptor The interface descriptor prefaces the endpoint set, and thanks to our simple single endpoint, and no plans for alternate interfaces, we can construct the interface simply as:

Our single simple interface

Field Name	Value	Bytes
bLength	9	0x09
bDescriptorType	INTERFACE	0x04
bInterfaceNumber	1	0x01
bAlternateSetting	1	0x01
bNumEndpoints	1	0x01
bInterfaceClass	0	0x00
bInterfaceSubClass	0	0x00
bInterfaceProtocol	0	0x00
iInterface	5	0x05

All that gives us:

0x09 0x04 0x01 0x01 0x01 0x00 0x00 0x00 0x05

Configuration descriptor Finally we can put it all together and get the configuration descriptor...

Our sole configuration, encapsulating the interface and endpoint above

Field Name	Value	Bytes
bLength	9	0x09
bDescriptorType	CONFIG	0x02
wTotalLength	9+9+7	0x19 0x00
bNumInterfaces	1	0x01
bConfigurationValue	1	0x01
iConfiguration	4	0x04
bmAttributes	Bus powered, no wake	0x80
bMaxPower	500mA	0xfa

The wTotalLength field is interesting. It contains the configuration length, the interface length, and the endpoint length, hence 9 plus 9 plus 7 is 25. This gives:

0x09 0x02 0x19 0x00 0x01 0x01 0x04 0x80 0xfa

String descriptors We allowed ourselves a total of five strings, they were iManufacturer, iProduct, iSerial (from the device descriptor), iConfiguration (from the configuration descriptor), and iInterface (from the interface descriptor) respectively. Our string descriptors will therefore be:

String descriptor zero, en_GB only

Field Name	Value	Bytes
bLength	4	0x04
bDescriptorType	STRING	0x03
wLangID[0]	en_GB	0x09 0x08

0x04 0x03 0x09 0x08

...and...

String descriptor one, iManufacturer

Field Name	Value	Bytes
bLength	38	0x26
bDescriptorType	STRING	0x03
bString	"Rusty Manufacturer"	...

0x26 0x03 0x52 0x00 0x75 0x00 0x73 0x00 0x74 0x00 0x79 0x00 0x20 0x00 0x4d 0x00 0x61 0x00 0x6e 0x00 0x75 0x00 0x66 0x00 0x61 0x00 0x63 0x00 0x74 0x00 0x75 0x00 0x72 0x00 0x65 0x00 0x72 0x00

(You get the idea, there's no point me breaking down the rest of the string descriptors here, suffice it to say that the other strings are appropriate for the values they represent - namely product, serial, configuration, and interface.)

Putting it all together Given all the above, we have a device descriptor which is standalone, then a configuration descriptor which encompasses the interface and endpoint descriptors too. Finally we have a string descriptor table with six entries, the first is the language sets available, and the rest are our strings. In total we have:

    // Device descriptor
    const DEV_DESC: [u8; 18] =  
        0x12, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x40,
        0xff, 0xff, 0xff, 0xff, 0x01, 0x00, 0x01, 0x02,
        0x03, 0x01
     ;
    // Configuration descriptor
    const CONF_DESC: [u8; 25] =  
        0x09, 0x02, 0x19, 0x00, 0x01, 0x01, 0x04, 0x80, 0xfa,
        0x09, 0x04, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00, 0x05,
        0x07, 0x05, 0x82, 0x02, 0x40, 0x00, 0x00
     ;
    // String Descriptor zero
    const STR_DESC_0: [u8; 4] =  0x04, 0x03, 0x09, 0x08 ;
    // String Descriptor 1, "Rusty Manufacturer"
    const STR_DESC_1: [u8; 38] =  
        0x26, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x4d, 0x00,
        0x61, 0x00, 0x6e, 0x00, 0x75, 0x00, 0x66, 0x00,
        0x61, 0x00, 0x63, 0x00, 0x74, 0x00, 0x75, 0x00,
        0x72, 0x00, 0x65, 0x00, 0x72, 0x00
     ;
    // String Descriptor 2, "Rusty Product"
    const STR_DESC_2: [u8; 28] =  
        0x1c, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x50, 0x00,
        0x72, 0x00, 0x6f, 0x00, 0x64, 0x00, 0x75, 0x00,
        0x63, 0x00, 0x74, 0x00
     ;
    // String Descriptor 3, "123ABC"
    const STR_DESC_3: [u8; 14] =  
        0x0e, 0x03, 0x31, 0x00, 0x32, 0x00, 0x33, 0x00,
        0x41, 0x00, 0x42, 0x00, 0x43, 0x00
     ;
    // String Descriptor 4, "Rusty Configuration"
    const STR_DESC_4: [u8; 40] =  
        0x28, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x43, 0x00,
        0x6f, 0x00, 0x6e, 0x00, 0x66, 0x00, 0x69, 0x00,
        0x67, 0x00, 0x75, 0x00, 0x72, 0x00, 0x61, 0x00,
        0x74, 0x00, 0x69, 0x00, 0x6f, 0x00, 0x6e, 0x00
     ;
    // String Descriptor 5, "Rusty Interface"
    const STR_DESC_5: [u8; 32] =  
        0x20, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x49, 0x00,
        0x6e, 0x00, 0x74, 0x00, 0x65, 0x00, 0x72, 0x00,
        0x66, 0x00, 0x61, 0x00, 0x63, 0x00, 0x65, 0x00
     ;

With the above, we're a step closer to our first prototype which will hopefully be enumerable. Next time we'll look at beginning our prototype low level USB device stack mock-up.

• The excellent Google Summer of Code work by Binxiang Ni had only encountered direct use of sparse matrices as produced by the Matrix. However, while we waited for 0.7.960.1.0 to make it onto CRAN, the quanteda package switched to derived classes---which we now account for via the is() method of our S4 class. Thanks to Kevin Ushey for reminding me we had is().
• We somehow missed to account for the R 3.4.* and Rcpp 0.12. 11,12 changes for package registration (with .registration=TRUE), so ensured we only have one fastLm symbol.
• The build did not take not too well to systems without OpenMP, so we now explicitly unset supported via an Armadillo configuration variable. In general, client packages probably want to enable C++11 support when using OpenMP (explicitly) but we prefer to not upset too many (old) users. However, our configure check now also wants g++ 4.7.2 or later just like Armadillo.

Changes in RcppArmadillo version 0.7.960.1.1 (2017-08-20)

• Added improved check for inherited S4 matrix classes (#162 fixing #161)
• Changed fastLm C++ function to fastLm_impl to not clash with R method (#164 fixing #163)
• Added OpenMP check for configure (#166 fixing #165)

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Lookup trie implementation Looking up a prefix in a routing table comes down to find the most specific entry matching the requested destination. A common structure for this task is the trie, a tree structure where each node has its parent as prefix. With IPv4, Linux uses a level-compressed trie (or LPC-trie), providing good performances with low memory usage. For IPv6, Linux uses a more classic radix tree (or Patricia trie). There are three reasons for not sharing:

• The IPv6 implementation (introduced in Linux 2.1.8, 1996) predates the IPv4 implementation based on LPC-tries (in Linux 2.6.13, commit 19baf839ff4a).
• The feature set is different. Notably, IPv6 supports source-specific routing¹ (since Linux 2.1.120, 1998).
• The IPv4 address space is denser than the IPv6 address space. Level-compression is therefore quite efficient with IPv4. This may not be the case with IPv6.

The trie in the below illustration encodes 6 prefixes: For more in-depth explanation on the different ways to encode a routing table into a trie and a better understanding of radix trees, see the explanations for IPv4. The following figure shows the in-memory representation of the previous radix tree. Each node corresponds to a struct fib6_node. When a node has the RTN_RTINFO flag set, it embeds a pointer to a struct rt6_info containing information about the next-hop. The fib6_lookup_1() function walks the radix tree in two steps:

1. walking down the tree to locate the potential candidate, and
2. checking the candidate and, if needed, backtracking until a match.

Here is a slightly simplified version without source-specific routing:

static struct fib6_node *fib6_lookup_1(struct fib6_node *root,
                                       struct in6_addr  *addr)
 
    struct fib6_node *fn;
    __be32 dir;
    /* Step 1: locate potential candidate */
    fn = root;
    for (;;)  
        struct fib6_node *next;
        dir = addr_bit_set(addr, fn->fn_bit);
        next = dir ? fn->right : fn->left;
        if (next)  
            fn = next;
            continue;
         
        break;
     
    /* Step 2: check prefix and backtrack if needed */
    while (fn)  
        if (fn->fn_flags & RTN_RTINFO)  
            struct rt6key *key;
            key = fn->leaf->rt6i_dst;
            if (ipv6_prefix_equal(&key->addr, addr, key->plen))  
                if (fn->fn_flags & RTN_RTINFO)
                    return fn;
             
         
        if (fn->fn_flags & RTN_ROOT)
            break;
        fn = fn->parent;
     
    return NULL;
 

Caching While IPv4 lost its route cache in Linux 3.6 (commit 5e9965c15ba8), IPv6 still has a caching mechanism. However cache entries are directly put in the radix tree instead of a distinct structure. Since Linux 2.1.30 (1997) and until Linux 4.2 (commit 45e4fd26683c), almost any successful route lookup inserts a cache entry in the radix tree. For example, a router forwarding a ping between 2001:db8:1::1 and 2001:db8:3::1 would get those two cache entries:

$ ip -6 route show cache
2001:db8:1::1 dev r2-r1  metric 0
    cache
2001:db8:3::1 via 2001:db8:2::2 dev r2-r3  metric 0
    cache

These entries are cleaned up by the ip6_dst_gc() function controlled by the following parameters:

$ sysctl -a   grep -F net.ipv6.route
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.mtu_expires = 600

The garbage collector is triggered at most every 500 ms when there are more than 1024 entries or at least every 30 seconds. The garbage collection won t run for more than 60 seconds, except if there are more than 4096 routes. When running, it will first delete entries older than 30 seconds. If the number of cache entries is still greater than 4096, it will continue to delete more recent entries (but no more recent than 512 jiffies, which is the value of gc_elasticity) after a 500 ms pause. Starting from Linux 4.2 (commit 45e4fd26683c), only a PMTU exception would create a cache entry. A router doesn t have to handle those exceptions, so only hosts would get cache entries. And they should be pretty rare. Martin KaFai Lau explains:

Out of all IPv6 RTF_CACHE routes that are created, the percentage that has a different MTU is very small. In one of our end-user facing proxy server, only 1k out of 80k RTF_CACHE routes have a smaller MTU. For our DC traffic, there is no MTU exception.

Here is how a cache entry with a PMTU exception looks like:

$ ip -6 route show cache
2001:db8:1::50 via 2001:db8:1::13 dev out6  metric 0
    cache  expires 573sec mtu 1400 pref medium

Performance We consider three distinct scenarios:

Excerpt of an Internet full view

In this scenario, Linux acts as an edge router attached to the default-free zone. Currently, the size of such a routing table is a little bit above 40,000 routes.

/48 prefixes spread linearly with different densities

Linux acts as a core router inside a datacenter. Each customer or rack gets one or several /48 networks, which need to be routed around. With a density of 1, /48 subnets are contiguous.

/128 prefixes spread randomly in a fixed /108 subnet

Linux acts as a leaf router for a /64 subnet with hosts getting their IP using autoconfiguration. It is assumed all hosts share the same OUI and therefore, the first 40 bits are fixed. In this scenario, neighbor reachability information for the /64 subnet are converted into routes by some external process and redistributed among other routers sharing the same subnet².

Route lookup performance With the help of a small kernel module, we can accurately benchmark³ the ip6_route_output_flags() function and correlate the results with the radix tree size: Getting meaningful results is challenging due to the size of the address space. None of the scenarios have a fallback route and we only measure time for successful hits⁴. For the full view scenario, only the range from 2400::/16 to 2a06::/16 is scanned (it contains more than half of the routes). For the /128 scenario, the whole /108 subnet is scanned. For the /48 scenario, the range from the first /48 to the last one is scanned. For each range, 5000 addresses are picked semi-randomly. This operation is repeated until we get 5000 hits or until 1 million tests have been executed. The relation between the maximum depth and the lookup time is incomplete and I can t explain the difference of performance between the different densities of the /48 scenario. We can extract two important performance points:

• With a full view, the lookup time is 450 ns. This is almost ten times the budget for forwarding at 10 Gbps which is about 50 ns.
• With an almost empty routing table, the lookup time is 150 ns. This is still over the time budget for forwarding at 10 Gbps.

With IPv4, the lookup time for an almost empty table was 20 ns while the lookup time for a full view (500,000 routes) was a bit above 50 ns. How to explain such a difference? First, the maximum depth of the IPv4 LPC-trie with 500,000 routes was 6, while the maximum depth of the IPv6 radix tree for 40,000 routes is 40. Second, while both IPv4 s fib_lookup() and IPv6 s ip6_route_output_flags() functions have a fixed cost implied by the evaluation of routing rules, IPv4 has several optimizations when the rules are left unmodified⁵. Those optimizations are removed on the first modification. If we cancel those optimizations, the lookup time for IPv4 is impacted by about 30 ns. This still leaves a 100 ns difference with IPv6 to be explained. Let s compare how time is spent in each lookup function. Here is a CPU flamegraph for IPv4 s fib_lookup(): Only 50% of the time is spent in the actual route lookup. The remaining time is spent evaluating the routing rules (about 30 ns). This ratio is dependent on the number of routes we inserted (only 1000 in this example). It should be noted the fib_table_lookup() function is executed twice: once with the local routing table and once with the main routing table. The equivalent flamegraph for IPv6 s ip6_route_output_flags() is depicted below: Here is an approximate breakdown on the time spent:

• 50% is spent in the route lookup in the main table,
• 15% is spent in handling locking (IPv4 is using the more efficient RCU mechanism),
• 5% is spent in the route lookup of the local table,
• most of the remaining is spent in routing rule evaluation (about 100 ns)⁶.

Why does the evaluation of routing rules is less efficient with IPv6? Again, I don t have a definitive answer.

History The following graph shows the performance progression of route lookups through Linux history: All kernels are compiled with GCC 4.9 (from Debian Jessie). This version is able to compile older kernels as well as current ones. The kernel configuration is the default one with CONFIG_SMP, CONFIG_IPV6, CONFIG_IPV6_MULTIPLE_TABLES and CONFIG_IPV6_SUBTREES options enabled. Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark. There are three notable performance changes:

• In Linux 3.1, Eric Dumazet delays a bit the copy of route metrics to fix the undesirable sharing of route-specific metrics by all cache entries (commit 21efcfa0ff27). Each cache entry now gets its own metrics, which explains the performance hit for the non-/128 scenarios.
• In Linux 3.9, Yoshifuji Hideaki removes the reference to the neighbor entry in struct rt6_info (commit 887c95cc1da5). This should have lead to a performance increase. The small regression may be due to cache-related issues.
• In Linux 4.2, Martin KaFai Lau prevents the creation of cache entries for most route lookups. The most sensible performance improvement comes with commit 4b32b5ad31a6. The second one is from commit 45e4fd26683c, which effectively removes creation of cache entries, except for PMTU exceptions.

Insertion performance Another interesting performance-related metric is the insertion time. Linux is able to insert a full view in less than two seconds. For some reason, the insertion time is not linear above 50,000 routes and climbs very fast to 60 seconds for 500,000 routes. Despite its more complex insertion logic, the IPv4 subsystem is able to insert 2 million routes in less than 10 seconds.

Memory usage Radix tree nodes (struct fib6_node) and routing information (struct rt6_info) are allocated with the slab allocator⁷. It is therefore possible to extract the information from /proc/slabinfo when the kernel is booted with the slab_nomerge flag:

# sed -ne 2p -e '/^ip6_dst/p' -e '/^fib6_nodes/p' /proc/slabinfo   cut -f1 -d:
   name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>
fib6_nodes         76101  76104     64   63    1
ip6_dst_cache      40090  40090    384   10    1

In the above example, the used memory is 76104 64+40090 384 bytes (about 20 MiB). The number of struct rt6_info matches the number of routes while the number of nodes is roughly twice the number of routes: The memory usage is therefore quite predictable and reasonable, as even a small single-board computer can support several full views (20 MiB for each): The LPC-trie used for IPv4 is more efficient: when 512 MiB of memory is needed for IPv6 to store 1 million routes, only 128 MiB are needed for IPv4. The difference is mainly due to the size of struct rt6_info (336 bytes) compared to the size of IPv4 s struct fib_alias (48 bytes): IPv4 puts most information about next-hops in struct fib_info structures that are shared with many entries.

Conclusion The takeaways from this article are:

• upgrade to Linux 4.2 or more recent to avoid excessive caching,
• route lookups are noticeably slower compared to IPv4 (by an order of magnitude),
• CONFIG_IPV6_MULTIPLE_TABLES option incurs a fixed penalty of 100 ns by lookup,
• memory usage is fair (20 MiB for 40,000 routes).

Compared to IPv4, IPv6 in Linux doesn t foster the same interest, notably in term of optimizations. Hopefully, things are changing as its adoption and use at scale are increasing.

---

1. For a given destination prefix, it s possible to attach source-specific prefixes:

   ip -6 route add 2001:db8:1::/64 \
     from 2001:db8:3::/64 \
     via fe80::1 \
     dev eth0
   

   Lookup is first done on the destination address, then on the source address.
2. This is quite different of the classic scenario where Linux acts as a gateway for a /64 subnet. In this case, the neighbor subsystem stores the reachability information for each host and the routing table only contains a single /64 prefix.
3. The measurements are done in a virtual machine with one vCPU and no neighbors. The host is an Intel Core i5-4670K running at 3.7 GHz during the experiment (CPU governor set to performance). The benchmark is single-threaded. Many lookups are performed and the result reported is the median value. Timings of individual runs are computed from the TSC.
4. Most of the packets in the network are expected to be routed to a destination. However, this also means the backtracking code path is not used in the /128 and /48 scenarios. Having a fallback route gives far different results and make it difficult to ensure we explore the address space correctly.
5. The exact same optimizations could be applied for IPv6. Nobody did it yet.
6. Compiling out table support effectively removes those last 100 ns.
7. There is also per-CPU pointers allocated directly (4 bytes per entry per CPU on a 64-bit architecture). We ignore this detail.

$ strip --strip-debug /usr/local/lib/R/site-library/*/libs/*.so

$ apt-get update
$ apt-get install r-base libcurl4-openssl-dev libssl-dev libxml2-dev
$ apt-get install mg          # a tiny editor
$ mg /etc/R/Rprofile.site     # to add a default CRAN repo
$ R -e 'install.packages("tidyverse")'

root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
4.3M    /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
2.3M    /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
144K    /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
204K    /usr/local/lib/R/site-library/curl/libs/curl.so
328K    /usr/local/lib/R/site-library/digest/libs/digest.so
33M     /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
36K     /usr/local/lib/R/site-library/glue/libs/glue.so
3.2M    /usr/local/lib/R/site-library/haven/libs/haven.so
272K    /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
52K     /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
64K     /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
16K     /usr/local/lib/R/site-library/mime/libs/mime.so
124K    /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
372K    /usr/local/lib/R/site-library/openssl/libs/openssl.so
772K    /usr/local/lib/R/site-library/plyr/libs/plyr.so
92K     /usr/local/lib/R/site-library/purrr/libs/purrr.so
13M     /usr/local/lib/R/site-library/readr/libs/readr.so
4.7M    /usr/local/lib/R/site-library/readxl/libs/readxl.so
1.2M    /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
160K    /usr/local/lib/R/site-library/rlang/libs/rlang.so
928K    /usr/local/lib/R/site-library/scales/libs/scales.so
4.9M    /usr/local/lib/R/site-library/stringi/libs/stringi.so
1.3M    /usr/local/lib/R/site-library/tibble/libs/tibble.so
2.0M    /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
1.2M    /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
4.7M    /usr/local/lib/R/site-library/xml2/libs/xml2.so
78M     total
root@de443801b3fc:/# 

root@de443801b3fc:/# strip --strip-debug /usr/local/lib/R/site-library/*/libs/*so
root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
440K    /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
220K    /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
52K     /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
56K     /usr/local/lib/R/site-library/curl/libs/curl.so
120K    /usr/local/lib/R/site-library/digest/libs/digest.so
2.5M    /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
16K     /usr/local/lib/R/site-library/glue/libs/glue.so
404K    /usr/local/lib/R/site-library/haven/libs/haven.so
76K     /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
20K     /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
24K     /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
8.0K    /usr/local/lib/R/site-library/mime/libs/mime.so
52K     /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
84K     /usr/local/lib/R/site-library/openssl/libs/openssl.so
76K     /usr/local/lib/R/site-library/plyr/libs/plyr.so
32K     /usr/local/lib/R/site-library/purrr/libs/purrr.so
648K    /usr/local/lib/R/site-library/readr/libs/readr.so
400K    /usr/local/lib/R/site-library/readxl/libs/readxl.so
128K    /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
56K     /usr/local/lib/R/site-library/rlang/libs/rlang.so
100K    /usr/local/lib/R/site-library/scales/libs/scales.so
496K    /usr/local/lib/R/site-library/stringi/libs/stringi.so
124K    /usr/local/lib/R/site-library/tibble/libs/tibble.so
164K    /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
104K    /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
344K    /usr/local/lib/R/site-library/xml2/libs/xml2.so
6.6M    total
root@de443801b3fc:/#

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• the Debian wiki page for Suricata
• the official Suricata upstream docs

% sudo aptitude install suricata

% sudo systemctl status suricata
  suricata.service - Suricata IDS/IDP daemon
   Loaded: loaded (/lib/systemd/system/suricata.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-08-19 12:50:49 CEST; 44min ago
     Docs: man:suricata(8)
           man:suricatasc(8)
           https://redmine.openinfosecfoundation.org/projects/suricata/wiki
 Main PID: 1101 (Suricata-Main)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/suricata.service
            1101 /usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid
ago 19 12:50:44 nostromo systemd[1]: Starting Suricata IDS/IDP daemon...
ago 19 12:50:47 nostromo suricata[1032]: 19/8/2017 -- 12:50:47 - <Notice> - This is Suricata version 4.0.0 RELEASE
ago 19 12:50:49 nostromo systemd[1]: Started Suricata IDS/IDP daemon.

% sudo suricatasc -c uptime
 "message": 3892, "return": "OK" 

[...]
# Linux high speed capture support
af-packet:
  - interface: wlan0
[...]

% sudo systemctl restart suricata
% sudo tail -f /var/log/suricata/fast.log
08/19/2017-14:03:04.025898  [**] [1:2012648:3] ET POLICY Dropbox Client Broadcasting [**] \
	[Classification: Potential Corporate Privacy Violation] [Priority: 1]  UDP  192.168.1.36:17500 -> 255.255.255.255:17500

% sudo aptitude install suricata-oinkmaster

Changes in RcppArmadillo version 0.7.960.1.0 (2017-08-11)

• Upgraded to Armadillo release 7.960.1 (Northern Banana Republic Deluxe)
  • faster randn() when using OpenMP (NB: usually omitted when used fromR)
  • faster gmm_diag class, for Gaussian mixture models with diagonal covariance matrices
  • added .sum_log_p() to the gmm_diag class
  • added gmm_full class, for Gaussian mixture models with full covariance matrices
  • expanded .each_slice() to optionally use OpenMP for multi-threaded execution
• Upgraded to Armadillo release 7.950.0 (Northern Banana Republic)
  • expanded accu() and sum() to use OpenMP for processing expressions with computationally expensive element-wise functions
  • expanded trimatu() and trimatl() to allow specification of the diagonal which delineates the boundary of the triangular part
• Enhanced support for sparse matrices (Binxiang Ni as part of Google Summer of Code 2017)
  • Add support for dtCMatrix and dsCMatrix (#135)
  • Add conversion and unit tests for dgT, dtT and dsTMatrix (#136)
  • Add conversion and unit tests for dgR, dtR and dsRMatrix (#139)
  • Add conversion and unit tests for pMatrix and ddiMatrix (#140)
  • Rewrite conversion for dgT, dtT and dsTMatrix, and add file-based tests (#142)
  • Add conversion and unit tests for indMatrix (#144)
  • Rewrite conversion for ddiMatrix (#145)
  • Add a warning message for matrices that cannot be converted (#147)
  • Add new vignette for sparse matrix support (#152; Dirk in #153)
  • Add support for sparse matrix conversion from Python SciPy (#158 addressing #141)
• Optional return of row or column vectors in collapsed form if appropriate #define is set (Serguei Sokol in #151 and #154)
• Correct speye() for non-symmetric cases (Qiang Kou in #150 closing #149).
• Ensure tests using Scientific Python and reticulate are properly conditioned on the packages being present.
• Added .aspell/ directory with small local directory now supported by R-devel.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• There were two talks mainly about Reproducible Builds at DebConf17:
  • Reproducible builds: Status update by Chris Lamb, Holger Levsen, Maria Glukhova, Steven Chamberlain, Vagrant Cascadian, Valerie Young and Ximin Luo.
  • Fun with .buildinfo by Steven Chamberlain.
• Reproducible builds were also mentioned in many other talks including:
  • Using Qubes OS from the POV of a Debian developer by Holger Levsen.
  • Debian for Medical Software by Andr Roth.
  • Let's maintain jenkins.debian.org as a team by Holger Levsen.
  • Signing package contents: why and how by Matthew Garrett.
  • Bits from the DPL by Chris Lamb.
  • Installing Debian by Vagrant Cascadian.
  • in-toto -- Securing supply chains as a whole by Lukas Puehringer.
  • Debian Cloud BoF by Steve McIntyre.
  • Will there be Debian in your next BMW car? by Aigars Mahinovs.
• There has also been substantial activity regarding updating the Debian Policy regarding Reproducible Builds (#844431). This work is still ongoing and will be covered in future issues of this weekly blog.

• Adrian Bunk:
  • #871818 filed against debian-zh-faq.
  • #871907 filed against praat.
• Katsuhiko Nishimra:
  • #871591 filed against llvm-toolchain-snapshot.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
  • #871345 filed against texlive-base.
  • #871349 filed against ispell-uk.
  • #871357 filed against po4a.
• jathan:
  • #870890 filed against apg.

• Bernhard M. Wiedemann:
  • enblend (merged), SOURCE_DATE_EPOCH support
  • systemtap
  • splint (dead project)
  • skelcd-openSUSE

• During our reproducibility testing, Adrian Bunk filed 48 FTBFS bugs this week.

• Mattia Rizzolo:
  • debian/copyright: coalesce some file paragraphs and update information.
  • Bump Standards-Version to 4.0.1, no changes needed.

• Chris Lamb:
  • Test --help output in autopkgtests

• Mattia:
  • Notify the#debian-reproducible-changes IRC channel for unreproducible -> FTBFS transitions.
  • Update squid.conf for all nodes to 5.2.23 (and fixup some).
  • Enable the Munin Squid plugin on the Codethink arm64 nodes as well.
  • Force reconfiguration of Apache and Munin when update_jdn.sh is updated.
• Holger worked on slides for his DebConf17 BoF about migrating to jenkins.debian.org, which affects tests.r-b.o as well.

SHLIB_CXXLDFLAGS = -Wl,-S -shared
SHLIB_CXX11LDFLAGS = -Wl,-S -shared
SHLIB_CXX14LDFLAGS = -Wl,-S -shared
SHLIB_FCLDFLAGS = -Wl,-S -shared
SHLIB_LDFLAGS = -Wl,-S -shared

edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ ls -lh src/RProtoBuf.so
-rwxr-xr-x 1 edd edd 12M Aug 14 20:22 src/RProtoBuf.so
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ 

edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ ls -lh src/RProtoBuf.so 
-rwxr-xr-x 1 edd edd 626K Aug 14 20:29 src/RProtoBuf.so
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ 

strippedLib: $(SHLIB)
        if test -e "/usr/bin/strip"; then /usr/bin/strip --strip-debug $(SHLIB); fi
.phony: strippedLib

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Changes in RProtoBuf version 0.4.10 (2017-08-13)

• More careful operation in deserializer checking for a valid class attribute (Jeffrey Shen in #29 fixing #28)
• At the request of CRAN, correct one .Call() argument to PACKAGE=; update routine registration accordingly

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

In Canada "consent means the voluntary agreement of the complainant to engage in sexual activity" without abuse or exploitation of "trust, power or authority", coercion or threats.[7] Consent can also be revoked at any moment.[8] There are 3 pillars often included in the description of sexual consent, or "the way we let others know what we're up for, be it a good-night kiss or the moments leading up to sex." They are:

• Knowing exactly what and how much I'm agreeing to
• Expressing my intent to participate
• Deciding freely and voluntarily to participate[20]

• https://www.morethantwo.com/blog/2013/10/guest-post-on-consent-in-romantic-relationships

• how many of you do things in Debian because you want to?
• how many of you do things in Debian because you have to?
• how many of you do both?

• healthy sustain the relationship
• unhealthy may explode spectacularly at some point

• I have an idea for for something fun or useful
• I see something broken and I have an idea how to fix it

• Nobody else can do it
• Nobody else can be trusted to do it
• Nobody else who can and can be trusted to do it is doing it
• Somebody is paying me to do it

• it takes me little time?
• it's useful for me
• It's fun
• It saves me time to keep something running that if it breaks when I or somebody else needs it
• it makes me feel useful? (then if I stop, I become useless, then I can't afford to stop?)
• Getting positive feedback (more "you're good" than "i use it") ?

• what tasks are done by people motivated by guilt / motivated by Galadriel?
• if I volunteer to help team X that is in trouble, will I get stuck doing all the work as soon as people realise things move fine again and finally feel free to step back?

"Spoons" are a visual representation used as a unit of measure used to quantify how much energy a person has throughout a given day. Each activity requires a given number of spoons, which will only be replaced as the person "recharges" through rest. A person who runs out of spoons has no choice but to rest until their spoons are replenished.

• https://s-media-cache-ak0.pinimg.com/736x/c3/05/dc/c305dc9d3ac3c2219f267dbc47bfacf5.jpg
• https://feckingme.files.wordpress.com/2015/05/keep-track-of-your-spoons.jpg

• Routine task: 1 spoon
• Context switch: 2 spoons
• New corner case: 5 spoons
• Well written bug report: -1 spoon
• Interesting thread: -1 spoon
• New flamewar: 3 spoons

• are the things that we expect a person to do the things that one person can do?
• having a history of people who can do it anyway is no excuse: show compassion to those people, take some of their work, thank them, but don't glorify them

• If someone is out of spoons, they're out of spoons; they don't get more spoons if you insist; if you insist, you probably take even more spoons from them
• If someone needs their spoons for something else, they are entitled to
• If someone gets out of spoons for something important, and nobody else can do it, it's not their fault but a shared responsibility

• If you are out of spoons, you are out of spoons
• If you need spoons for something else, use them for something else
• If you are out of spoons for something important, and nobody else can do it, it's not your problem, it's a problem in the community

• Don't expect few people to take care of everything
• Leave space for more people to take responsibility for things
• Turnover empowers
• Humans are a renewable resource, but only if you think of them that way

• routine is ok
• non-routine tends to get stuck in the mailbox waiting for a day with more free time
  • are we able to listen when tricky cases happen?
  • are we able to respond when tricky cases happen?
  • are we able to listen/respond when socially tricky cases happen?
  • are we able to listen/respond when harassment happens?
  • can we tell people raising valid issues from troublemakers?
• in NM
  • it's easier to accept a new maintainer than to reject them
  • it's hard or impossible to make a call on a controversial candidate when one doesn't know that side of Debian
• I'm tired, why you report a bug? I don't want to deal with your bug. Maybe you are wrong and your bug is invalid? It would be good if you were wrong and your bug was invalid.
• even politeness goes when out of spoons because it's too much effort

• release team
• DAM (amount of energy to deal with when things go wrong; only dealing with when things go spectacularly wrong?)
• DPL (how many people candidate to this year elections?)

• heroes are rare
• heroes are hard to replace
• heroes burn out
• heroes can become martyrs
• heroes can become villains
• heroes tend to be accidents waiting to happen

• Someone who has a lot of energy can take more and more tasks out of people who have less, and slowly drive all the others away
• Good for results, bad for the team
• Then we have another person who is too big to fail, and another accident waiting to happen

• you think everything is fine, and you discover they've been suffering all the time and you never had a clue

• They depend on a constant supply of achievement or admiration to have a sense of worth, therefore have a lot of spoons to invest into getting it
• However, when something they do is challenged, such as by pointing out a mistake one has made, or a problem with their behaviour, all hell breaks loose, beacuse they see their whole sense of worth challenged, too

• Examples of abuse patterns (trigger warning: judgemental)
• Wikipedia on narcissist personality, supply, injury and rage (trigger warning: medical)

• These topics are often very medicalised, and targeted at people who are victims of abuse and domestic violence.
• I find them useful to develop regular expressions for pattern matching of behaviours, and I consider them dangerous if they are used for pattern matching of people.

• you heard someone say no
• you had an argument with someone and the outcome was positive for both

• you feel demeaned
• you feel invalidated

My software ate
the files
that where in
your home directory and which
you were probably
needing
for work Forgive me
it was so quick to write
without tests
and it worked so well for me

When you get involved in a new community, such as Debian, find out early where, if that happens, you can find support, understanding, and help to make it stop. If you cannot find any, or if the only thing you can find is people who say "it never happens here", consider whether you really want to be in that community.

There are some nice people in the world. I mean nice people, the sort I couldn t describe myself as. People who are friends with everyone, who are somehow never involved in any argument, who seem content to spend their time drawing pictures of bumblebees on flowers that make everyone happy. Those people are great to have around. You want to hold onto them as much as you can. But people only have so much tolerance for jerkiness, and really nice people often have less tolerance than the rest of us. The trouble with not ejecting a jerk whether their shenanigans are deliberate or incidental is that you allow the average jerkiness of the community to rise slightly. The higher it goes, the more likely it is that those really nice people will come around less often, or stop coming around at all. That, in turn, makes the average jerkiness rise even more, which teaches the original jerk that their behavior is acceptable and makes your community more appealing to other jerks. Meanwhile, more people at the nice end of the scale are drifting away.

• Angela, my wife, I met daily on Jabber. Thanks for letting me go to this great DebConf17 conference and keeping our family up and running
• Andreas asking people to either impersonate his wife or adoptive daughter for a photo shooting. You gave such a touching talk on Friday, together with Minh from Vietnam.
• Holger for nagging us about stone age bugs in the Debian Blends package and the outdated software list in Debian Edu (Kernel 2.6.32 package are finally not mentioned anymore)
• Vagrant, Foetini and Alkis for there efforts on LTSP and their success in Greece with bringing Debian into Greek schools
• Tiago, Jerome and all the others from the local team, providing us with such great food and support. THANKS folks!!!
• Enrico who showed my his 20 liner version of nodm aka lightdm-autologin-greeter (and also made me curious on staticsite)
• Jonas Smedegaard for teaching me the solarized theme and loads of other things
• Siri for being around and really having a stand for making Debian look more like a product
• Dimitri John Ledkov for chiming in on Ayatana Indicators as next Indicators upstream for Ubuntu 18.04 LTS
• Chrys for chiming on .desktop file proxying, meet you back in #arctica on Freenode
• Sean for asking me daily, if my luggage had arrived (see below), as he shared the same fate during DebCamp
• the owner of that nice shop where I bought loads of clothes while waiting for my luggage still stuck in Hamburg
• Steven for looking into gcc compiler macros with me for nx-libs autotools conversion, also probably - luggage wise - the lightest traveller among us
• Fabian for sharing is sadness and pain about the FLOSS non-situation in schools all over Quebec
• Mario from New Zealand and Jos from the Netherlands chiming in on FLOSS and education on IRC after having watch my talk about Debian Edu / Skolelinux. Mario, we will soon ask you for opensourcing your teacher screen over WiFi solution...
• Lior who thinks about bringing Debian Edu / Skolelinux to Israel. (That would be awesome!)
• Rhonda for having time for my Debian Backports woes and probably having been quite forgiving ;-)
• Bobby who is a font engineer during the week and (used to be) a cave explorer and mapper in his spare time
• Ximin for providing deep insight in the key signing workflow and the caff approach to it
• Daniel for sharing work experiences and nudging me to go with Remote Desktop stuff (out of pure personal interest *g*, of course, but still)
• Tzfarir and Gunnar for a nice chat on the last night of DebConf

• Finding my luggage
  • After I had arrived at Montreal Airport, I found out that my luggage stayed in Hamburg
  • So the first 4.5 days, I was continuously busy with calling Lufthansa for package item tracking...
  • Go shopping twice, to update my plastic bag of fresh clothes...
• MATE 1.18 in Debian
  • Finalize package builds of MATE 1.18 in Debian unstable (because of some GLib2.0 regression, thanks to Iain Lane for the prompt fix and upload)
• Debian Edu
  • clear up src:package debian-edu regarding the task files related to Debian Pure Blends
  • this work is still in progress...
• Debian Blends (esp. the blends-dev part of the blends src:package)
  • You can now have empty Depends: / Recommends: / Suggests: fields with the list of packages then starting in the next line.
  • It is now possible to have real Depends: fields in task files that become Depends: fields in debian/control. Packages targetting Depends: that are not in unstable get de-promoted to the Suggests: field in debian/control.
  • Tested with most available Debian Pure Blends meta-packages
  • I also pointed Daniel Pocock with his new GnuPG clean-room project towards Debian Pure Blends
• Ring: A 'new' distributed video chat tool without mediating servers. Good concept, however, we could not get it to work on the DebConf campus.
• Debian Design Team (which I am now a member of, I guess)
  • Dive into and out of the vision of a Debian Uniform set of packages, turning the collection of software in Debian into one thing.
  • Run my terminal applications now with the Base16 profile 'solarized-universal'. However, Debian Design will be much more than 16 colors in a console terminal.
  • Let's turn Debian into something like a potential product!
• Debian Policy:
  • I even helped with a Debian Policy bug...
• Skolab Groupware: Forking Kolab (v2) as a new project, named the Skolab Groupware. Instead of migrating my own mail server away from Kolab (v2), I chose continuing maintenance for it, at least for the core compoents:
  • https://github.com/Skolab-Groupware
  • Targetting small to medium sized installations with little administration effort.
  • I mainly ported Kolab's kolab-webadmin PHP project to a recent status so that it can run under Debian testing/unstable: https://github.com/Skolab-Groupware/skolab-webadmin
  • Big thanks to Siri Reiter for consulting me on artwork and for helping me putting together a simple logo for the project.
• nx-libs: Work on several PRs:
  • https://github.com/ArcticaProject/nx-libs/pull/496 (review + merge)
  • https://github.com/ArcticaProject/nx-libs/pull/498 (code)
  • https://github.com/ArcticaProject/nx-libs/pull/499 (code)
  • https://github.com/ArcticaProject/nx-libs/pull/501 (review + merge)
  • Play with KDE5 and nx-libs: IT WORKS!
• Weblate:
  • Become hosted by hosted Weblate for...
  • Ayatana Indicators
  • Arctica Project
  • Skolab Groupware
  Thanks to Michal iha for providing this fine translation service

• Talk: Supporting Debian Edu / Skolelinux as a Product: Lessons Learned
  • Schedule: https://debconf17.debconf.org/talks/169
  • Video: http://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/su...
  • Outcome: https://sunweavers.net/blog/node/59
• Talk: Ayatana Indicators
  • Schedule: https://debconf17.debconf.org/talks/168
  • Video: http://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/ay...
  • Outcome: https://sunweavers.net/blog/node/60
• Ad-hoc BoF: Bits from the Debian+Ubuntu MATE Packaging Team
  • Schedule: https://debconf17.debconf.org/talks/223
  • Outcome: still to be published...
• Ad-hoc BoF: Debian for Remote Desktop Computing
  • Schedule: https://debconf17.debconf.org/talks/60
  • Outcome: still to be published...

• mate-settings-daemon
• mate-dock-applet
• brisk-menu
• mate-optimus
• caja-actions
• mate-common
• mate-tweak
• plank
• freerdp (2x)
• freerdp2
• gosa-plugin-mailaddress

• python-wither
• lightdm-autologin-greeter
• caja-rename

• freerdp (1.1) (actually twice, one of them a security upload)
• gosa-plugin-mailaddress
• mate-themes

• Prepare upload of caja-admin by asking for release tags upstream
• Talk Clint Byrums into a fresh upload of the long not touch undistract-me package
• Breed on different desktop layouts for Debian MATE (like in Ubuntu MATE)
• Do quite a bit of GnuPG key signing
• Update my consent with NM to pick up my work on collab-maint request processing again

• Various performance improvements.
• Added support for nested JSON format.
• Added support for WebExtension JSON format.
• Fixed git exporter authentication.
• Improved CSV import in certain situations.
• Improved look of Other translations widget.
• The max-length checks is now enforcing length of text in form.
• Make the commit_pending age configurable per component.
• Various user interface cleanups.
• Fixed component/project/sitewide search for translations.

Filed under: Debian English SUSE Weblate

edd@bud:~/git/rcpparmadillo/.aspell(master)$ cat defaults.R 
Rd_files <- vignettes <- R_files <- description <-
    list(encoding = "UTF-8",
         language = "en",
         dictionaries = c("en_stats", "RcppArmadillo"))
edd@bud:~/git/rcpparmadillo/.aspell(master)$ r -p -e 'readRDS("RcppArmadillo.rds")'
[1] "MPL"            "Sanderson"      "Templated"
[4] "decompositions" "onwards"        "templated"
edd@bud:~/git/rcpparmadillo/.aspell(master)$     

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

pristine-tar being is unable to reproduce a tarball After importing the latest upstream tarball using gbp import-orig, I tried to build the package but ran into this pristine-tar error:

$ gbp buildpackage
gbp:error: Pristine-tar couldn't checkout "planetfilter_0.7.4.orig.tar.gz": xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3 decode failed! at /usr/share/perl5/Pristine/Tar/DeltaTools.pm line 56.
pristine-tar: command failed: pristine-gz --no-verbose --no-debug --no-keep gengz /tmp/user/1000/pristine-tar.mgnaMjnwlk/wrapper /tmp/user/1000/pristine-tar.EV5aXIPWfn/planetfilter_0.7.4.orig.tar.gz.tmp
pristine-tar: failed to generate tarball

So I decided to throw away what I had, re-import the tarball and try again. This time, I got a different pristine-tar error:

$ gbp buildpackage
gbp:error: Pristine-tar couldn't checkout "planetfilter_0.7.4.orig.tar.gz": xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3 decode failed! at /usr/share/perl5/Pristine/Tar/DeltaTools.pm line 56.
pristine-tar: command failed: pristine-gz --no-verbose --no-debug --no-keep gengz /tmp/user/1000/pristine-tar.mgnaMjnwlk/wrapper /tmp/user/1000/pristine-tar.EV5aXIPWfn/planetfilter_0.7.4.orig.tar.gz.tmp
pristine-tar: failed to generate tarball

I filed bug 871938 for this. As a work-around, I simply symlinked the upstream tarball I already had and then built the package using the tarball directly instead of the upstream git branch:

ln -s ~/deve/remote/planetfilter/dist/planetfilter-0.7.4.tar.gz ../planetfilter_0.7.4.orig.tar.gz
gbp buildpackage --git-tarball-dir=..

Given that only the upstream and master branches are signed, the .delta file on the pristine-tar branch could be fixed at any time in the future by committing a new .delta file once pristine-tar gets fixed. This therefore seems like a reasonable work-around.

git-buildpackage doesn't import the upstream tarball signature The second problem I ran into was a missing upstream signature after building the package with git-buildpackage:

$ lintian -i planetfilter_0.7.4-1_amd64.changes
E: planetfilter changes: orig-tarball-missing-upstream-signature planetfilter_0.7.4.orig.tar.gz
N: 
N:    The packaging includes an upstream signing key but the corresponding
N:    .asc signature for one or more source tarballs are not included in your
N:    .changes file.
N:    
N:    Severity: important, Certainty: certain
N:    
N:    Check: changes-file, Type: changes
N: 

This problem (and the lintian error I suspect) is fairly new and hasn't been solved yet. So until gbp import-orig gets proper support for upstream signatures, my work-around was to copy the upstream signature in the export-dir output directory (which I set in ~/.gbp.conf) so that it can be picked up by the final stages of gbp buildpackage:

ln -s ~/deve/remote/planetfilter/dist/planetfilter-0.7.4.tar.gz.asc ../build-area/planetfilter_0.7.4.orig.tar.gz.asc

If there's a better way to do this, please feel free to leave a comment (authentication not required)!