• Remindr on Gitlab.com (would be awesome starring it if you like/use it)
• The official documentation on Readthedocs.io

o en Automatically Send Toots To The Mastodon Social Network https://carlchenet.com/automatically-send-toots-to-the-mastodon-social-network/ #Mastodon
x fr Sur Mastodon, cr er son compte de secours  ou tout perdre https://carlchenet.com/sur-mastodon-creer-son-compte-de-secours-ou-tout-perdre/ #Mastodon
x en Automatically boost cool toots on Mastodon with the Boost bot https://carlchenet.com/automatically-boost-cool-toots-on-mastodon-with-the-boost-bot/ #Mastodon
x en Cryptocurrencies On the New Social Network Mastodon #Mastodon #bitcoin #ethereum #monero

• JPEG is really impressive technology. It's made in 1992, and it's still frighteningly close to state of the art! Beating it without getting a lot slower isn't trivial at all; witness that even with WebP etc., we still don't have a JPEG-killer because it's just so incredibly good. (Granted, in 1992, decompressing a JPEG would easily take half a minute or more.)
• rANS is pretty neat; it gives you almost all the advantages of arithmetic coding, but does without divisions in the decoder (and if you have static probabilities and fast high multiplies, also in the encoder). I still don't feel like I understand it fully, though. And it has the odd property of having to encode and decode in different directions, which may or may not be a problem to you. But ryg's interleaving tricks to get SIMD are pretty nice; I won't claim to fully understand those either. Maybe I will need to eventually.
• GPU performance models continue to bend my head. GPUs may have managed to make a parallel programming model that people actually manage to use, but getting really high performance out of compute shaders is still hard to get use to.
• It really annoys me that Quick Sync doesn't support luma-only encoding (ie. 4:0:0); the best you can do is seemingly 4:2:0 and then have empty chroma planes, which wastes resources. It would have been a nice way to hack around the limitation that you can't have 4:2:2 or alpha.
• Time-frequency switching is the solution to a problem that sounds trivial if you're a DSP beginner and nearly impossible if you've actually done some DSP. I'm not intuitively convinced it's worth it if you have fast float muladds, but it's pretty neat. Allowing multiple block sizes to an intraframe codec has a cost in that you'll need a size decision heuristic, though, which pretty much instantly complicates the encoder.

• The excellent Google Summer of Code work by Binxiang Ni had only encountered direct use of sparse matrices as produced by the Matrix. However, while we waited for 0.7.960.1.0 to make it onto CRAN, the quanteda package switched to derived classes---which we now account for via the is() method of our S4 class. Thanks to Kevin Ushey for reminding me we had is().
• We somehow missed to account for the R 3.4.* and Rcpp 0.12. 11,12 changes for package registration (with .registration=TRUE), so ensured we only have one fastLm symbol.
• The build did not take not too well to systems without OpenMP, so we now explicitly unset supported via an Armadillo configuration variable. In general, client packages probably want to enable C++11 support when using OpenMP (explicitly) but we prefer to not upset too many (old) users. However, our configure check now also wants g++ 4.7.2 or later just like Armadillo.

Changes in RcppArmadillo version 0.7.960.1.1 (2017-08-20)

• Added improved check for inherited S4 matrix classes (#162 fixing #161)
• Changed fastLm C++ function to fastLm_impl to not clash with R method (#164 fixing #163)
• Added OpenMP check for configure (#166 fixing #165)

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Lookup trie implementation Looking up a prefix in a routing table comes down to find the most specific entry matching the requested destination. A common structure for this task is the trie, a tree structure where each node has its parent as prefix. With IPv4, Linux uses a level-compressed trie (or LPC-trie), providing good performances with low memory usage. For IPv6, Linux uses a more classic radix tree (or Patricia trie). There are three reasons for not sharing:

• The IPv6 implementation (introduced in Linux 2.1.8, 1996) predates the IPv4 implementation based on LPC-tries (in Linux 2.6.13, commit 19baf839ff4a).
• The feature set is different. Notably, IPv6 supports source-specific routing¹ (since Linux 2.1.120, 1998).
• The IPv4 address space is denser than the IPv6 address space. Level-compression is therefore quite efficient with IPv4. This may not be the case with IPv6.

The trie in the below illustration encodes 6 prefixes: For more in-depth explanation on the different ways to encode a routing table into a trie and a better understanding of radix trees, see the explanations for IPv4. The following figure shows the in-memory representation of the previous radix tree. Each node corresponds to a struct fib6_node. When a node has the RTN_RTINFO flag set, it embeds a pointer to a struct rt6_info containing information about the next-hop. The fib6_lookup_1() function walks the radix tree in two steps:

1. walking down the tree to locate the potential candidate, and
2. checking the candidate and, if needed, backtracking until a match.

Here is a slightly simplified version without source-specific routing:

static struct fib6_node *fib6_lookup_1(struct fib6_node *root,
                                       struct in6_addr  *addr)
 
    struct fib6_node *fn;
    __be32 dir;
    /* Step 1: locate potential candidate */
    fn = root;
    for (;;)  
        struct fib6_node *next;
        dir = addr_bit_set(addr, fn->fn_bit);
        next = dir ? fn->right : fn->left;
        if (next)  
            fn = next;
            continue;
         
        break;
     
    /* Step 2: check prefix and backtrack if needed */
    while (fn)  
        if (fn->fn_flags & RTN_RTINFO)  
            struct rt6key *key;
            key = fn->leaf->rt6i_dst;
            if (ipv6_prefix_equal(&key->addr, addr, key->plen))  
                if (fn->fn_flags & RTN_RTINFO)
                    return fn;
             
         
        if (fn->fn_flags & RTN_ROOT)
            break;
        fn = fn->parent;
     
    return NULL;
 

Caching While IPv4 lost its route cache in Linux 3.6 (commit 5e9965c15ba8), IPv6 still has a caching mechanism. However cache entries are directly put in the radix tree instead of a distinct structure. Since Linux 2.1.30 (1997) and until Linux 4.2 (commit 45e4fd26683c), almost any successful route lookup inserts a cache entry in the radix tree. For example, a router forwarding a ping between 2001:db8:1::1 and 2001:db8:3::1 would get those two cache entries:

$ ip -6 route show cache
2001:db8:1::1 dev r2-r1  metric 0
    cache
2001:db8:3::1 via 2001:db8:2::2 dev r2-r3  metric 0
    cache

These entries are cleaned up by the ip6_dst_gc() function controlled by the following parameters:

$ sysctl -a   grep -F net.ipv6.route
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.mtu_expires = 600

The garbage collector is triggered at most every 500 ms when there are more than 1024 entries or at least every 30 seconds. The garbage collection won t run for more than 60 seconds, except if there are more than 4096 routes. When running, it will first delete entries older than 30 seconds. If the number of cache entries is still greater than 4096, it will continue to delete more recent entries (but no more recent than 512 jiffies, which is the value of gc_elasticity) after a 500 ms pause. Starting from Linux 4.2 (commit 45e4fd26683c), only a PMTU exception would create a cache entry. A router doesn t have to handle those exceptions, so only hosts would get cache entries. And they should be pretty rare. Martin KaFai Lau explains:

Out of all IPv6 RTF_CACHE routes that are created, the percentage that has a different MTU is very small. In one of our end-user facing proxy server, only 1k out of 80k RTF_CACHE routes have a smaller MTU. For our DC traffic, there is no MTU exception.

Here is how a cache entry with a PMTU exception looks like:

$ ip -6 route show cache
2001:db8:1::50 via 2001:db8:1::13 dev out6  metric 0
    cache  expires 573sec mtu 1400 pref medium

Performance We consider three distinct scenarios:

Excerpt of an Internet full view

In this scenario, Linux acts as an edge router attached to the default-free zone. Currently, the size of such a routing table is a little bit above 40,000 routes.

/48 prefixes spread linearly with different densities

Linux acts as a core router inside a datacenter. Each customer or rack gets one or several /48 networks, which need to be routed around. With a density of 1, /48 subnets are contiguous.

/128 prefixes spread randomly in a fixed /108 subnet

Linux acts as a leaf router for a /64 subnet with hosts getting their IP using autoconfiguration. It is assumed all hosts share the same OUI and therefore, the first 40 bits are fixed. In this scenario, neighbor reachability information for the /64 subnet are converted into routes by some external process and redistributed among other routers sharing the same subnet².

Route lookup performance With the help of a small kernel module, we can accurately benchmark³ the ip6_route_output_flags() function and correlate the results with the radix tree size: Getting meaningful results is challenging due to the size of the address space. None of the scenarios have a fallback route and we only measure time for successful hits⁴. For the full view scenario, only the range from 2400::/16 to 2a06::/16 is scanned (it contains more than half of the routes). For the /128 scenario, the whole /108 subnet is scanned. For the /48 scenario, the range from the first /48 to the last one is scanned. For each range, 5000 addresses are picked semi-randomly. This operation is repeated until we get 5000 hits or until 1 million tests have been executed. The relation between the maximum depth and the lookup time is incomplete and I can t explain the difference of performance between the different densities of the /48 scenario. We can extract two important performance points:

• With a full view, the lookup time is 450 ns. This is almost ten times the budget for forwarding at 10 Gbps which is about 50 ns.
• With an almost empty routing table, the lookup time is 150 ns. This is still over the time budget for forwarding at 10 Gbps.

With IPv4, the lookup time for an almost empty table was 20 ns while the lookup time for a full view (500,000 routes) was a bit above 50 ns. How to explain such a difference? First, the maximum depth of the IPv4 LPC-trie with 500,000 routes was 6, while the maximum depth of the IPv6 radix tree for 40,000 routes is 40. Second, while both IPv4 s fib_lookup() and IPv6 s ip6_route_output_flags() functions have a fixed cost implied by the evaluation of routing rules, IPv4 has several optimizations when the rules are left unmodified⁵. Those optimizations are removed on the first modification. If we cancel those optimizations, the lookup time for IPv4 is impacted by about 30 ns. This still leaves a 100 ns difference with IPv6 to be explained. Let s compare how time is spent in each lookup function. Here is a CPU flamegraph for IPv4 s fib_lookup(): Only 50% of the time is spent in the actual route lookup. The remaining time is spent evaluating the routing rules (about 30 ns). This ratio is dependent on the number of routes we inserted (only 1000 in this example). It should be noted the fib_table_lookup() function is executed twice: once with the local routing table and once with the main routing table. The equivalent flamegraph for IPv6 s ip6_route_output_flags() is depicted below: Here is an approximate breakdown on the time spent:

• 50% is spent in the route lookup in the main table,
• 15% is spent in handling locking (IPv4 is using the more efficient RCU mechanism),
• 5% is spent in the route lookup of the local table,
• most of the remaining is spent in routing rule evaluation (about 100 ns)⁶.

Why does the evaluation of routing rules is less efficient with IPv6? Again, I don t have a definitive answer.

History The following graph shows the performance progression of route lookups through Linux history: All kernels are compiled with GCC 4.9 (from Debian Jessie). This version is able to compile older kernels as well as current ones. The kernel configuration is the default one with CONFIG_SMP, CONFIG_IPV6, CONFIG_IPV6_MULTIPLE_TABLES and CONFIG_IPV6_SUBTREES options enabled. Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark. There are three notable performance changes:

• In Linux 3.1, Eric Dumazet delays a bit the copy of route metrics to fix the undesirable sharing of route-specific metrics by all cache entries (commit 21efcfa0ff27). Each cache entry now gets its own metrics, which explains the performance hit for the non-/128 scenarios.
• In Linux 3.9, Yoshifuji Hideaki removes the reference to the neighbor entry in struct rt6_info (commit 887c95cc1da5). This should have lead to a performance increase. The small regression may be due to cache-related issues.
• In Linux 4.2, Martin KaFai Lau prevents the creation of cache entries for most route lookups. The most sensible performance improvement comes with commit 4b32b5ad31a6. The second one is from commit 45e4fd26683c, which effectively removes creation of cache entries, except for PMTU exceptions.

Insertion performance Another interesting performance-related metric is the insertion time. Linux is able to insert a full view in less than two seconds. For some reason, the insertion time is not linear above 50,000 routes and climbs very fast to 60 seconds for 500,000 routes. Despite its more complex insertion logic, the IPv4 subsystem is able to insert 2 million routes in less than 10 seconds.

Memory usage Radix tree nodes (struct fib6_node) and routing information (struct rt6_info) are allocated with the slab allocator⁷. It is therefore possible to extract the information from /proc/slabinfo when the kernel is booted with the slab_nomerge flag:

# sed -ne 2p -e '/^ip6_dst/p' -e '/^fib6_nodes/p' /proc/slabinfo   cut -f1 -d:
   name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>
fib6_nodes         76101  76104     64   63    1
ip6_dst_cache      40090  40090    384   10    1

In the above example, the used memory is 76104 64+40090 384 bytes (about 20 MiB). The number of struct rt6_info matches the number of routes while the number of nodes is roughly twice the number of routes: The memory usage is therefore quite predictable and reasonable, as even a small single-board computer can support several full views (20 MiB for each): The LPC-trie used for IPv4 is more efficient: when 512 MiB of memory is needed for IPv6 to store 1 million routes, only 128 MiB are needed for IPv4. The difference is mainly due to the size of struct rt6_info (336 bytes) compared to the size of IPv4 s struct fib_alias (48 bytes): IPv4 puts most information about next-hops in struct fib_info structures that are shared with many entries.

Conclusion The takeaways from this article are:

• upgrade to Linux 4.2 or more recent to avoid excessive caching,
• route lookups are noticeably slower compared to IPv4 (by an order of magnitude),
• CONFIG_IPV6_MULTIPLE_TABLES option incurs a fixed penalty of 100 ns by lookup,
• memory usage is fair (20 MiB for 40,000 routes).

Compared to IPv4, IPv6 in Linux doesn t foster the same interest, notably in term of optimizations. Hopefully, things are changing as its adoption and use at scale are increasing.

---

1. For a given destination prefix, it s possible to attach source-specific prefixes:

   ip -6 route add 2001:db8:1::/64 \
     from 2001:db8:3::/64 \
     via fe80::1 \
     dev eth0
   

   Lookup is first done on the destination address, then on the source address.
2. This is quite different of the classic scenario where Linux acts as a gateway for a /64 subnet. In this case, the neighbor subsystem stores the reachability information for each host and the routing table only contains a single /64 prefix.
3. The measurements are done in a virtual machine with one vCPU and no neighbors. The host is an Intel Core i5-4670K running at 3.7 GHz during the experiment (CPU governor set to performance). The benchmark is single-threaded. Many lookups are performed and the result reported is the median value. Timings of individual runs are computed from the TSC.
4. Most of the packets in the network are expected to be routed to a destination. However, this also means the backtracking code path is not used in the /128 and /48 scenarios. Having a fallback route gives far different results and make it difficult to ensure we explore the address space correctly.
5. The exact same optimizations could be applied for IPv6. Nobody did it yet.
6. Compiling out table support effectively removes those last 100 ns.
7. There is also per-CPU pointers allocated directly (4 bytes per entry per CPU on a 64-bit architecture). We ignore this detail.

$ strip --strip-debug /usr/local/lib/R/site-library/*/libs/*.so

$ apt-get update
$ apt-get install r-base libcurl4-openssl-dev libssl-dev libxml2-dev
$ apt-get install mg          # a tiny editor
$ mg /etc/R/Rprofile.site     # to add a default CRAN repo
$ R -e 'install.packages("tidyverse")'

root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
4.3M    /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
2.3M    /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
144K    /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
204K    /usr/local/lib/R/site-library/curl/libs/curl.so
328K    /usr/local/lib/R/site-library/digest/libs/digest.so
33M     /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
36K     /usr/local/lib/R/site-library/glue/libs/glue.so
3.2M    /usr/local/lib/R/site-library/haven/libs/haven.so
272K    /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
52K     /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
64K     /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
16K     /usr/local/lib/R/site-library/mime/libs/mime.so
124K    /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
372K    /usr/local/lib/R/site-library/openssl/libs/openssl.so
772K    /usr/local/lib/R/site-library/plyr/libs/plyr.so
92K     /usr/local/lib/R/site-library/purrr/libs/purrr.so
13M     /usr/local/lib/R/site-library/readr/libs/readr.so
4.7M    /usr/local/lib/R/site-library/readxl/libs/readxl.so
1.2M    /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
160K    /usr/local/lib/R/site-library/rlang/libs/rlang.so
928K    /usr/local/lib/R/site-library/scales/libs/scales.so
4.9M    /usr/local/lib/R/site-library/stringi/libs/stringi.so
1.3M    /usr/local/lib/R/site-library/tibble/libs/tibble.so
2.0M    /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
1.2M    /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
4.7M    /usr/local/lib/R/site-library/xml2/libs/xml2.so
78M     total
root@de443801b3fc:/# 

root@de443801b3fc:/# strip --strip-debug /usr/local/lib/R/site-library/*/libs/*so
root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
440K    /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
220K    /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
52K     /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
56K     /usr/local/lib/R/site-library/curl/libs/curl.so
120K    /usr/local/lib/R/site-library/digest/libs/digest.so
2.5M    /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
16K     /usr/local/lib/R/site-library/glue/libs/glue.so
404K    /usr/local/lib/R/site-library/haven/libs/haven.so
76K     /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
20K     /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
24K     /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
8.0K    /usr/local/lib/R/site-library/mime/libs/mime.so
52K     /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
84K     /usr/local/lib/R/site-library/openssl/libs/openssl.so
76K     /usr/local/lib/R/site-library/plyr/libs/plyr.so
32K     /usr/local/lib/R/site-library/purrr/libs/purrr.so
648K    /usr/local/lib/R/site-library/readr/libs/readr.so
400K    /usr/local/lib/R/site-library/readxl/libs/readxl.so
128K    /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
56K     /usr/local/lib/R/site-library/rlang/libs/rlang.so
100K    /usr/local/lib/R/site-library/scales/libs/scales.so
496K    /usr/local/lib/R/site-library/stringi/libs/stringi.so
124K    /usr/local/lib/R/site-library/tibble/libs/tibble.so
164K    /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
104K    /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
344K    /usr/local/lib/R/site-library/xml2/libs/xml2.so
6.6M    total
root@de443801b3fc:/#

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

1. Use threading module of python to send the server code in back ground and give it a callback to communicate the data to main thread.
2. Periodically read some messages from server code and then dispose of server.

from socket import socket, AF_INET, SOCK_DGRAM
class BroadCastReceiver:
    def __init__(self, port, msg_len=8192):
        self.sock = socket(AF_INET, SOCK_DGRAM)
        self.sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
        self.sock.bind(('', port))
        self.msg_len = msg_len
    def __iter__(self):
             return self
    def __next__(self):
             try:
                 addr, data = self.sock.recvfrom(self.msg_len)
                 return addr, data
             except Exception as e:
                 print("Got exception trying to recv %s" % e)
                 raise StopIteration

...
    def __init__(self, port, msg_len=8192, timeout=15):
        self.sock = socket(AF_INET, SOCK_DGRAM)
        self.sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
        self.sock.settimeout(timeout)
        self.sock.msg_len = msg_len
        self.sock.bind(('', port))
 ...

...
    def __del__(self):
        self.sock.close()

r = BroadCastReceiver(5000, timeout=10)
count = 0
for (address, data) in r:
    print('Got packet from %s: %s' % address, data)
    count += 1
    # do whatever you want with data
    if count > 10:
        break

r = BroadCastReceiver(5000,  timeout=10)
for i in range(20):
    try:
        address, data = next(r)
        # do whatever you want with data
    except:
        break

...
    def __enter__(self):
        return self
    def __exit__(self, exc_type, exc_value, traceback):
        self.sock.close()

with BroadCastReceiver(2000) as r:
    # use server object as you wish
    ...

def __next__(self):
    try:
        addr, data = self.sock.recvfrom(self.msg_len)
        return addr, data
    except Exception as e:
        print("Got exception trying to recv %s" % e)
        raise StopIteration
    finally:
        self.sock.close()

GNU Ring project with GNU organisation

1. Me Before getting into the thick of my project, let me present myself:
I am Olivier Gr goire (Gasuleg), and I study IT engineering at cole de Technologie Sup rieure in Montreal.
I am a technician in electronics, and I study now computing sciences.
I applied to GSoC because I love the concept of the project that I worked on and I really wanted to be part of it.

2. My Project During this GSoC, I worked on the GNU Ring project.
What is ring?

• A telephone: a simple tool to connect, communicate and share.
• A teleconferencing tool: easily join calls to create conferences with multiple participants.
• A media sharing tool: Ring supports a variety of video input options, including mutliple cameras and image and video files, and the selection of audio inputs and outputs; all this is supported by multiple high quality audio and video codecs.
• A messenger: send text messeges during calls or out of calls (as long as your peer is connected).
• A building block for your IoT project: re-use the universal communications technology of Ring with its portable library on your system of choice.
  ring.cx

What I need to do?
This project is, at the moment, unstable due to a lack of automated tests. Only a part of the code is tested. I need to improve this.
To do that, I need to:

• Research and test automation strategies that integrate the compilation system and Jenkins verification like the SIP tests
• Implement some unit tests to check the components
  

See more in my proposal!

3. The Code Links to my work
Here are the links to the code I was working on all throughout the Google Summer of Code:

Patch	Status
Black box testing sip	Merged
New unit test: smartools	Merged
New unit test:account_factory	Merged
New unit test: util classes	On Review
New unit test: archiver, conference, preferences	On Review
New unit test: dring, threadloop	On Review
Documentation	On Review
Refactoring + video_input unit test	Abandoned

How to use the code?
Follow the instructions to build the daemon and instead of doing make , do make check . You will see something composed of:

• Test automation strategies
  
  
• Unit test
  

Some explanations of the code
The tests are based on CPPunit framework.
I created some Black box testing to do some calls using SIP protocol. To do that, I created scenarios using SIPp .
I also create a lot of unit tests.
Difficulties encountered
My project was a little bit difficult to setup because I never touch autotools before that.
The sip tests were not really difficult to create. SIPp is really easy to use.
The unit test part was more tricky. It s really easier to do the tests before coding like the Test-driven_development said. Here the code is already written and I need to test it. So, I needed to create tests based on the code and not what he needed to do. Also, I got some issues with classes who were really linked together so I couldn t make unit test with it.

4. What s next?

• Merge all the patchs
• Add gcov to know the code coverage
• Increase the code coverage

5. Thanks I would like to thank the following:

• The Google Summer of Code organisation, for this wonderful experience.
  
• GNU, for accepting my project proposal and letting me embark on this fantastic adventure.
  
• My mentor, Mr Guillaume Roguez, and all the ring team, for being there to help me.

• the Debian wiki page for Suricata
• the official Suricata upstream docs

% sudo aptitude install suricata

% sudo systemctl status suricata
  suricata.service - Suricata IDS/IDP daemon
   Loaded: loaded (/lib/systemd/system/suricata.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-08-19 12:50:49 CEST; 44min ago
     Docs: man:suricata(8)
           man:suricatasc(8)
           https://redmine.openinfosecfoundation.org/projects/suricata/wiki
 Main PID: 1101 (Suricata-Main)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/suricata.service
            1101 /usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid
ago 19 12:50:44 nostromo systemd[1]: Starting Suricata IDS/IDP daemon...
ago 19 12:50:47 nostromo suricata[1032]: 19/8/2017 -- 12:50:47 - <Notice> - This is Suricata version 4.0.0 RELEASE
ago 19 12:50:49 nostromo systemd[1]: Started Suricata IDS/IDP daemon.

% sudo suricatasc -c uptime
 "message": 3892, "return": "OK" 

[...]
# Linux high speed capture support
af-packet:
  - interface: wlan0
[...]

% sudo systemctl restart suricata
% sudo tail -f /var/log/suricata/fast.log
08/19/2017-14:03:04.025898  [**] [1:2012648:3] ET POLICY Dropbox Client Broadcasting [**] \
	[Classification: Potential Corporate Privacy Violation] [Priority: 1]  UDP  192.168.1.36:17500 -> 255.255.255.255:17500

% sudo aptitude install suricata-oinkmaster

• a 10 point drop in your IQ; or
• drinking a glass of wine.

• Antoine Beaupr did 20h (out of 16h allocated + 4 extra hours).
• Ben Hutchings did 14 hours (out of 15h allocated, thus keeping 1 extra hour for August).
• Chris Lamb did 18 hours.
• Emilio Pozuelo Monfort did 18.5 hours (out of 23.5 hours allocated + 8 hours remaining, thus keeping 13 hours for August).
• Guido G nther did 10 hours.
• Hugo Lefeuvre did nothing due to personal problems (out of 2h allocated + 10 extra hours, thus keeping 12 extra hours for August).
• Markus Koschany did 23.5 hours.
• Ola Lundqvist did not publish his report yet (out of 14h allocated + 2 extra hours).
• Rapha l Hertzog did 7 hours (out of 12 hours allocated but he gave back his remaining hours).
• Roberto C. Sanchez did 19.5 hours (out of 23.5 hours allocated + 12 hours remaining, thus keeping 16 extra hours for August).
• Thorsten Alteholz did 23.5 hours.

• Platinum sponsors:
• TOSHIBA (for 22 months)
• GitHub (for 13 months)
• Gold sponsors:
• The Positive Internet (for 38 months)
• Blablacar (for 37 months)
• Linode (for 27 months)
• Babiel GmbH (for 16 months)
• Plat Home (for 16 months)
• Silver sponsors:
• Domeneshop AS (for 37 months)
• Universit Lille 3 (for 37 months)
• Trollweb Solutions (for 35 months)
• Nantes M tropole (for 32 months)
• Dalenys (for 28 months)
• Univention GmbH (for 23 months)
• Universit Jean Monnet de St Etienne (for 23 months)
• Sonus Networks (for 17 months)
• UR Communications BV (for 12 months)
• maxcluster GmbH (for 11 months)
• Exonet B.V. (for 7 months)
• Leibniz Rechenzentrum
• Bronze sponsors:
• David Ayers IntarS Austria (for 38 months)
• Evolix (for 38 months)
• Offensive Security (for 38 months)
• Seznam.cz, a.s. (for 38 months)
• Freeside Internet Service (for 37 months)
• MyTux (for 37 months)
• Intevation GmbH (for 35 months)
• Linuxhotel GmbH (for 35 months)
• Daevel SARL (for 33 months)
• Bitfolk LTD (for 32 months)
• Megaspace Internet Services GmbH (for 32 months)
• NUMLOG (for 32 months)
• Greenbone Networks GmbH (for 31 months)
• WinGo AG (for 31 months)
• Ecole Centrale de Nantes LHEEA (for 27 months)
• Sig-I/O (for 24 months)
• Entr ouvert (for 22 months)
• Adfinis SyGroup AG (for 19 months)
• GNI MEDIA (for 14 months)
• Quarantainenet BV (for 14 months)
• RHX Srl (for 11 months)
• Bearstech (for 5 months)
• LiHAS (for 5 months)
• People Doc
• Catalyst IT Ltd

No comment Liked this article? Click here. My blog is Flattr-enabled.

• Upstream: https://github.com/mate-desktop
• Debian Packaging: https://qa.debian.org/developer.php?login=pkg-mate-team@lists.alioth.deb...
• Ubuntu MATE: http://ubuntu-mate.org/
• Git Packaging Repositories: https://anonscm.debian.org/cgit/pkg-mate/
• IRC (upstream): #mate-dev on Freenode (irc.freenode.net) # IRC (Debian + Ubuntu): #debian-mate on OFTC (irc.debian.org)

• Improve the MATE User Guide, I am quite optimistic that we get help from the person that asked for a better help tool on the MATE Desktop (like it was back in the times of old GNOMEv2)
• Provide several desktop layouts in Debian, like available in Ubuntu MATE, that can be chosen / configured by the MATE Tweak Tool
• Port the Ubuntu MATE Welcome application to Debian and autostart it on first login, like people know it from Ubuntu MATE

Changes in RcppArmadillo version 0.7.960.1.0 (2017-08-11)

• Upgraded to Armadillo release 7.960.1 (Northern Banana Republic Deluxe)
  • faster randn() when using OpenMP (NB: usually omitted when used fromR)
  • faster gmm_diag class, for Gaussian mixture models with diagonal covariance matrices
  • added .sum_log_p() to the gmm_diag class
  • added gmm_full class, for Gaussian mixture models with full covariance matrices
  • expanded .each_slice() to optionally use OpenMP for multi-threaded execution
• Upgraded to Armadillo release 7.950.0 (Northern Banana Republic)
  • expanded accu() and sum() to use OpenMP for processing expressions with computationally expensive element-wise functions
  • expanded trimatu() and trimatl() to allow specification of the diagonal which delineates the boundary of the triangular part
• Enhanced support for sparse matrices (Binxiang Ni as part of Google Summer of Code 2017)
  • Add support for dtCMatrix and dsCMatrix (#135)
  • Add conversion and unit tests for dgT, dtT and dsTMatrix (#136)
  • Add conversion and unit tests for dgR, dtR and dsRMatrix (#139)
  • Add conversion and unit tests for pMatrix and ddiMatrix (#140)
  • Rewrite conversion for dgT, dtT and dsTMatrix, and add file-based tests (#142)
  • Add conversion and unit tests for indMatrix (#144)
  • Rewrite conversion for ddiMatrix (#145)
  • Add a warning message for matrices that cannot be converted (#147)
  • Add new vignette for sparse matrix support (#152; Dirk in #153)
  • Add support for sparse matrix conversion from Python SciPy (#158 addressing #141)
• Optional return of row or column vectors in collapsed form if appropriate #define is set (Serguei Sokol in #151 and #154)
• Correct speye() for non-symmetric cases (Qiang Kou in #150 closing #149).
• Ensure tests using Scientific Python and reticulate are properly conditioned on the packages being present.
• Added .aspell/ directory with small local directory now supported by R-devel.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• released version 4.0.1.0 of the Policy Manual
• figured out documenting reproducibility in policy. Formulating the wording turned out to be easier than I had expected
• approx. ten years after they were first published, incorporated marga s maintscript flowcharts into policy proper
• converted policy from docbook to rST built with the Sphinx toolchain. Many, many thanks to Hideki Yamane and David Bremner for helping Russ and I get this merged to our master branch
• triage of every single bug against policy, and mass closure of inactive bugs, bringing the total down from more than 200 to around 125
• conversations with Technical Committee members about how the two teams can help each other s work (mainly us helping them to help us!)
• conversations about how we handle disagreement and plans to streamline our overly complex BTS usertags (watch this space)
• very useful input from policy consumers about how the upgrading checklist is formatted, and how we can recruit more people to get patches written

• plans to finally drop our emacsXY binary packages, and just have a single version of Emacs in the archive, so that we no longer have to deal with bugs due to someone still having emacs21 installed (David s idea; Rob s implementation; Sean s mostly-helpful comments)
• other plans to simplify and otherwise improve the Debian Emacsen policy
• finally finished off the work needed to RM emacs24 nine months later including a lot of NMUs
• mentoring a junior team member

• There were two talks mainly about Reproducible Builds at DebConf17:
  • Reproducible builds: Status update by Chris Lamb, Holger Levsen, Maria Glukhova, Steven Chamberlain, Vagrant Cascadian, Valerie Young and Ximin Luo.
  • Fun with .buildinfo by Steven Chamberlain.
• Reproducible builds were also mentioned in many other talks including:
  • Using Qubes OS from the POV of a Debian developer by Holger Levsen.
  • Debian for Medical Software by Andr Roth.
  • Let's maintain jenkins.debian.org as a team by Holger Levsen.
  • Signing package contents: why and how by Matthew Garrett.
  • Bits from the DPL by Chris Lamb.
  • Installing Debian by Vagrant Cascadian.
  • in-toto -- Securing supply chains as a whole by Lukas Puehringer.
  • Debian Cloud BoF by Steve McIntyre.
  • Will there be Debian in your next BMW car? by Aigars Mahinovs.
• There has also been substantial activity regarding updating the Debian Policy regarding Reproducible Builds (#844431). This work is still ongoing and will be covered in future issues of this weekly blog.

• Adrian Bunk:
  • #871818 filed against debian-zh-faq.
  • #871907 filed against praat.
• Katsuhiko Nishimra:
  • #871591 filed against llvm-toolchain-snapshot.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
  • #871345 filed against texlive-base.
  • #871349 filed against ispell-uk.
  • #871357 filed against po4a.
• jathan:
  • #870890 filed against apg.

• Bernhard M. Wiedemann:
  • enblend (merged), SOURCE_DATE_EPOCH support
  • systemtap
  • splint (dead project)
  • skelcd-openSUSE

• During our reproducibility testing, Adrian Bunk filed 48 FTBFS bugs this week.

• Mattia Rizzolo:
  • debian/copyright: coalesce some file paragraphs and update information.
  • Bump Standards-Version to 4.0.1, no changes needed.

• Chris Lamb:
  • Test --help output in autopkgtests

• Mattia:
  • Notify the#debian-reproducible-changes IRC channel for unreproducible -> FTBFS transitions.
  • Update squid.conf for all nodes to 5.2.23 (and fixup some).
  • Enable the Munin Squid plugin on the Codethink arm64 nodes as well.
  • Force reconfiguration of Apache and Munin when update_jdn.sh is updated.
• Holger worked on slides for his DebConf17 BoF about migrating to jenkins.debian.org, which affects tests.r-b.o as well.

• Arbitrary installation attacks. We protect against that with the outer signature and hashes
• Endless data attacks. Yes, we impose a limit on Release files (the sizes of other files are specified in there and this file is signed)
• Extraneous dependencies attacks That s verified by the signed hashes of Packages files
• Fast-forward attacks same
• Indefinite freeze attacks APT has a Valid-Until field that can be used to specify a maximum life time of a release file
• Malicious mirrors preventing updates. Well, the user configures the mirror, so usually not applicable. if the user has multiple mirrors, APT deals with that fine
• Mix-and-match attacks Again, signed Release file and hashes of other files
• Rollback attacks We do not allow Date fields in Release files to go backwards
• Slow retrieval attacks TUF cannot protect against that either. APT has very high timeouts, and there is no reasonable answer to that.
• Vulnerability to key compromises For our purposes where we need all repository signing keys to be online, as we need to sign new releases and metadata fairly often, it does not make it less vulnerable to require a threshold of keys (APT allows repositories to specify concrete key ids they may be signed with though, that has the same effect)
• Wrong software installation. Does not happen, the .deb files are hashed in the Packages files which are signed by the release file