Spritz seems like a very interesting way to read quickly. It's the opposite of everything I've read (slowly) about speed reading, which focuses on using peripheral vision and not reading word-by-word. You're supposed to do things like move your eyes straight down the page, taking in whole lines at a time. Interruptions seem like a big problem; interruptions that make me look away, or interruptions in my brain, where I might realize I've not been paying attention for some amount of time. Maybe they should have navigation buttons similar to video players, so you can skip backward 15 seconds at a time. I also do want to go back and review previous pages sometimes for reasons that have nothing to do with interruption, so I wouldn't want word-by-word to be the only way to view a text -- especially when reading nonfiction. I might event want it to work in a mode where you hold down the button on the side of your phone or tablet in order to move the words, and then have them automatically pause when you release. It feels like I'd want a lot of short breaks when reading in this style. It should also be free software, but unfortunately I'm guessing it won't be. I hope someone will make a free software application along these lines -- the basics seem pretty basic.

So it may sound normal to receive compressed response for CSS/JS file from Apache2 but what I faced is completely different behavior, here I saw not just body of response but also header are getting compressed and send to browser and browser was unable to interpret the response there by leading page rendered without CSS and JS. That was long story short, so let me explain in detail the case and what I found out. I have hosted SILPA on my VPS using uWSGI. I had used till date using libapache2-mod-proxy-uwsgi plugin for which uWSGI should be using network socket and in Apache2 config I need ProxyPass directive pointing to uwsgi://host:port and everything used to work out of box. There is a drawback for using network socket in uWSGI that is limited amount of ports (65535) and possible clash with other services if we use wrong port number, additionally network sockets will be slower compared to the file socket so yesterday I thought of changing it to file socket and removed socket directive in the ini file for uWSGI application container for SILPA and here is where all problems started. My updated configuration can be seen in our documentation page that is if you are interested to look my configuration file :-).

TL;DR: Do not enable net.ipv4.tcp_tw_recycle. The Linux kernel documentation is not very helpful about what net.ipv4.tcp_tw_recycle does:

Enable fast recycling TIME-WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts.

Its sibling, net.ipv4.tcp_tw_reuse is a little bit more documented but the language is about the same:

Allow to reuse TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. Default value is 0. It should not be changed without advice/request of technical experts.

The mere result of this lack of documentation is that we find numerous tuning guides advising to set both these settings to 1 to reduce the number of entries in the TIME-WAIT state. However, as stated by tcp(7) manual page, the net.ipv4.tcp_tw_recycle option is quite problematic for public-facing servers as it won t handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you:

Enable fast recycling of TIME-WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).

I will provide here a more detailed explanation in the hope to teach people who are wrong on the Internet. As a sidenote, despite the use of ipv4 in its name, the net.ipv4.tcp_tw_recycle control also applies to IPv6. Also, keep in mind we are looking at the TCP stack of Linux. This is completely unrelated to Netfilter connection tracking which may be tweaked in other ways¹.

About TIME-WAIT state Let s rewind a bit and have a close look at this TIME-WAIT state. What is it? See the TCP state diagram below²: Only the end closing the connection first will reach the TIME-WAIT state. The other end will follow a path which usually permits to quickly get rid of the connection. You can have a look at the current state of connections with ss -tan:

$ ss -tan   head -5
LISTEN     0  511             *:80              *:*     
SYN-RECV   0  0     192.0.2.145:80    203.0.113.5:35449
SYN-RECV   0  0     192.0.2.145:80   203.0.113.27:53599
ESTAB      0  0     192.0.2.145:80   203.0.113.27:33605
TIME-WAIT  0  0     192.0.2.145:80   203.0.113.47:50685

Purpose There are two purposes for the TIME-WAIT state:

• The most known one is to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows. RFC 1337 explains in details what happens when the TIME-WAIT state is deficient³. Here is an example of what could be avoided if the TIME-WAIT state wasn t shortened:

• The other purpose is to ensure the remote end has closed the connection. When the last ACK is lost, the remote end stays in the LAST-ACK state⁴. Without the TIME-WAIT state, a connection could be reopened while the remote end still thinks the previous connection is valid. When it receives a SYN segment (and the sequence number matches), it will answer with a RST as it is not expecting such a segment. The new connection will be aborted with an error:

RFC 793 requires the TIME-WAIT state to last twice the time of the MSL. On Linux, this duration is not tunable and is defined in include/net/tcp.h as one minute:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
                                  * state, about 60 seconds     */

There have been propositions to turn this into a tunable value but it has been refused on the ground the TIME-WAIT state is a good thing.

Problems Now, let s see why this state can be annoying on a server handling a lot of connections. There are three aspects of the problem:

• the slot taken in the connection table preventing new connections of the same kind,
• the memory occupied by the socket structure in the kernel, and
• the additional CPU usage.

The result of ss -tan state time-wait wc -l is not a problem per se!

Connection table slot A connection in the TIME-WAIT state is kept for one minute in the connection table. This means, another connection with the same quadruplet (source address, source port, destination address, destination port) cannot exist. For a web server, the destination address and the destination port are likely to be constant. If your web server is behind a L7 load-balancer, the source address will also be constant. On Linux, the client port is by default allocated in a port range of about 30,000 ports (this can be changed by tuning net.ipv4.ip_local_port_range). This means that only 30,000 connections can be established between the web server and the load-balancer every minute, so about 500 connections per second. If the TIME-WAIT sockets are on the client side, such a situation is easy to detect. The call to connect() will return EADDRNOTAVAIL and the application will log some error message about that. On the server side, this is more complex as there is no log and no counter to rely on. In doubt, you should just try to come with something sensible to list the number of used quadruplets:

$ ss -tan 'sport = :80'   awk ' print $(NF)" "$(NF-1) '   \
>     sed 's/:[^ ]*//g'   sort   uniq -c
    696 10.24.2.30 10.33.1.64
   1881 10.24.2.30 10.33.1.65
   5314 10.24.2.30 10.33.1.66
   5293 10.24.2.30 10.33.1.67
   3387 10.24.2.30 10.33.1.68
   2663 10.24.2.30 10.33.1.69
   1129 10.24.2.30 10.33.1.70
  10536 10.24.2.30 10.33.1.73

The solution is more quadruplets⁵. This can be done in several ways (in the order of difficulty to setup):

• use more client ports by setting net.ipv4.ip_local_port_range to a wider range,
• use more server ports by asking the web server to listen to several additional ports (81, 82, 83, ),
• use more client IP by configuring additional IP on the load balancer and use them in a round-robin fashion,
• use more server IP by configuring additional IP on the web server⁶.

Of course, a last solution is to tweak net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle. Don t do that yet, we will cover those settings later.

Memory With many connections to handle, leaving a socket open for one additional minute may cost your server some memory. For example, if you want to handle about 10,000 new connections per second, you will have about 600,000 sockets in the TIME-WAIT state. How much memory does it represent? Not that much! First, from the application point of view, a TIME-WAIT socket does not consume any memory: the socket has been closed. In the kernel, a TIME-WAIT socket is present in three structures (for three different purposes):

1. A hash table of connections, named the TCP established hash table (despite containing connections in other states) is used to locate an existing connection, for example when receiving a new segment. Each bucket of this hash table contains both a list of connections in the TIME-WAIT state and a list of regular active connections. The size of the hash table depends on the system memory and is printed at boot:

   $ dmesg   grep "TCP established hash table"
   [    0.169348] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
   

   It is possible to override it by specifying the number of entries on the kernel command line with the thash_entries parameter. Each element of the list of connections in the TIME-WAIT state is a struct tcp_timewait_sock, while the type for other states is struct tcp_sock⁷:

   struct tcp_timewait_sock  
       struct inet_timewait_sock tw_sk;
       u32    tw_rcv_nxt;
       u32    tw_snd_nxt;
       u32    tw_rcv_wnd;
       u32    tw_ts_offset;
       u32    tw_ts_recent;
       long   tw_ts_recent_stamp;
    ;
   struct inet_timewait_sock  
       struct sock_common  __tw_common;
       int                     tw_timeout;
       volatile unsigned char  tw_substate;
       unsigned char           tw_rcv_wscale;
       __be16 tw_sport;
       unsigned int tw_ipv6only     : 1,
                    tw_transparent  : 1,
                    tw_pad          : 6,
                    tw_tos          : 8,
                    tw_ipv6_offset  : 16;
       unsigned long            tw_ttd;
       struct inet_bind_bucket *tw_tb;
       struct hlist_node        tw_death_node;
    ;
   

2. A set of lists of connections, called the death row , is used to expire the connections in the TIME-WAIT state. They are ordered by how much time left before expiration. It uses the same memory space as for the entries in the hash table of connections. This is the struct hlist_node tw_death_node member of struct inet_timewait_sock.
3. A hash table of bound ports, holding the locally bound ports and the associated parameters, is used to determine if it is safe to listen to a given port or to find a free port in the case of dynamic bind. The size of this hash table is the same as the size of the hash table of connections:

   $ dmesg   grep "TCP bind hash table"
   [    0.169962] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
   

   Each element is a struct inet_bind_socket. There is one element for each locally bound port. A TIME-WAIT connection to a web server is locally bound to the port 80 and shares the same entry as its sibling TIME-WAIT connections. On the other hand, a connection to a remote service is locally bound to some random port and does not share its entry.

So, we are only concerned by the space occupied by struct tcp_timewait_sock and struct inet_bind_socket. There is one struct tcp_timewait_sock for each connection in the TIME-WAIT state, inbound or outbound. There is one dedicated struct inet_bind_socket for each outbound connection and none for an inbound connection. A struct tcp_timewait_sock is only 168 bytes while a struct inet_bind_socket is 48 bytes:

$ sudo apt-get install linux-image-$(uname -r)-dbg
[...]
$ gdb /usr/lib/debug/boot/vmlinux-$(uname -r)
(gdb) print sizeof(struct tcp_timewait_sock)
 $1 = 168
(gdb) print sizeof(struct tcp_sock)
 $2 = 1776
(gdb) print sizeof(struct inet_bind_bucket)
 $3 = 48

So, if you have about 40,000 inbound connections in the TIME-WAIT state, it should eat less than 10MB of memory. If you have about 40,000 outbound connections in the TIME-WAIT state, you need to account for 2.5MB of additional memory. Let s check that by looking at the output of slabtop. Here is the result on a server with about 50,000 connections in the TIME-WAIT state, 45,000 of which are outbound connections:

$ sudo slabtop -o   grep -E '(^  OBJS tw_sock_TCP tcp_bind_bucket)'
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
 50955  49725  97%    0.25K   3397       15     13588K tw_sock_TCP            
 44840  36556  81%    0.06K    760       59      3040K tcp_bind_bucket

There is nothing to change here: the memory used by TIME-WAIT connections is really small. If your server need to handle thousands of new connections per second, you need far more memory to be able to efficiently push data to clients. The overhead of TIME-WAIT connections is negligible.

CPU On the CPU side, searching for a free local port can be a bit expensive. The work is done by the inet_csk_get_port() function which uses a lock and iterate on locally bound ports until a free port is found. A large number of entries in this hash table is usually not a problem if you have a lot of outbound connections in the TIME-WAIT state (like ephemeral connections to a memcached server): the connections usually share the same profile, the function will quickly find a free port as it iterates on them sequentially.

Other solutions If you still think you have a problem with TIME-WAIT connections after reading the previous section, there are three additional solutions to solve them:

• disable socket lingering,
• net.ipv4.tcp_tw_reuse, and
• net.ipv4.tcp_tw_recycle.

Socket lingering When close() is called, any remaining data in the kernel buffers will be sent in the background and the socket will eventually transition to the TIME-WAIT state. The application can continue to work immediatly and assume that all data will eventually be safely delivered. However, an application can choose to disable this behaviour, known as socket lingering. There are two flavors:

1. In the first one, any remaining data will be discarded and instead of closing the connection with the normal four-packet connection termination sequence, the connection will be closed with a RST (and therefore, the peer will detect an error) and will be immediatly destroyed. No TIME-WAIT state in this case.
2. With the second flavor, if there is any data still remaining in the socket send buffer, the process will sleep when calling close() until either all the data is sent and acknowledged by the peer or the configured linger timer expires. It is possible for a process to not sleep by setting the socket as non-blocking. In this case, the same process happens in the background. It permits the remaining data to be sent during a configured timeout but if the data is succesfully sent, the normal close sequence is run and you get a TIME-WAIT state. And on the other case, you ll get the connection close with a RST and the remaining data is discarded.

In both cases, disabling socket lingering is not a one-size-fits-all solution. It may be used by some applications like HAProxy or Nginx when it is safe to use from the upper protocol point of view. There are good reasons to not disable it unconditionnaly.

net.ipv4.tcp_tw_reuse The TIME-WAIT state prevents delayed segments to be accepted in an unrelated connection. However, on certain conditions, it is possible to assume a new connection s segment cannot be misinterpreted with an old connection s segment. RFC 1323 presents a set of TCP extensions to improve performance over high-bandwidth paths. Among other things, it defines a new TCP option carrying two four-byte timestamp fields. The first one is the current value of the timestamp clock of the TCP sending the option while the second one is the most recent timestamp received from the remote host. By enabling net.ipv4.tcp_tw_reuse, Linux will reuse an existing connection in the TIME-WAIT state for a new outgoing connection if the new timestamp is strictly bigger than the most recent timestamp recorded for the previous connection: an outgoing connection in the TIME-WAIT state can be reused after just one second. How is it safe? The first purpose of the TIME-WAIT state was to avoid duplicate segments to be accepted in an unrelated connection. Thanks to the use of timestamps, such a duplicate segments will come with an outdated timestamp and therefore be discarded. The second purpose was to ensure the remote end is not in the LAST-ACK state because of the lost of the last ACK. The remote end will retransmit the FIN segment until:

1. it gives up (and tear down the connection), or
2. it receives the ACK it is waiting (and tear down the connection), or
3. it receives a RST (and tear down the connection).

If the FIN segments are received in a timely manner, the local end socket will still be in the TIME-WAIT state and the expected ACK segments will be sent. Once a new connection replaces the TIME-WAIT entry, the SYN segment of the new connection is ignored (thanks to the timestamps) and won t be answered by a RST but only by a retransmission of the FIN segment. The FIN segment will then be answered with a RST (because the local connection is in the SYN-SENT state) which will allow the transition out of the LAST-ACK state. The initial SYN segment will eventually be resent (after one second) because there was no answer and the connection will be established without apparent error, except a slight delay: It should be noted that when a connection is reused, the TWRecycled counter is increased (despite its name).

net.ipv4.tcp_tw_recycle This mechanism also relies on the timestamp option but affects both incoming and outgoing connections which is handy when the server usually closes the connection first⁸. The TIME-WAIT state is scheduled to expire sooner: it will be removed after the retransmission timeout (RTO) interval which is computed from the RTT and its variance. You can spot the appropriate values for a living connection with the ss command:

$ ss --info  sport = :2112 dport = :4057
State      Recv-Q Send-Q    Local Address:Port        Peer Address:Port   
ESTAB      0      1831936   10.47.0.113:2112          10.65.1.42:4057    
         cubic wscale:7,7 rto:564 rtt:352.5/4 ato:40 cwnd:386 ssthresh:200 send 4.5Mbps rcv_space:5792

To keep the same guarantees the TIME-WAIT state was providing, while reducing the expiration timer, when a connection enters the TIME-WAIT state, the latest timestamp is remembered in a dedicated structure containing various metrics for previous known destinations. Then, Linux will drop any segment from the remote host whose timestamp is not strictly bigger than the latest recorded timestamp, unless the TIME-WAIT state would have expired:

if (tmp_opt.saw_tstamp &&
    tcp_death_row.sysctl_tw_recycle &&
    (dst = inet_csk_route_req(sk, &fl4, req, want_cookie)) != NULL &&
    fl4.daddr == saddr &&
    (peer = rt_get_peer((struct rtable *)dst, fl4.daddr)) != NULL)  
        inet_peer_refcheck(peer);
        if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
            (s32)(peer->tcp_ts - req->ts_recent) >
                                        TCP_PAWS_WINDOW)  
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
         
 

When the remote host is in fact a NAT device, the condition on timestamps will forbid allof the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems. The LAST-ACK state is handled in the exact same way as for net.ipv4.tcp_tw_recycle.

Summary The universal solution is to increase the number of possible quadruplets by using, for example, more server ports. This will allow you to not exhaust the possible connections with TIME-WAIT entries. On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are pretty sure you will never have NAT devices in the mix. Enabling net.ipv4.tcp_tw_reuse is useless for incoming connections. On the client side, enabling net.ipv4.tcp_tw_reuse is another almost-safe solution. Enabling net.ipv4.tcp_tw_recycle in addition to net.ipv4.tcp_tw_reuse is mostly useless. And a final quote by W. Richard Stevens, in Unix Network Programming:

The TIME_WAIT state is our friend and is there to help us (i.e., to let old duplicate segments expire in the network). Instead of trying to avoid the state, we should understand it.

---

1. Notably, fiddling with net.netfilter.nf_conntrack_tcp_timeout_time_wait won t change anything on how the TCP stack will handle the TIME-WAIT state.
2. This diagram is licensed under the LaTeX Project Public License 1.3. The original file is available on this page.
3. The first work-around proposed in RFC 1337 is to ignore RST segments in the TIME-WAIT state. This behaviour is controlled by net.ipv4.rfc1337 which is not enabled by default on Linux because this is not a complete solution to the problem described in the RFC.
4. While in the LAST-ACK state, a connection will retransmit the last FIN segment until it gets the expected ACK segment. Therfore, it is unlikely we stay long in this state.
5. On the client side, older kernels also have to find a free local tuple (source address and source port) for each outgoing connection. Increasing the number of server ports or IP won t help in this case. Linux 3.2 is recent enough to be able to share the same local tuple for different destinations. Thanks to Willy Tarreau for his insight on this aspect.
6. This last solution may seem a bit dumb since you could just use more ports but some servers are not able to be configured this way. The before last solution can also be quite cumbersome to setup, depending on the load-balancing software, but uses less IP than the last solution.
7. The use of a dedicated memory structure for sockets in the TIME-WAIT is here since Linux 2.6.14. The struct sock_common structure is a bit more verbose and I won t copy it here.
8. When the server closes the connection first, it gets the TIME-WAIT state while the client will consider the corresponding quadruplet free and hence may reuse it for a new connection.

Instances of the for those who care about X snowclone on Debian mailing lists:

• their packages in Ubuntu (the original!)
• lesbians
• Andrew Suffield
• debian-devel-announce
• their packages in klik
• the [GFDL] GR
• stable updates
• bits from [press and events] teams
• glib applications
• experimental
• Debconf
• Policy 7.6
• Switzerland/Liechtenstein
• X
• firmware
• bts-link
• lesbians (reloaded)
• pam-ssh
• Mercurial
• manpages
• their packages in Debian
• unbound
• irker
• [team metrics] statistics

Process Identifiers (PIDs) are a scarce resource. On Linux they are only 15 bits by default. The Process Identifier Preservation Society (PIPS) aims to reduce abuse and wastage of the PID space. To join the society please read the following advice.

Common issues Several languages generally allow you to run all your code in one process. Some of the code that you might want to incorporate or use is not available in the form of ELF libraries or language specific libraries but only in the form of ELF binaries or interpreted scripts. As a result using additional PID space is sometimes unavoidable. Many languages have multiple methods of starting external processes and usually some of them waste PID space by running commands in a shell. You can avoid those methods or use the exec builtin to preserve the shell PID. It might be tempting to explicitly use shell in languages that don't allow implicit shell use but that just wastes extra PIDs. Several languages allow you to fork one process into two, this uses an extra PID and is to be avoided unless necessary.

Shell Programs written in the shell languages use a lot of PIDs. Even shells that have a lot of shell builtins (like busybox sh) appear to use the PID space by forking a child process. To join PIPS you should just stop writing programs in shell or use as many builtins as possible and use exec to preserve PIDs.

Init Several init systems are written in or use shell extensively and thus eat huge bowls of PIDs for breakfast. To join the PIPS you should switch away from sysvinit, openrc, init=/bin/sh etc.

C/C++ The common issues section applies to the C/C++ language. To join PIPS you should rewrite your code to use fork()+exec(), g_spawn*() or libpipeline instead of the g_spawn_command_line*(), system() and popen() functions.

Perl The common issues section applies to the Perl language. To join PIPS you should rewrite your code to avoid backticks and only pass arrays to the system(), open(), open2(), open3() functions.

PHP The common issues section applies to the PHP language. To join PIPS you should rewrite your code to use pcntl_exec() instead of backticks, exec(), system(), passthru(), shell_exec(), popen() and proc_open(). Apparently pcntl_exec() is disabled by default on Debian and pcntl_* are often unavailable so you should just not spawn processes. You could also just drop PHP already.

Python The common issues section applies to the Python language. To join PIPS you should rewrite your code to use the subprocess module and avoid passing shell=True to the subprocess.Popen() function. The os.system(), os.popen() functions and the commands module all run their commands in a shell, wasting PID space. The popen2 module requires passing arrays instead of strings in order to avoid the command being run in a shell.

Java The common issues section applies to the Java language. To join PIPS you should rewrite your code to use ProcessBuilder or only pass arrays of strings to Runtime.getRuntime().exec().

Haskell The common issues section applies to the Haskell language. To join PIPS you should only ever pass a RawCommand to createProcess and never use the shell, system, runCommand or runInteractiveCommand functions from the System.Process and System.Cmd libraries.

OCaml The common issues section applies to the OCaml language. To join PIPS you should rewrite your code to use fork+exec or the create_process* wrappers instead of system, open_process, open_process_in, open_process_out and open_process_full.

Go Go allows running external processes but doesn't allow you to waste PID space by running commands in shell. Avoid explicitly running the shell though.

Rust Rust allows running external processes but doesn't allow you to waste PID space by running commands in shell. Avoid explicitly running the shell though.

Erlang The common issues section applies to the Erlang language. To join PIPS you should rewrite your code to use erlang:open_port( spawn_executable, ... , ...) instead of os:cmd or the other options to erlang:open_port.

Node.js The common issues section applies to the Node.js language. To join PIPS you should rewrite your code to use the child_process.execFile() function (or other child_process functions) instead of child_process.exec().

Julia Julia allows running external processes but doesn't allow you to waste PID space by running commands in shell. It emulates a lot of shell features instead. Avoid explicitly running the shell though.

Dart The common issues section applies to the Dart language. To join PIPS simply do not enable the runInShell parameter of the Process object.

PS Let me know if I missed something in one of these languages. You should also do most of the above to avoid shell metacharacter injection attacks that usually allow arbitrary code execution. Dear language authors, don't allow running external processes in shell, kthxbye!

Let s assume you are a sysadmin and have to debug a daemon giving bad performance on one machine, but not on the other. Of course, you did not setup either machine, have only basic knowledge of the said daemon and would really love to watch that awesome piece of cinematographic art with a bunch of friends and a couple of beers. So it s like every day, right? The problem with understanding running setups is that you often have to read configuration files. And when reading one is not enough, you have to compare two or more of them. Suddenly, a wild problem occurs: order and indentation do not matter (unless they do), comments are often just beautiful noise and why the hell did that guy smoke/drink/eat while explicitly setting ALL THE OPTIONS to their defaults before actually setting them as he wanted. If you are using diff(1), you probably love to read a lot of differences, which are none in reality. Want an example? [foo]
bar = bar
foo = foo and # settings for foo
[foo]
# foo is best
foo = foo
# bar is ok here, FIXME?
bar = bar and [foo]
foo = x
bar = x
[foo]
foo = foo
bar = bar are actually the same, at least for some parsers. XTaran suggested using something like wdiff or dwdiff, which often helps, but not in the above case. Others suggested vimdiff, which is nice, but not really helpful here either. As there is a problem, and I love to solve these, I started a small new project: cfgdiff. It tries to parse two given files and give a diff of the content after normalizing it (merging duplicate keys, sorting keys, ignoring comments and blank lines, you name it). Currently it can parse various INI files, JSON, YAML and XML. That s probably not enough to be the single diff tool for configuration files, but it is quite a nice start. And you can extend it, of course ;)

0 comment(s) this blog is flattr enabled

I ve been hacking (on and off) on a small bit of code (written in Hy) called Acid. So, all of this is *really* fluid, and I m going to change it s API and code (it s currently just one massive hack), but I think the idea has kinda jelled enough for me to chat a bit about. Acid is a DSL for writing event-driven Hy. Currently it s just using global events (that ll change soon), and it s a monumental hack (not sure how much I can do there), but it works pretty well. It s built on top of tulip / asyncio, which is an amazing new Python 3.4 standard library package for working with async code. Although asyncio is designed around network-based usecases, I ve been trying to shoehorn in an event system on top of it. It s going well so far. A basic Acid script looks something like:

(trip
  (on :startup
    (every 1 second (emit :clock-pulse nil)))
  (on :clock-pulse (print "."))
  (emit :startup nil))

which will print a dot ( . ) every second forever. Something a bit more advanced (fetch the MBTA Red line T information once a minute forever)

(defn get-endpoint-url [line]
  (.format "http://developer.mbta.com/lib/rthr/ 0 .json"
    (get  :red-line  "red"
          :blue-line "blue"  line)))
(trip
  ;; OK. Let's do some work with MBTA feeds.
  (on :update-feed
      ;; let's just update feeds on a cron.
      (print (.json (.get requests (get-endpoint-url event))))
      (emit :feed-updated event))
  (every 1 minute
         (emit :update-feed :red-line)))

Clearly, both of these examples are just as clear as their straight functional counterparts, but I m keen to see what I can do with temporal recursion for long-running daemons. I think my first job will be porting snitch's codebase to Acid. Let me know if you have any ideas!

Late last week, and just before leaving to participate in this crazy thing, I managed to get a new version of RcppSMC onto CRAN. RcppSMC combines the SMCTC template classes for Sequential Monte Carlo and Particle Filters (Johansen, 2009, JSS) with the Rcpp package for R/C++ Integration (Eddelbuettel and Francois, 2011, JSS) and thereby allows for easier and more direct access from R to the computational core of the Sequential Monte Carlo algorithm. This release regroups a few minor changes we accumulated over the last few months, but was triggered by the Rcpp 0.11.0 release last week. The NEWS entry is below:

Changes in RcppSMC version 0.1.2 (2014-02-06)

• Updated for Rcpp 0.11.0 with explicit importFrom in NAMESPACE and corresponding versioned Imports: in DESCRIPTION; also removed linking instruction from src/Makevars as no it is longer needed with this new Rcpp version
• Addded GitHub / Travis CI support
• Use more portable dev.new() rather than x11() in pfLinearBS.R
• Applied some corrections to pfNonlinBS.R example
• Converted NEWS to NEWS.Rd

Courtesy of CRANberries, there is also a diffstat report relative to the previous release. As always, more detailed information is on the RcppSMC page,

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

The home lab is my LAVA development environment and I ve recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs:

 3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011

As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I m developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy.

Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)

I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image. Manual build steps First, I ve already discovered that the u-boot on the iMX53 doesn t like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I m also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap. I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem:

# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/

Start the debootstrap:

# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian

Various actions in this chroot will need proc, so mount it here:

# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit

You may also have to edit the apt sources the LAVA master image doesn t have editors installed, so either use echo or download an edited file. I m using:

deb http://ftp.uk.debian.org/debian sid main

flash-kernel needs changes For the initial tests, I ve got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch:

# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2

The patch itself goes through a couple of iterations. Initially, it is enough to use:

 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes

Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model):

diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes
 
+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000

(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel:

# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit

LAVA overlays This will be a LAVA test image and it needs an overlay if you want a vanilla image, set up a passwd inside the chroot instead.

# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit

Changes to allow the chroot to boot Now the chroot needs setting up as a boot image:

# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit

Partitioning as a LAVA test image This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I m adapting the image to suit the expectations inside LAVA. Yes, I know but it s better to get something working before spending time fixing things to make it work better. It will be fixed, in time. So I needed to separate out the /boot contents from the rest of the rootfs whilst keeping the chroot itself in a state which I can easily update and use to create other images.

# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*

Now the test image file and its partitions:

# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024

I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier:

# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary

Use the Start numbers and use losetup to create the loop devices for each
partition

# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2

Now clean up the loop mountpoints:

# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a

losetup -a should return nothing. If it doesn t, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image:

# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/

and the / contents (removing the duplicate ./boot contents)

# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root

Check the image

# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img

Downloads Yes, I have put that file online, if you are interested. Do read the readme first though. Getting the image off the device

# ip a
# python -m SimpleHTTPServer 80 2>/dev/null

then wget (just http://, IP address / and the filename), md5sum , finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed
file over a LAN than to let the device compress it. Your main machine
will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can
test the mount offsets more easily and do check that the image
can be mounted with the offsets from parted.) Results

• complete log of the first successful LAVA test of a Debian ARMMP kernel on an iMX53 (153k)
• LAVA submission JSON (only usable with a local URL of the image and a few tweaks to the standard LAVA mx53loco device configuration)
• LAVA result bundle (JSON) (75k)

# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc08
CPU revision	: 5
Hardware	: Freescale i.MX53 (Device Tree Support)
Revision	: 0000
Serial		: 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid

Next! Yes, there is a lot to do from here on.

• The creation of the test image needs to be smoothed out and scripted. I m thinking that this would go into vmdebootstrap. Lars?
• LAVA needs to be less presumptive about the contents and partitioning of the test image.
• the Debian ARMMP kernel itself needs to regain SATA support (an update is already pending).
• I ve also got two Cubieboard2 devices and there is a sun7i-a20-cubieboard2.dtb file in the ARMMP kernel which deserves some investigation .

Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate .

The iMX53 Quick Start Board has been a mainstay of the Debian armhf port and has been part of LAVA until recently. It s a natural choice for seeing how LAVA and Debian can work together more closely. So, in the first steps to getting my home lab running, I set about restoring two of these boards to LAVA support. I will be looking into each of the possible boot methods with these boards, but the way to start is with a LAVA master image on an SD card. With the pause in iMX53 development within LAVA, this raises a couple of issues:

• u-boot on the iMX53 is old and the Linaro image tools have advanced without making allowances for the unsupported mx53loco type.
  • For LAVA, this means that u-boot on the iMX53 does not understand ext4 and although there are guides on updating it, I wanted to stay with the original, at least at the start.
  • Using ext4 will lead to this error:

    mmc0: error 110 whilst initialising SD card
    

• iMX53 quick start boards come with a working Ubuntu installation on an SD card but the release is Lucid Lynx 10.04LTS. LAVA has been on Precise Pangolin 12.04LTS for some time.
  • This causes issues with debootstrap which cannot unpack packages from Debian sid due to changes in the compression format inside the .deb files. It is possible to use wheezy and note that lucid, by default, will produce armel. Use --arch=armhf, then apt-get upgrade and apt-get dist-upgrade to testing and then to sid (not in one single step). Remember to mount proc before starting the upgrade.

LAVA master image on iMX53 The LAVA master image is more recent and can fix these problems. See lava-image-creation preparing a master image
$ git clone git://git.linaro.org/lava/lava-master-image-scripts.git
$ cd lava-master-image-scripts
Use patch -p1 to apply this patch (apologies for the line wrapping in the blog):

diff --git a/lava-create-master b/lava-create-master
index 2e1d0e8..a1e6131 100755
--- a/lava-create-master
+++ b/lava-create-master
@@ -366,7 +366,7 @@ make_master()  
         verbose " * Building vanilla image with linaro-media-create..."
         if ! linaro-media-create \
             --dev $LMC_DEV \
-            --rootfs ext4 \
+            --rootfs ext3 \
             --image-file $CACHE_DIR/pristine-images/$dev-vanilla.img \
             --image-size 1G \
             --align-boot-part \

$ sudo ./lava-create-master imx53 Without the patch applied, I simply removed the SD card, mounted the third partition on my main machine and created a tarball of the contents:

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -czf /tmp/imx53-rootfs.tgz ./*
$ popd
$ sudo umount /mnt/

Reformat p3 to ext3 (or ext2 if you prefer)

$ sudo mkfs.ext3 /dev/mmcblk0p3

Put the rootfs back onto the replacement ext3 partition.

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -xzvf /tmp/imx53-rootfs.tgz
$ popd
$ sudo sync
$ sudo umount /mnt/

The bootscript is buggy, so run:

> run loaduimage
> run mmcboot

Then fix bootscript or you won t be able to reboot: Simplest way may be to set u-boot to ignore boot.scr

> setenv script
> saveenv
> reset

Device configuration in LAVA I found screen to be a bit awkward as a serial connection, (use Ctrl-A : quit) to quit. I tried autologin for the lucid image but that was not needed for the master image which uses the linaro overlay. The PDU commands are for my own setup. The daemon (the machine running the pdu daemon scripts), hostname (the hostname or IP of the PDU itself) and the port numbers will change between labs. Again, the blog has changed the line endings.

device_type = mx53loco
hostname = imx53-01
connection_command = telnet hobbes 4002
#login_prompt = lucid-desktop login:
#password_prompt = Password:
#username = lucid
#password = lucid
hard_reset_command = /usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 05
power_off_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 05
power_on_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 05
testboot_offset = 3

Critical testboot_offset is currently undocumented. Without it, you will see:

 ** Partition 4 not valid on device 0 **

You can trace this back in a LAVA test log to:

  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:4 0x72000000 uInitrd; fatload mmc 0:4 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

The correct line needs to use mmc 0:5 in each case.
This is what testboot_offset modifies the default is 2.

  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:5 0x72000000 uInitrd; fatload mmc 0:5 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

Hacking session in LAVA A hacking session in LAVA is a way of getting an SSH login directly into a deployed test image, as root. It s a test image, so look around, trash stuff, build stuff, reboot the board When you are done, use stop-hacking and the test completes, leaving the master image unchanged, ready for the next rootfs to go onto the test image you were using. This will be improved in later blog entries but as a starter and so that you can play with the SATA drive with a recent installation, this is the JSON for a LAVA job which gives you an SSH session inside the LAVA test image on an iMX53. The test image comes from Linaro releases 12.01. I haven t set up a proxy yet, so to save time, I downloaded it once and used a local file:// URL in the testjob. Note the rootfstype parameter without this, LAVA will helpfully format the partition onto which the rootfs is installed to ext4, with predictable results. (This bug is currently being fixed in LAVA so that LAVA uses the same filesystem as the downloaded image. All part of making LAVA less presumptive.) The smoke-tests-basic testdef is gratuitous and can be removed but this particular run won t be quick it took over an hour to get into the hacking session, but future topics here will improve that so the extra few minutes for the smoke-tests is not a big win. Change the value for GATEWAY to something sane for your network, change the value for PUB_KEY to your SSH public key. Change the image location to something which works for your setup.

 
 "health_check": false,
 "job_name": "iMX53-hacking",
 "logging_level": "DEBUG",
 "device_type": "mx53loco",
 "timeout": 900,
 "actions": [
 
 "command": "deploy_linaro_image",
 "parameters": 
 "image": "file:///home/linaro/lava/mx53loco-ubuntu-desktop.img.gz",
 "rootfstype": "ext3"
 
 ,
 
 "command": "lava_test_shell",
 "parameters": 
 "testdef_repos": [
 
 "git-repo": "git://git.linaro.org/qa/test-definitions.git",
 "testdef": "ubuntu/smoke-tests-basic.yaml"
 ,
 
 "git-repo": "http://git.linaro.org/git/lava-team/hacking-session.git",
 "parameters": 
 "GATEWAY": "10.15.0.1",
 "PUB_KEY": "...."
 ,
 "testdef": "hacking-session-debian.yaml"
 
 ],
 "timeout": 18000
 
 ,
 
 "command": "submit_results_on_host",
 "parameters": 
 "server": "http://sylvester.codehelp/RPC2/",
 "stream": "/anonymous/codehelp/"
 
 
 ]
 

(Use http://jsonlint.com to reformat.) I was part of the way through running debootstrap on the SATA drive when ser2net timed out because I hadn t changed the values at that time. Configuring set2net for LAVA ser2net can run on any machine capable of providing the serial connections, including a dedicated device. Decide on a port range to use on the machine running ser2net. If the device has more than one network interface, ensure that ser2net only offers connections on the relevant interface by prefixing the port with the IP address or hostname. telnet is generally the easiest ser2net state to use with LAVA.

#  ::::
#     TCP port
#            Name   or  number of the TCP/IP port to accept con-
#            nections from for this device.  A port number may
#            be of the form [host,]port, such as 127.0.0.1,2000
#            or localhost,2000.  If this is specified, it will
#            only bind to the IP address specified. Otherwise
#            it will bind to all the ports on the machine.
# 10.15.0.1,4000:telnet:600:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner

Clear the timeout or long running jobs will fail.

10.15.0.1,4000:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner
10.15.0.1,4001:telnet:0:/dev/ttyUSB1:115200 8DATABITS NONE 1STOPBIT banner

I ve tried relatively carefully to preserve the instructions here from my notes. However, if you do try this yourself and there are problems, let me know and I ll update. I m also planning on putting all of the config files, patches, JSON examples and other code into a git repo fairly soon to make things easier.

The simple external-comments code is now complete enough for me to stop poking it on a daily basis:

• Although the comments are styled minimally you can override that with CSS.
• Although the default "Add your reply" form is ugly you can replace it with your own.
  • The reply-form may go above or below the comments.
• If you add an email field then your comments will include a gravitar link.
• Comments are assumed to be in markdown now.
• The commments may be retrieved in newest-first, or oldest-first order.
• There's now a simple anti-spam plugin system present.

All in all I'm pretty happy with the way it works, and the server-code. The client-side Javascript is less good, but I'm probably done poking that too. In an ideal world the client-side code should be a jQuery plugin, but I've not worked out how to make a static method (the JSONP callback) be a member of a jQuery plugin-object. So without that I have to re-pass the options around too many places, rather than making them a member of "this". Meh, pull requests welcome for adding new storage back-ends (redis and sqlite are supported by default), and similarly for cleanups. Links:

• https://github.com/skx/e-comments
• Simple demo & Customized demo.

When you run Puppet, it is very important to monitor whether all nodes have an uptodate catalog and did not miss the last year of changes because of a typo in a manifest or a broken cron-script. The most common solution to this is a script that checks /var/lib/puppet/state/last_run_summary.yaml on each node. While this is nice and easy in a small setup, it can get a bit messy in a bigger environment as you have to do an NRPE call for every node (or integrate the check as a local check into check_mk). Given a slightly bigger Puppet environment, I guess you already have PuppetDB running. Bonuspoints if you already let it save the reports of the nodes via reports = store,puppetdb. Given a central knowledgebase about your Puppet environment one could ask PuppetDB about the last node runs, right? I did not find any such script on the web, so I wrote my own: check_puppetdb_nodes. The script requires a recent (1.5) PuppetDB and a couple of Perl modules (JSON, LWP, Date::Parse, Nagios::Plugin) installed. When run, the script will contact the PuppetDB via HTTP on localhost:8080 (obviously configurable via -H and -p, HTTPS is available via -s) and ask for a list of nodes from the /nodes endpoint of the API. PuppetDB will answer with a list of all nodes, their catalog timestamps and whether the node is deactivated. Based on this result, check_puppetdb_nodes will check the last catalog run of all not deactivated nodes and issue a WARNING notification if there was none in the last 2 hours (-w) or a CRITICAL notification if there was none for 24 hours (-c). As a fresh catalog does not mean that the node was able to apply it, check_puppetdb_nodes will also query the /event-counts endpoint for each node and verify that the node did not report any failures in the last run (for this feature to work, you need reports stored in PuppetDB). You can modify the thresholds for the number of failures that trigger a WARNING/CRITICAL with -W and -C, but I think 1 is quite a reasonable default for a CRITICAL in this case. Using check_puppetdb_nodes you can monitor the health of ALL your Puppet nodes with a singe NRPE call. Or even with zero, if your monitoring host can access PuppetDB directly.

0 comment(s) this blog is flattr enabled

Last night I was up again, really hard to sleep when you have a bad cold. I decided to do something fun, and allow my tweaking guide to accept comments. Like many of my sites this is 100% static, and generated by templer, so comments are "hard". I've seen a few people try to rewrite disqus as a general-purpose solution, and I like that idea, because I don't trust that particular service. I wasn't so ambitious though, I just hacked up a quick sinatra server:

• "GET /comments/ID"
  • Retrieves the comments on the specified identifier as a JSON array of comment-hashes.
• "POST /comments/ID"
  • Append the submitted comment to a redis set.

My jquery/javascript is nasty, but the thing seems to work pretty well. The page loads and comments are populated, and new ones are persisted as expected.

I can see the appeal of putting all this magic in one javascript file. You include that and get both the existing comments and the form to add new ones - my approach is to hardwire the submission/display in my generated site. Perhaps something for the future.

In conclusion if people wish they can now leave feedback on most of the pages :)

Total keys: 996
Key versions:
    996 4
Primary key pubkey algorithms:
    623 "DSA"
    373 "RSA"
Primary key pubkey sizes:
    624 1024
     27 2048
      2 3072
    340 4096
      2 8192
      1 10240
Total number of UIDs + UAts: 4394
Hash algorithm used for most recent self-sig:
      1 "RIPEMD160"
   3188 "SHA1"
   1041 "SHA256"
      1 "SHA384"
    163 "SHA512"
Judgment on preferred hash algorithms:
   1776 null
   2618 "weak hash with higher preference"
Judgment on expiration times:
     53 "expiration passed"
    111 "expiration too far in future"
   3887 "no expiration set"
    343 null

This was generated with debian-keyring 2013.12.13, hopenpgp-tools 0.4-1, jshon 20131010-3, and the following inefficient script:

#!/bin/zsh
infile=/usr/share/keyrings/debian-keyring.gpg
tempfile=$(mktemp)
trap 'rm $ tempfile ' EXIT
hokey lint --output-format JSON <$ infile  >$ tempfile 
print -n "Total keys: "
jshon -a -e keyFingerprint <$ tempfile    wc -l
print "Key versions: "
jshon -a -e keyVer -e val <$ tempfile    sort   uniq -c
print "Primary key pubkey algorithms: "
jshon -a -e keyAlgorithmAndSize -e pubkeyalgo -e val <$ tempfile    sort   uniq -c
print "Primary key pubkey sizes: "
jshon -a -e keyAlgorithmAndSize -e pubkeysize -e val <$ tempfile    sort -n   uniq -c
print -n "Total number of UIDs + UAts: "
jshon -a -e keyUIDsAndUAts -k <$ tempfile    wc -l
print "Hash algorithm used for most recent self-sig: "
jshon -a -e keyUIDsAndUAts -a -e uidSelfSigHashAlgorithms -a -e val <$ tempfile    sort   uniq -c
print "Judgment on preferred hash algorithms: "
jshon -a -e keyUIDsAndUAts -a -e uidPreferredHashAlgorithms -a -e explanation <$ tempfile    sort   uniq -c
print "Judgment on expiration times: "
jshon -a -e keyUIDsAndUAts -a -e uidKeyExpirationTimes -a -e explanation <$ tempfile    sort   uniq -c

This represents incorrect handling of revoked UIDs and user attributes, and possibly unknown bugs. "Judgments" are based on this document and are not generalized per key.

One week before the wedding, to Laura: Mono won t just clear up right away. One week before the wedding, to me: That s going to need stitches. Yes, not long before the wedding, Laura had come down with mononucleosis and I had cut into my finger with a very sharp knife while cutting bread requiring a trip to the emergency room to get stitches. Two days before we got married, instead of moving furniture, I was getting stitches out of my finger. It wasn t the kind of week we had planned. But it was the happiest, most amazing occasion I could have ever imagined. As I wrote last month, I am richly blessed indeed. Our wedding was three days after Christmas. The church was still decorated for Christmas, with the tree in on corner, glittering stars suspended in mid-air on cables from the walls, wreaths and candles in the windows, and it was a joy-filled day. Before the ceremony, we took pictures the only part of the day Jacob and Oliver weren t thrilled with. Nevertheless, we got some fun ones. Laura and I seem to know quite a few pastors between us and not just because Laura is a pastor. My brother officiated with the wedding vows, his wife with scripture and a prayer of blessing, and the church s pastor gave the message. Laura and I wrote in our wedding program, Music has long been a thread running through both our lives. We have enjoyed singing together, playing piano and pennywhistle duets, attending concerts, and even exploring old hymnals. Music is also one of the best ways to have a conversation even a conversation with God. We wrote a page in the program about each of the hymns that were a part of the wedding, and why we picked them. The combined church choirs of my home church and Laura s church sang John Rutter s beautiful arrangement of For the Beauty of the Earth (click here to listen to a different choir). Hearing For the beauty of each hour , For the joy of human love , and Lord of all, to thee we raise this our joyful hymn of praise was perfect for the day. It was with such great happiness that we walked out of the sanctuary, a married couple, to the sound of the congregation singing Joy to the World! Jacob and Oliver were so very excited on our wedding day. They happily explored the church while waiting for things to happen. We had them help us light our unity candle, and they were pleased with that. Jacob loved his suit, which made him look just like me. And they were, of course, delighted with the cake and in the middle of it all. For our honeymoon, we managed to get two weeks of vacation, and spent about half of it at home. We had looked at various options for retreats in the country, but eventually concluded that our house is a retreat in the country, so might as well enjoy it at home. We also went to the Palo Duro Canyon area near Amarillo in the Texas panhandle, staying in a small B&B in Canyon, TX. Palo Duro is the second-largest canyon in North America, and quite colorful year-round. What a beautiful place to go for our honeymoon! By the time we got there, Laura was getting past mono, and we went for hikes in the canyon on two different days hiking a total of 10 miles, including a hike up the side of the canyon. After we got back home, on the last weekday of our honeymoon, we went back to the Flint Hills of Kansas, to some of the same places we had spent our third date. We climbed the windy staircase at the Chase County Courthouse, the oldest courthouse still in use in Kansas. And peered out its famous oval window. We found the last remnant of the old ghost town of Elk, ate at the same restaurant we had that day. It brought back wonderful memories, and it was a good day in itself. Because even though it was a gold, drizzly, overcast day in January, this time, we were married. And this year, Thanksgiving is all year.