When it comes to provide redundant services, several options are available:

• The service can be hosted behind a set of load-balancers. They will detect any faulty node. However, you need to ensure that this new layer is also fault-tolerant.
• The nodes providing the service can rely on IP failover to share a set of IP using protocols like VRRP¹ or CARP. The IP address of a faulty node will be assigned to another node. This requires all nodes to be part of the same IP subnet.
• The clients of the service can ask a third-party for available nodes. Usually, this is achieved through round-robin DNS where only working nodes are advertised in a DNS record. The failover can be quite long because of caches.

A common setup is a combination of those solutions: web servers are behind a couple of load-balancers achieving both redundancy and load-balancing. The load-balancers use VRRP to ensure redundancy. The whole setup is replicated to another datacenter and round-robin DNS is used to ensure both redundancy and load-balacing of the datacenters. There is a fourth option which is similar to VRRP but relies on dynamic routing and therefore is not limited to nodes in the same subnet:

• The nodes advertise their availability with BGP to announce the set of service IP addresses they are able to serve. Each address is weighted such that IP addresses are balanced among the available nodes.

We will explore how to implement this fourth option using ExaBGP, the BGP swiss army knife of networking, in a small lab based on KVM. You can grab the complete lab from GitHub. ExaBGP 3.2.5 is needed to run this lab.

Environment We will be working in the following (IPv6-only) environment:

BGP configuration BGP is enabled on ER2 and ER3 to exchange routes with peers and transits (only R1 for this lab). BIRD is used as a BGP daemon. Its configuration is pretty basic. Here is a fragment:

router id 1.1.1.2;
protocol static NETS   #  
  import all;
  export none;
  route 2001:db8::/40 reject;
 
protocol bgp R1   #  
  import all;
  export where proto = "NETS";
  local as 64496;
  neighbor 2001:db8:1000::1 as 64511;
 
protocol bgp ER3   #  
  import all;
  export all;
  next hop self;
  local as 64496;
  neighbor 2001:db8:1::3 as 64496;
 

First, in , we declare the routes that we want to export to our peers. We don t try to summarize them from the IGP. We just unconditonnaly export the networks that we own. Then, in , R1 is defined as a neighbor and we export the static route we declared previously. We import any route that R1 is willing to send us. In , we share everything we know with our pal, ER3, using internal BGP.

OSPF configuration OSPF will distribute routes inside the AS. It is enabled on ER2, ER3, DR6, DR7 and DR8. For example, here is the relevant part of the configuration of DR6:

router id 1.1.1.6;
protocol kernel  
   persist;
   import none;
   export all;
 
protocol ospf INTERNAL  
  import all;
  export none;
  area 0.0.0.0  
    networks  
      2001:db8:1::/64;
      2001:db8:6::/64;
     ;
    interface "eth0";
    interface "eth1"   stub yes;  ;
   ;
 

ER2 and ER3 inject a default route into OSPF:

protocol static DEFAULT  
  import all;
  export none;
  route ::/0 via 2001:db8:1000::1;
 
filter default_route  
  if proto = "DEFAULT" then accept;
  reject;
 
protocol ospf INTERNAL  
  import all;
  export filter default_route;
  area 0.0.0.0  
    networks  
      2001:db8:1::/64;
     ;
    interface "eth1";
   ;
 

Web nodes The web nodes are pretty basic. They have a default static route to the nearest router and that s all. The interesting thing here is that they are each on a separate IP subnet: we cannot share an IP using VRRP². Why are those web servers on different subnets? Maybe they are not in the same datacenter or maybe your network architecture is using a routed access layer. Let s see how to use BGP to enable redundancy of those web nodes.

Redundancy with ExaBGP ExaBGP is a convenient tool to plug scripts into BGP. They can then receive and advertise routes. ExaBGP does the hard work of speaking BGP with your routers. The scripts just have to read routes from standard input or advertise them on standard output.

The big picture Here is the big picture: Let s explain it step by step:

1. Three IP addresses will be allocated for our web service: 2001:db8:30::1, 2001:db8:30::2 and 2001:db8:30::3. Those are distinct from the real IP addresses of W1, W2 and W3.
2. Each web node will advertise all of them to the route servers we added in the network. I will talk more about those route servers later.
3. Each route comes with a metric to help the route server choose where it should be routed. We choose the metrics such that each IP address will be routed to a distinct web node (unless there is a problem).
4. The route servers (which are not routers) will then advertise the best routes they learned to all the routers in our network. This is still done using BGP.
5. Now, for a given IP address, each router knows to which web node the traffic should be directed.

Here are the respective metrics announced routes for W1, W2 and W3 when everything works as expected:

Route	W1	W2	W3	Best	Backup
2001:db8:30::1	102	101	100	W3	W2
2001:db8:30::2	101	100	102	W2	W1
2001:db8:30::3	100	102	101	W1	W3

ExaBGP configuration The configuration of ExaBGP is quite simple:

group rs  
  neighbor 2001:db8:1::4  
    router-id 1.1.1.11;
    local-address 2001:db8:6::11;
    local-as 65001;
    peer-as 65002;
   
  neighbor 2001:db8:8::5  
    router-id 1.1.1.11;
    local-address 2001:db8:6::11;
    local-as 65001;
    peer-as 65002;
   
  process watch-nginx  
      run /usr/bin/python /lab/healthcheck.py -s --config /lab/healthcheck-nginx.conf --start-ip 0;
   
 

A script is entailed to check if the service (an nginx web server) is up and running and advertise the appropriate IP addresses to the two declared route servers. If we run the script manually, we can see the advertised routes:

$ python /lab/healthcheck.py --config /lab/healthcheck-nginx.conf --start-ip 0
INFO[healthcheck] send announces for UP state to ExaBGP
announce route 2001:db8:30::3/128 next-hop self med 100
announce route 2001:db8:30::2/128 next-hop self med 101
announce route 2001:db8:30::1/128 next-hop self med 102
[...]
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
INFO[healthcheck] send announces for DOWN state to ExaBGP
announce route 2001:db8:30::3/128 next-hop self med 1000
announce route 2001:db8:30::2/128 next-hop self med 1001
announce route 2001:db8:30::1/128 next-hop self med 1002

When the service becomes unresponsive, the healthcheck script detect the situation and retry several times before acknowledging that the service is dead. Then, the IP addresses are advertised with higher metrics and the service will be routed to another node (the one advertising 2001:db8:30::3/128 with metric 101). This healthcheck script is now part of ExaBGP.

Route servers We could have connected our ExaBGP servers directly to each router. However, if you have 20 routers and 10 web servers, you now have to manage a mesh of 200 sessions. The route servers are here for three purposes:

1. Reduce the number of BGP sessions (from 200 to 60) between equipments (less configuration, less errors).
2. Avoid modifying the configuration on routers each time a new service is added.
3. Separate the routing decision (the route servers) from the routing process (the routers).

You may also ask yourself: why not use OSPF? . Good question!

• OSPF could be enabled on each web node and the IP addresses advertised using this protocol. However, OSPF has several drawbacks: it does not scale, there are restrictions on the allowed topologies, it is difficult to filter routes inside OSPF and a misconfiguration will likely impact the whole network. Therefore, it is considered a good practice to limit OSPF to network equipments.
• The routes learned by the route servers could be injected into OSPF. On paper, OSPF has a next-hop field to provide an explicit next-hop. This would be handy as we wouldn t have to configure adjacencies with each router. However, I have absolutely no idea how to inject BGP next-hop into OSPF next-hop. What happens is that BGP next-hop is resolved locally using OSPF routes. For example, if we inject BGP routes into OSPF from RS4, RS4 will know the appropriate next-hop but other routers will route the traffic to RS4.

Let s look at how to configure our route servers. RS4 will use BIRD while RS5 will use Quagga. Using two different implementations will help with resiliency by avoiding a bug to hit the two route servers at the same time.

Bird configuration There are two sides for BGP configuration: the BGP sessions with the ExaBGP nodes and the ones with the routers. Here is the configuration for the later:

template bgp INFRABGP  
  export all;
  import none;
  local as 65002;
  rs client;
 
protocol bgp ER2 from INFRABGP  
  neighbor 2001:db8:1::2 as 65003;
 
protocol bgp ER3 from INFRABGP  
  neighbor 2001:db8:1::3 as 65003;
 
protocol bgp DR6 from INFRABGP  
  neighbor 2001:db8:1::6 as 65003;
 
protocol bgp DR7 from INFRABGP  
  neighbor 2001:db8:1::7 as 65003;
 
protocol bgp DR8 from INFRABGP  
  neighbor 2001:db8:1::8 as 65003;
 

The AS number used by our route server is 65002 while the AS number used for routers is 65003 (the AS numbers for web nodes will be 65001). They are reserved for private use by RFC 6996. All routes known by the route server are exported to the routers but no routes are accepted from them. Let s have a look at the other side:

# Only import loopback IPs
filter only_loopbacks   #  
  if net ~ [ 2001:db8:30::/64 128,128  ] then accept;
  reject;
 
# General template for an EXABGP node
template bgp EXABGP  
  local as 65002;
  import filter only_loopbacks; #  
  export none;
  route limit 10; #  
  rs client;
  hold time 6; #  
  multihop 10;
  igp table internal;
 
protocol bgp W1 from EXABGP  
  neighbor 2001:db8:6::11 as 65001;
 
protocol bgp W2 from EXABGP  
  neighbor 2001:db8:7::12 as 65001;
 
protocol bgp W3 from EXABGP  
  neighbor 2001:db8:8::13 as 65001;
 

To ensure separation of concerns, we are being a bit more picky. With and , we only accept loopback addresses and only if they are contained in the subnet that we reserved for this use. No server should be able to inject arbitrary addresses into our network. With , we also limit the number of routes that a server can advertise. With , we reduce the hold time from 240 to 6. It means that after 6 seconds, the peer is considered dead. This is quite important to be able to recover quickly from a dead machine. The minimal value is 3. We could have use a similar setting with the session with the routers.

Quagga configuration Quagga s configuration is a bit more verbose but should be strictly equivalent:

router bgp 65002 view EXABGP
 bgp router-id 1.1.1.5
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor R peer-group
 neighbor R remote-as 65003
 neighbor R ebgp-multihop 10
 neighbor EXABGP peer-group
 neighbor EXABGP remote-as 65001
 neighbor EXABGP ebgp-multihop 10
 neighbor EXABGP timers 2 6
!
 address-family ipv6
 neighbor R activate
 neighbor R soft-reconfiguration inbound
 neighbor R route-server-client
 neighbor R route-map R-IMPORT import
 neighbor R route-map R-EXPORT export
 neighbor 2001:db8:1::2 peer-group R
 neighbor 2001:db8:1::3 peer-group R
 neighbor 2001:db8:1::6 peer-group R
 neighbor 2001:db8:1::7 peer-group R
 neighbor 2001:db8:1::8 peer-group R
 neighbor EXABGP activate
 neighbor EXABGP soft-reconfiguration inbound
 neighbor EXABGP maximum-prefix 10
 neighbor EXABGP route-server-client
 neighbor EXABGP route-map RSCLIENT-IMPORT import
 neighbor EXABGP route-map RSCLIENT-EXPORT export
 neighbor 2001:db8:6::11 peer-group EXABGP
 neighbor 2001:db8:7::12 peer-group EXABGP
 neighbor 2001:db8:8::13 peer-group EXABGP
 exit-address-family
!
ipv6 prefix-list LOOPBACKS seq 5 permit 2001:db8:30::/64 ge 128 le 128
ipv6 prefix-list LOOPBACKS seq 10 deny any
!
route-map RSCLIENT-IMPORT deny 10
!
route-map RSCLIENT-EXPORT permit 10
  match ipv6 address prefix-list LOOPBACKS
!
route-map R-IMPORT permit 10
!
route-map R-EXPORT deny 10
!

The view is here to not install routes in the kernel³.

Routers Configuring BIRD to receive routes from route servers is straightforward:

# BGP with route servers
protocol bgp RS4  
  import all;
  export none;
  local as 65003;
  neighbor 2001:db8:1::4 as 65002;
  gateway recursive;
 
protocol bgp RS5  
  import all;
  export none;
  local as 65003;
  neighbor 2001:db8:8::5 as 65002;
  multihop 4;
  gateway recursive;
 

It is important to set gateway recursive because most of the time, the next-hop is not reachable directly. In this case, by default, BIRD will use the IP address of the advertising router (the route servers).

Testing Let s check that everything works as expected. Here is the view from RS5:

# show ipv6  bgp  
BGP table version is 0, local router ID is 1.1.1.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network          Next Hop            Metric LocPrf Weight Path
*  2001:db8:30::1/128
                    2001:db8:6::11         102             0 65001 i
*>                  2001:db8:8::13         100             0 65001 i
*                   2001:db8:7::12         101             0 65001 i
*  2001:db8:30::2/128
                    2001:db8:6::11         101             0 65001 i
*                   2001:db8:8::13         102             0 65001 i
*>                  2001:db8:7::12         100             0 65001 i
*> 2001:db8:30::3/128
                    2001:db8:6::11         100             0 65001 i
*                   2001:db8:8::13         101             0 65001 i
*                   2001:db8:7::12         102             0 65001 i
Total number of prefixes 3

For example, traffic to 2001:db8:30::2 should be routed through 2001:db8:7::12 (which is W2). Other IP are affected to W1 and W3. RS4 should see the same thing⁴:

$ birdc6 show route 
BIRD 1.3.11 ready.
2001:db8:30::1/128 [W3 22:07 from 2001:db8:8::13] * (100/20) [AS65001i]
                   [W1 23:34 from 2001:db8:6::11] (100/20) [AS65001i]
                   [W2 22:07 from 2001:db8:7::12] (100/20) [AS65001i]
2001:db8:30::2/128 [W2 22:07 from 2001:db8:7::12] * (100/20) [AS65001i]
                   [W1 23:34 from 2001:db8:6::11] (100/20) [AS65001i]
                   [W3 22:07 from 2001:db8:8::13] (100/20) [AS65001i]
2001:db8:30::3/128 [W1 23:34 from 2001:db8:6::11] * (100/20) [AS65001i]
                   [W3 22:07 from 2001:db8:8::13] (100/20) [AS65001i]
                   [W2 22:07 from 2001:db8:7::12] (100/20) [AS65001i]

Let s have a look at DR6:

$ birdc6 show route
2001:db8:30::1/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]
2001:db8:30::2/128 via fe80::5054:60ff:fe02:3681 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:60ff:fe02:3681 on eth0 (100/20) [AS65001i]
2001:db8:30::3/128 via 2001:db8:6::11 on eth1 * (100/10) [AS65001i]
                   via 2001:db8:6::11 on eth1 (100/10) [AS65001i]

So, 2001:db8:30::3 is owned by W1 which is behind DR6 while the two others are in another part of the network and will be reached through the appropriate link-local addresses learned by OSPF. Let s kill W1 by stopping nginx. A few seconds later, DR6 learns the new routes:

$ birdc6 show route
2001:db8:30::1/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]
2001:db8:30::2/128 via fe80::5054:60ff:fe02:3681 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:60ff:fe02:3681 on eth0 (100/20) [AS65001i]
2001:db8:30::3/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).

---

1. The primary target of VRRP is to make a default gateway for an IP network highly available by exposing a virtual router which is an abstract representation of multiple routers. The virtual IP address is held by the master router and any backup router can be promoted to master in case of a failure. In practice, VRRP can be used to achieve high availability of services too.
2. However, you could deploy an L2 overlay network using, for example, VXLAN and use VRRP on it.
3. I have been told on the Quagga mailing-list that such a setup is quite uncommon and that it would be better to not use views but use --no_kernel flag for bgpd instead. You may want to look at the whole thread for more details.
4. The output of birdc6 shows both the next-hop as advertised in BGP and the resolved next-hop if the route have to be exported into another protocol. I have simplified the output to avoid confusion.

JPNIC published The Internet Timeline (Japanese version is also there). My first contact with the Internet was about 20 years ago. I was a Universal student. The University had 64kbps bandwidth, it was really slow. Nowerday, I have 128kbps mobile network with 950 yen/month. It is usable for some use case, like text messaging.

Pretend I run a cluster, for hosting a site. Pretend that I have three-six web-nodes, and each one needs to know which database host to contact. How do I control that? Right now I have a /etc/settings.conf file, more or less, deployed by Slaughter. That works. Another common pattern is to use a hostname - for example pmaster.example.org. However failover isn't considered here. If I wanted to update to point to a secondary database I'd need to either:

• Add code to retry the second host on failure.
  • Worry about divergence if some hosts used DB1, then DB2, then DB1 came back online.
  • Failover is easy. Fail-back is probably best avoided.
• Worry about DNS caches and TTL.

In short I'm imagining there are several situations where you want to abstract away the configuration in a cluster-wide manner. (A real solution is obviously floating per-service IPs. Via HAProxy, Keepalived, ucarp, etc. People do that quite often for database specifically, but not for redis-servers, etc.) So I'm pondering what is essentially a multi-cast accessible key-value storage system. Have a deamon on the VLAN which will respond to multicast questions like "get db", or "get cache", with a hostname/IP/result. Suddenly your code would read:

• Send mcast question ("which db?").
• Get mcast reply ("db1").
• Connect to db1.

To me that seems like it should be genuinely useful. But I'm unsure if I'm trading one set of problems for another. I can't find any examples of existing tools/deamons in this area, which either means I'm being novel, innovate, and interesting. Or I'm over thinking...

Kim Collins was perhaps thinking more about physical improvement but his advice holds well for software.

A lot has been written about the problems around software engineers wanting to rewrite a codebase because of "legacy" issues. Experience has taught me that refactoring is a generally better solution than rewriting because you always have something that works and can be released if necessary.

Although that observation applies to the whole of a project, sometimes the technical debt in component modules means they need reimplementing. Within NetSurf we have historically had problems when such a change was done because of the large number of supported platforms and configurations.
HistoryA year ago I implemented a Continuous Integration (CI) solution for NetSurf which, combined with our switch to GIT for revision control, has transformed our process. Making several important refactor and rewrites possible while being confident about the overall project stability.

I know it has been a year because the VPS hosting bill from Mythic turned up and we are still a very happy customer. We have taken the opportunity to extend the infrastructure to add additional build systems which is still within the NetSurf projects means.

Over the last twelve months the CI system has attempted over 100,000 builds including the projects libraries and browser. Each commit causes an attempt to build for eight platforms, in multiple configurations with multiple compilers. Because of this the response time to a commit is dependant on the slowest build slave (the mac mini OS X leopard system).

Currently this means a browser build, not including the support libraries, completes in around 450 seconds. The eleven support libraries range from 30 to 330 seconds each. This gives a reasonable response time for most operations. The worst case involves changing the core buildsystem which causes everything to be rebuilt from scratch taking some 40 minutes.

The CI system has gained capability since it was first set up, there are now jobs that:

• Perform and publish the results of a static analysis for each component using the llvm/clang project scan-build tool.
• Run coverage reports on modules which support gcov.
• Build and install full toolchains for the cross compiled target platforms.
• Run components automated tests on native platforms
• Generate release sources and packages on a correctly tagged git commit.
• Generates and publishes the automated Doxygen documentation.

The first alpha of syslog-ng 3.4 was released more than a year ago, and the first stable release earlier this year, development on syslog-ng 3.5 has been going on steadlily since then, with a lot of interesting features and changes pouring in. While the 3.5 release will not bring as many overwhelming changes as the previous two releases did, the sheer number of improvements is still substantial. Our focus was on smaller developments all over the place this time, instead of big sweeping changes. In this post, we will go over most of these features, with use-cases, examples and a few words about why they are so useful and awesome.Multi-line support, new destinations, new template functions, type hinting, unit suffixes, and other things. Read on, and be amazed!These are only bigger features, there has been a lot of small tweaking going on, and of course, all of the fixes that went into prior versions will also be in 3.5 too, and there's still a chance that something else may find its way in before the feature freeze.Multi-line supportOne of the major new features in the 3.5 release will be support for multi-line messages, a feature that has been available in syslog-ng PE for a good while, and which has been ported to and improved upon to the open source edition. Two variants of multi-line are supported, which will be detailed below. Both of them are available for the file() and pipe() sources only.Indented multi-lineThe easiest variant is indented multi-line, where each line can be followed by others, indented by whitespace, and the message continues until the first non-indented line. This is the format used by the Linux kernel too, from version 3.5, for /dev/log. This type of multi-line can be used as follows:

source s_multiline   file("/path/to/file" multi-line-mode(indented));  ;

Consider that /path/to/file has the following content:

First line
 Continuation 1;
 Continuation 2;
Second line

With the indented multi-line-mode() setting, this would turn into two log messages:

First line\n Continuation 1;\n Continuation 2;
Second line

Regexp-based multilineIf multi-line input is not based on indentation, one can use the regexp multi-line-mode() instead, which makes two new settings available: multi-line-prefix() and multi-line-garbage(). These can be used to define the start and the end of a log message: any string between a the beginning matching prefix and a matching garbage will be considered a single message. That is, the prefix will be included, the garbage will not be: it will be discarded.To illustrate:

source s_multiline  
 file("/path/to/file" multi-line-mode(regexp)
                      multi-line-prefix("^prefix")
                      multi-line-garbage(" garbage$"));
 ;

If the source contains these lines:

prefix message
continuing garbage

This will turn into the following log message:

prefix message\ncontinuing

New destinationsIt would be pretty hard to do a syslog-ng release without new destinations, and with 3.5, we will have three of them!STOMPWhile we had AMQP before, we now have a STOMP destination too, so we can stream logs to any STOMP server, with a few simple lines:

stomp(
 host("localhost")
 port(61613)
 destination("/topic/syslog")
 routing_key("")
 persistent(yes)
 ack(no)
 username("someone")
 password("something")
 body("This is the body! YAY! Here's a message: $MESSAGE!\n")
 value-pairs(scope(nv-pairs, syslog-proto, selected-macros))
)

Of course, none of these settings need to be set, you can just use the defaults, and it will just work!RiemannDid you know that syslog-ng is far more than a log collector and processor? No? You do now. When you have access to a lot of logs, and a tremendous amount of power in parsing them, you can use these tools for monitoring easily! And when we're talking monitoring, riemann is a great asset in our toolbox, and with the new destination, we can easily forward metrics to it:

riemann(
 server("localhost")
 port(5555)
 host("$HOST")
 service("$PROGRAM")
 description("$PROGRAM pids")
 metric(int("$PID"))
 ttl(300)
);

Of course, the above is a very silly example, it would make much more sense to extract data from a log message instead, but this will serve as an example.New template functions

$(upper-case TEXT...), $(lower-case TEXT...)

To turn some text into *upper-* or *lower-case*, these two template functions come in handy. They simply turn all their arguments into the respecitve case.

$(delimit DELIMITERS NEW-DELIMITER TEXT)

Sometimes one has a delimited string, where one wishes to replace the delimiters. This function does just that: give it a list of delimiters (you may need to quote it, if it contains whitespace), a replacement, and a text to replace delimiters in, and it does the rest. For example, to replace tabs and spaces with a vertical bar, one could write a template like this:

template("$(delimit \"\t \" \" \" $MESSAGE)\n")

$(env VARIABLE...)

Have you ever felt the need to check an environment variable from within syslog-ng? I did, and now I can.

Type hintingWhile syslog-ng supported sending log messages into various data stores and message queues for a while (SQL at first, MongoDB, JSON and AMQP later), even when those supported different types of data than strings, we could not do anything else. Until now. It is now possible - at places where it makes sense - to annotate templates with type hints, which the destination driver can optionally use. Type hinting is implemented for the mongodb() destination and the $(format-json) template function for now. When no type hint is specified, syslog-ng defaults to string.To add type hints, simply wrap the respective template with the hinted type, like this:

mongodb(
 value-pairs(pair("date", datetime("$UNIXDATE"))
             pair("pid", int64("$PID")))
);

Currently the following type hints exist: boolean (anything that begins with a t or 1 is true, anything that begins with f or 0 is false, everything else is an error), string, literal (same as string, but not quoted if it would be quoted otherwise), int32 (int is an alias for this), int64, and datetime. Only UNIX timestamps can be type-hinted to datetime, anything else will likely result in a casting error.It is also possible to control what happens when type casting fails: syslog-ng can drop the whole message, drop the property, or fall back to string. It can also do all of these silently:

options  
 typecast(on-error(silently-drop-property));
 ;

Using this feature with $(format-json) is very similary too:

$(format-json date=datetime("$UNIXDATE") pid=int64("$PID"))

With this feature in place, you can now store your non-string values with their proper types!Unit suffixesUnit suffixes make it considerably easier to set limits and describe numbers within the syslog-ng configuration. We no longer need to spell out sizes to the byte precious, it is now enough to write: log-fifo-size(200MiB). Now, syslog-ng will understand suffixes for kilo-, mega-, and giga-bytes, (K, M, G, respectively) either in base-10 or base-2 (with an extra i after the suffix). One can also omit the trailing b from the end.So, to set the log-fifo-size() to 2097152 bytes, one can simply use 2MiB. Or, to set it to 2000000, 2Mb. That's a whole lot easier, isn't it? No more counting zeros, no more silly typos in a ten-digit number, no more pain, but easily readable units!Miscellaneous featuresApart from the features above, there have been a lot of other changes and improvements in the code base:

• We now have a (mostly) non-recursive, quiet build system, which is not only much faster than what we had before, but more reliable too, and easier to glance through its logs, too.
• A few old settings that were never used were removed (the username() and password() settings of the mongodb() driver), some were renamed and deprecated: the replace() key transformation function of value-pairs() was renamed to replace-prefix(), as that makes the intent clearer. In this latter case, the old name is still valid, but obsoleted.
• Our integration with systemd is much tighter now: syslog-ng can notify systemd when it is ready. This also means that when systemd support is enabled, certain systemd libraries will have to be installed: syslog-ng no longer carries a convenience copy.
• A new filter was implemented: in-list, with which one can implement efficient white- or blacklists. To use it, you will need a file with one value a line, and do something along these lines:

  filter f_whitelist  
    in-list("/path/to/file.list", value("PROGRAM"));
   ;

  To do a blacklist, just negate the filter:

  filter f_blacklist  
    not in-list("/path/to/file.list", value("HOST"));
   ;

• syslog-ng now supports Linux 3.5+-style /dev/kmsg, and will use that instead of /proc/kmsg when a sufficiently recent kernel is detected (assuming one is using the system() source). The new kernel log format supports structured messages, and syslog-ng is smart enough to parse them, and make them accessible like all other message properties (with a .linux. prefix).

The blog engine PluXml is now available as a Debian package:

Package: pluxml
Version: 5.2-2
Description-en: Light blog engine using XML files
 PluXml is a light and simple blog and CMS engine that uses simple XML
 files to store its data and requires no database. It has all the usual
 features (static pages, comments, categories, tags, medias, RSS
 feed...) and supports multiple users, customizable themes and plugins.
Homepage: http://www.pluxml.org/
Section: web
Priority: extra

PluXml is a French blog engine that uses XML files to store its data, thus avoiding the need for an external database system as most blogs do. The name PluXml is a French portmanteau between plume (feather, evoking something light) and XML. Apart from that, although it was started in French only and its official website is still in French, it was internationalized about a year ago. It is designed to be easy enough to use, targeting regular users, but not too fancy: for instance, it provides a web-based administration interface, but content editing uses HTML code unless you install a graphical content editor plugin. That package may be of interest to people like me, that consider the almost systematic use of relational database systems in many web applications as a disease, and more generally to self-hosted people that would rather avoid running an additional database server on their small home servers. The package includes an interactive configuration, that allows to choose the blog language, title, description, admin login and password. Unless debconf was configured not to ask questions, in which case nothing is asked and the blog is left in an barely unusable state, of course.

Altimeter Testing at Airfest Bdale and I, along with AJ Towns and Mike Beattie, spent last weekend in Argonia, Kansas, flying rockets with our Kloudbusters friends at Airfest 19. We had a great time! AJ and Mike both arrived a week early at Bdale s to build L3 project airframes, and both flew successful cert flights at Airfest! Airfest was an opportunity for us to test fly prototypes of new flight electronics Bdale and I have spent the last few weeks developing, and I thought I d take a few minutes today to write some notes about what we built and flew. TeleMega We ve been working on TeleMega for quite a while. It s a huge step up in complexity from our original TeleMetrum, as it has a raft of external sensors and six pyro circuits. Bdale flew TeleMega in his new fiberglass 4 airframe on a Loki 75mm blue M demo motor. GPS tracking was excellent; you can see here that GPS altitude tracked the barometric sensor timing exactly: GPS lost lock when the motor lit, but about 3 seconds after motor burnout, it re-acquired the satellite signals and was reporting usable altitude data right away. The GPS reported altitude was higher than the baro sensor, but that can be explained by our approximation of an atmospheric model used to convert pressure into altitude. The rest of the flight was also nominal; TeleMega deployed drogue and main chutes just fine. TeleMetrum We ve redesigned TeleMetrum. The new version uses better sensors (MS5607 baro sensor, MMA6555 accelerometer) and a higher power radio (CC1120 40mW). The board is the same size, all the connectors are in the same places so it s a drop-in replacement, and it s still got two pyro channels and USB for configuration, data download and battery charging. I loaded up my Candy-Cane airframe with a small 5 grain 38mm CTI classic: The flight computer worked perfectly, but GPS reception was not as good as we d like to see: Given how well TeleMega was receiving GPS signals, I m hopeful that we ll be able to tweak TeleMetrum to improve performance. TeleMini We ve also redesigned TeleMini. It s still a two-channel flight computer with logging and telemetry, but we ve replaced the baro sensor with the MS5607, added on-board flash for increased logging space and added on-board screw terminals for an external battery and power switch. You can still use one of our 3.7V batteries, but you can also use another battery providing from 3.7 to 15V. I was hoping to finish up the firmware and fly it, but I ran out of time before the launch. The good news is that all of the components of the board have been tested and work correctly, and the firmware is feature complete , meaning we ve gotten all of the features coded, it s just not quite working yet. EasyMini EasyMini is a new product for us. It s essentially the same as a TeleMini, but without a radio. Two channels, baro-only, with logging. Like TeleMini, it includes an on-board USB connector and can use either one of our 3.7V batteries, or an external battery from 3.7V to 15V. EasyMini and TeleMini are the same size, and have holes in the same places, so you can swap between them easily. I flew EasyMini in my Koala airframe with a 29mm 3 grain CTI blue-streak motor. EasyMini successfully deployed the main chute and logged flight data: We also sent a couple of boards home with Kevin Trojanowski and Greg Rothman for them to play with. TeleGPS TeleGPS is a GPS tracker, incorporating a u-blox Max receiver and a 70cm transmitter. It can send position information via APRS or our usual digital telemetry formats. I was also hoping to have the TeleGPS firmware working, and I spent a couple of nights in the motel coding, but didn t manage to finish up. So, no data from this board either. Production Plans Given the success of the latest TeleMega prototype, we re hoping to have it into production first. We ll do some more RF testing on the bench with the boards to make sure it meets our standards before sending it out for the first production run. The goal is to have TeleMega ready to sell by the end of October. TeleMetrum clearly needs work on the layout to improve GPS RF performance. With the testing equipment that Bdale is in the midst of re-acquiring, it should be possible to finish this up fairly soon. However, the flight firmware looks great, so we re hoping to get these done in time to sell by the end of November. TeleMini is looking great from a hardware perspective, but the firmware needs work. Once the firmware is running, we ll need to make enough test flights to shake out any remaining issues before moving forward with it. EasyMini is also looking finished; I ve got a stack of prototypes and will be getting people to fly them at my local launch in another couple of weeks. The plan here is to build a small batch by hand and get them into the store once we re finished testing, using those to gauge interest before we pay for a larger production run.

Yesterday, I (with my X2Go upstream hat on) have added a new (long waited for) feature to PyHoca-GUI / Python X2Go. So far, in PyHoca-GUI (python-x2go) it was only possible to use passphrase-protected / encrypted SSH keys via an ssh-agent process that unlocks those keys when being added to the agent's keyring previous to logging into an X2Go Server. Unlocking those keys natively in PyHoca-GUI was not possible so far.
-> Now it is!!! This feature will be available with PyHoca-GUI 0.4.0.9 / Python X2Go 0.4.0.9 (and PyHoca-CLI 0.4.0.3). read more

Did you know that you can get a user's home directory with the "~username" tilde expansion? You probably do, but how about other tilde expansions?

Thanks to a few bashisms you can make your script even more difficult to read by using "~+" to get $PWD, and "~-" to get $OLDPWD.

Are you using bash and want to access the directories stack created when using the pushd and popd bashisms? The ~i (where i is an integer) expansion can give you that. It gives you forward (~+i) and backward (~-i) access to the directories stack.

But beware, if the directories stack is smaller than the number you used, there won't be any expansion.

When using tildes in shell scripts, make sure you quote to avoid unwanted expansions. Note that the posh shell in wheezy and older do support those non-POSIX expansions.

in the first examples the tilde is quoted for easier reading; the expansion doesn't occur if the tilde is quoted.

I think of Joshua often, and wonder how life might be different were he with us today. The long day of his delivery is one of the most poignant memories that I will carry with me forever. This year is the first year I've been back in Australia for Father's Day. Father's Day in the US is in June, so it's never really been the constant reminder that it will be back here (although I guess it'll vary year to year). The US Labor Day long weekend tended to be the trigger holiday for me. It's also the first year I've been separated, so this Father's Day was particularly difficult for me, but I found I enjoyed the day with Zoe all the more for it. Now that I'm no longer in the US, I miss seeing Eric and Katie's daughter, as she was my physical yardstick of how big Joshua would be today.

Last night at the Annual General Meeting we had a motion to disincorporate The Linux Users of Victoria (LUV) [1]. The proposal was for LUV to cease being an incorporated society on condition that Linux Australia (LA) [2] accepts us as a sub-committee. As a sub-committee of LA we would elect our own committee to run things locally but have LA hold the finances, deal with all the paperwork that the government demands, and generally do as many of the non-core tasks associated with running a users group as possible. When we discussed this at the LUV committee meetings it didn t seem like a big deal. But as is often the case with political discussions it turned out to be difficult. There was a lot of discussion about LUV supposedly ceasing to exist, people seem to think that LUV is defined by having an incorporated society. My impression was always that it was defined by a mailing list and having meetings and I was involved in both before there was an incorporated society. Lurkers and Ownership During the discussion we had some input from members who were typically lurkers who seemed to feel that their property rights towards LUV were being infringed, this annoys me. I think that if someone chooses not to be involved in running an organisation then they should choose not to concern themselves with the details of how the organisation is to be run. People who attend the meetings should have a say in how the meetings are run and have reason to be concerned about anything that might affect them and the opinions of speakers also matter. People who are involved with mailing list discussions should have a say in how the lists are run. But people who have never volunteered for a position on the committee shouldn t be greatly concerned about the internal issues of how things are run. Finance Some concern was expressed about the financial situation of LUV and whether we would still get enough donations to keep it running when combined with LA. There was even some FUD suggesting that LA would just take our money (they had assured us that all funds and donations would be ear-marked for us). The current LUV financial situation is that Red Hat pays for the venue for the monthly meetings and the rent for the venue comprises about 2/3 of all donations. The remaining 1/3 comes from one company. So in the current situation if Red Hat ceased donating then we would have 18 months to find another donor or cease holding meetings before our bank balance became unreasonably low. If the company which gives the other significant annual donation was to cease doing so then we could operate for a few years on savings but we would need to find some other source of funding. It seems to me that joining LA would give us more financial security. Then if Red Hat ceased paying for the venue then LA could keep things running until we found another donor, I m confident that LA wouldn t allow LUV to just shut down because of a shortage of donations. If people are really concerned about the financial situation of LUV then they should urgently seek further donations such that if any one donor decided to stop giving then we could still operate as normal. To achieve that goal I think we need at least another $1,000 per annum. This issue of redundancy in donations is something I raise every time that LUV finances are discussed. My conclusion is that people aren t really bothered about the financial security of LUV except when they are looking for reasons to avoid change. Doing New Things During the course of discussion about the future of LUV there were a number of requests for improvement. One significant request was for more support for regional Linux users. Some years ago we held a mini-conference in Ballarat which went well. I think it would be good to do such things again, the cost is not particularly great and I m sure it would be accepted by LA for funding, but we need to organise it. Organising such events is something that anyone can do. Any LUV member can plan an event, get costs for everything that is needed (food, accommodation, travel, etc) and then pitch it to the LUV committee in terms of which things should be paid by LUV and which by the members concerned. We could then work on getting additional funding from LA if necessary. But planning an event takes some effort and it s often effort that can only be done by a local. Finding a suitable venue and getting some assurance that a large enough audience will attend is something that can t be done remotely. I think that the problem for LUV in regard to such things isn t a lack of money or independence. I think that the problem is that the committee spends too much volunteer time on administrative tasks and not enough time directly doing things that benefit members and the community in general. In the past I have declined nomination to the LUV committee because I felt that I could contribute more by giving lectures, finding other speakers, and doing other things to directly improve the group. I was on the committee last year and have now been elected to it again, but I m starting to think that I made a mistake. Maybe I should have declined and let others work on the new model rules and other paperwork. One committee member has claimed that the time taken on administrative tasks isn t taking time away from other LUV related tasks, I invite any committee members who feel that way to address some of the services that members are requesting. Speaking for myself my lack of time directly impacts that I can do for the club. I think that ownership of a club should be related to what people do for the club. If you have a feeling of ownership and lack ideas for how to contribute then you can ask the LUV mailing list, there are lots of people with suggestions for things to do.

• [1] http://luv.asn.au/
• [2] http://linux.org.au/

The Garmin 910XT is the premier GPS watch for triathlon training and racing.

1. You'll probably need a newer version of pyusb:

   $ git clone git://github.com/walac/pyusb
   

   There is no need to install it systemwide - you can simply leave it checked out somewhere.

2. Get the Garmin-Forerunner-610-Extractor project from GitHub:

   $ git clone git://github.com/Tigge/Garmin-Forerunner-610-Extractor.git
   

3. Turn your watch on and plug in the USB adaptor.

4. You can now sync your activities:

   $ sudo env PYTHONPATH=/path/to/pyusb ./garmin.py
   Request basic information...
     ANT version:   AJK1.04RAF
     Capabilities:  array('B', [8, 3, 0, 186, 54, 0])
     Serial number: 3480039
   Starting system...
   Searching...
   Authenticating with Forerunner 910XT (3863343203)
    - Passkey: OK
   Downloading 1 file(s)
   Downloading 2013-09-03_21-09-32-80-19364.fit [..........]
   

   The first time you sync, you will have to authenticate on the watch itself. Downloading the activities will be much slower than you expect.

5. After the syncing has finished, don't forget to turn the watch off. If you are used to the Garmin Edge 800 (which turns itself off when you insert the USB cable) you will probably not have the habit of turning off the unit.

6. Rather confusingly, your .FIT files are available in the Xdg configuration folder. For example:

   ls -l ~/.config/garmin-extractor/3863343203/activities/
   -rw-r--r-- 1 root root   2688 Jun  9 15:23 2013-06-08_12-33-24-80-2688.fit
   -rw-r--r-- 1 root root   1171 Jun  9 15:23 2013-06-08_12-37-14-80-1171.fit
   -rw-r--r-- 1 root root   6111 Jun  9 15:23 2013-06-08_22-19-28-80-6111.fit
   -rw-r--r-- 1 root root 170327 Jun  9 15:26 2013-06-09_11-09-44-80-170327.fit
   

7. Strava supports these .fit files through the From File upload option.

The Garmin 910XT is the premier GPS watch for triathlon training and racing.

1. You'll probably need a newer version of pyusb:

   $ git clone git://github.com/walac/pyusb
   

   There is no need to install it systemwide - you can simply leave it checked out somewhere.

2. Get the Garmin-Forerunner-610-Extractor project from GitHub:

   $ git clone git://github.com/Tigge/Garmin-Forerunner-610-Extractor.git
   

3. Turn your watch on and plug in the USB adaptor.

4. You can now sync your activities:

   $ sudo env PYTHONPATH=/path/to/pyusb ./garmin.py
   Request basic information...
     ANT version:   AJK1.04RAF
     Capabilities:  array('B', [8, 3, 0, 186, 54, 0])
     Serial number: 3480039
   Starting system...
   Searching...
   Authenticating with Forerunner 910XT (3863343203)
    - Passkey: OK
   Downloading 1 file(s)
   Downloading 2013-09-03_21-09-32-80-19364.fit [..........]
   

   The first time you sync, you will have to authenticate on the watch itself. Downloading the activities will be much slower than you expect.

5. After the syncing has finished, don't forget to turn the watch off. If you are used to the Garmin Edge 800 (which turns itself off when you insert the USB cable) you will probably not have the habit of turning off the unit.

6. Rather confusingly, your .FIT files are available in the Xdg configuration folder. For example:

   ls -l ~/.config/garmin-extractor/3863343203/activities/
   -rw-r--r-- 1 root root   2688 Jun  9 15:23 2013-06-08_12-33-24-80-2688.fit
   -rw-r--r-- 1 root root   1171 Jun  9 15:23 2013-06-08_12-37-14-80-1171.fit
   -rw-r--r-- 1 root root   6111 Jun  9 15:23 2013-06-08_22-19-28-80-6111.fit
   -rw-r--r-- 1 root root 170327 Jun  9 15:26 2013-06-09_11-09-44-80-170327.fit
   

7. Strava supports these .fit files through the From File upload option.

Some of you might have noticed there is a new nagios-plugins version in the row. To support upstream in evaluating bugs and bringing the code to a wider audience, I uploaded the latest development version into Debian unstable. Since this package version contrib/ is dropped by upstream. But we stopped shipping the latest plugins of contrib/ with Debian wheezy. We also removed perlmods/ from the upstream tarball, as this was never used by our package. The packaging changes are:

* New upstream git snapshot (b15adb7762)
  - Adjust debian/copyright
* Droped the following upstream integrated patches
  - 05_fix_gets_undefined_in_iso_c11.dpatch
  - 10_check_apt_perfdata.dpatch
  - 11_check_nt_npe.dpatch
  - 12_check_smtp_double_threshold.dpatch
* Fix removing empty /usr/include
* Add new check_dbi into nagios-plugins-standard
* Remove whois from Recommands, as check_bgpstate is removed
* Cleanup debian/rules from unused stuff
* Provide information about repackaging
  - Extend debian/README.source
  - Add debian/bin/repack.sh
* Updating standards version to 3.9.4 (no changes needed)
* Add libfreeradius-client-dev to build-deps and remove libradiusclient-ng-dev
  (Closes: #721621)
* Remove traces of contrib/ from debian/copyright
* Remove command.cfg from examples in nagios-plugins-common, it was removed
  upstream

The Debian Nagios Maintainer and Upstream Developer are asking you to try out this new package on non production systems. Not that we think there might be a high risk for damage, but you know .. :) The latest version of the plugins are running on several systems already, but it is nearly impossible to test all scenarios even automated. If you find bugs, please report all packaging bugs into the BTS. Upstream bugs you might consider to report in the upstream bugtracker. If you want to give it a try on Debian wheezy, you could install the packages from our wheezy-backports repository:

    # wget "http://ftp.cyconet.org/debian/sources.list.d/wheezy-backports-cyconet.list" \
        -O /etc/apt/sources.list.d/wheezy-backports-cyconet.list
    # aptitude update
    # aptitude -t wheezy-backports install nagios-plugins

For adding our archive key, you can just install the package "debian-cyconet-archive-keyring"

A work-related blog post for a change, but still about Haskell. My student Alexander Kuhnle has submitted his bachelor thesis Modeling Uncertain Data using Monads and an Application to the Sequence Alignment Problem today, in which he explores way to generalize algorithms from bio informatics (in particular suffix tree) to work on data with uncertainties of various kinds (indeterminism, probabilities etc.). He utilizes monads and variations thereof to make the code polymorphic in the particular kind of uncertainty. If this sounds interesting to you, have a look, and feel free to share your optinion with us.

The Google Summer of Code rewriting the Package Tracking System is approaching its end and I m starting to think about deploying it on debian.org. Its scope has expanded over the years and the rewritten PTS will continue this trend by bringing some new features for teams (like the possibility to subscribe to all packages of a team). I believe that its current hostname (and name) doesn t reflect properly the role of the PTS. Add to this the fact that there s still some work left to be done to reach feature-parity with the current PTS, I m considering deploying it in parallel to the current PTS under a new name. Package Tracking System is also a bit too long for a name, and sounds more like a description than a name But if I get rid of packages.qa.debian.org and Package Tracking System , how should we call the new PTS? The PTS is a sort of central place that brings together information from many parts of Debian. It s currently mainly a consumer/dispatcher of information but I expect to integrate some of the external services that are useful for all Debian derivatives, and it will thus become more and more a producer of first-hand information as well. To replace packages.qa.debian.org, Stefano Zacchiroli suggested me hub.debian.org and I must say I like it, it s short and relatively close to what the PTS actually is (and reminds me of DEP-2 the new PTS will be an asset to make it a reality). My other ideas were devel.debian.org, inside.debian.org, watch.debian.org, track.debian.org, do you have better suggestions? what s your preference? Finding a better name is harder, but there s room to build on the hub concept and similar images. I would like a full name that s not too long and an associated abbreviation/short name for the top-level Python package (currently we use pts for that Python package). Can you come up with something original and satisfactory? My latest thoughts end up with DistroHub as full name and dhub as Python package name. Still boring So, dear lazy web, I heard that we re good at bikeshedding in Debian, so can you come up with something better? Share your suggestions in the comments!

31 comments Liked this article? Click here. My blog is Flattr-enabled.

I've been hacking on Python for a lot of years now, on various project. For the last two years, I've beenheavily involved in OpenStack, which makes an heavy usage of Python. Once you start working with a hundred of hackers, on several software and libraries representing more than half a million source lines of Python, things change. The scalability, testing and deployment problems inherent to a cloud platform meddle with everything in designing components. During these two years working on OpenStack development, I've learned a lot on Python from astounding Python hackers. From general architecture and design principles to various tips and tricks of the language. It seemed to me like a good opportunity to share what I learnt doing so with others so you can benefit from it in other projects. I've started working a book, entitled "The Hacker Guide to Python", where I will try to share what I learnt while working with Python. The book is still a work in progress at this stage, but if you'd like to get in touch and keep updated on its advancement, you can subscribe in the following form or from the book homepage.