There are times when I consider launching my own company again, most often when it is late at night and the inpetitude of so many other companies gets me too worked up. Then I sit back and think about details and write it off. I've worked for myself in the past a couple of times, and each time it was both more fun and more difficult than expected. Getting a couple of clients is usually easy, getting a ten more is common, but getting "many" is hard and getting "lots" is something I've never done - lots of users for free sites though, along with the associated support burdon! So the though dies away once I sit down and work out the net profit I'd need to live. My expenses are low, so let us pretend I can easily live on 1000 a month. So the "company" has to make more than that, to cover costs, but perhaps not much. Pretend you were offering DNS hosting you'd probably be able to implement that easily on, say 10, virtual machines, net of 150 a month. Imagine clients pay 5 for an unlimited number of domains that means you need to have 1000+150/5 = 230 clients. Not impossible, but also not easy. Pretend instead you're offering backup space, and the numbers get bigger because disk is expensive. Again getting some users would be easy, but getting lots would be hard because your competition is dropbox, skydrive, etc, etc. Once you start thinking of "ideas" they come easily, but the hard part is being realistic about what people would pay for. As always the idea is the easy part, the execution is the hardest part. Realistically if I were to be desperate to work for myself at short notic I'd do the obvious thing - I'd buy a pair of ladders, a bucket, and clean windows. Low overheads, reasonable demand, and I'd be both "fit" and "outdoors". When it comes to paying for online services off the top of my head I personally pay for maybe two things, both of them niche (although profitable for their providers I'm sure), and I know many people who live on the internet but pay for nothing.

For example I'm a VIP member of an online modeling community, which in theory allows me a higher chance of persuading interesting people to pose for me. In practice the turnover on those sites is immense. Lots of cute boys and girls hear constantly "You're so pretty, you should be a model", which is true in perhaps 1% of cases, and the net result is you have a few hard working people who do good things day in day out, and many flighty teenagers who'll pose for two-three people, and then never do it again because they realise it is neither glamourous nor easy money.

Two things I've semi-serously considered recently where hosted "status pages", and hosted "domain parking", but both have many competitors and both I can see a) some people would pay for but b) not very many. I suspect there is no universal "I'd pay for this" online service hwich is both competition free and genuinely trivial to setup, but I'd be curious to see what people are missing, and even more curious to see what people do pay for.

My right ankle has never been quite right ever since I sprained it in 2008. The ankle is always a bit enlarged, sometimes more so after running. With all the running I've been doing back in Brisbane, I had this niggling concern in the back of my head that I might be doing myself some long term damage, so last year I packed myself off to see a physiotherapist that was within walking distance of home. He declared that I had blown out two of the ligaments in my foot from the sprain, and that my calf was compensating for it a bit. The swelling was just scar tissue. He gave me a bunch of exercises to do, and I had some electrotherapy on my calf for a few weeks and I went on my way. I think I'm supposed to do the exercises every time before I run, but that hasn't happened. It's hard enough to make the time to run, without having to prepend a bunch of exercises as well. One of my neighbours works for my FootDr, so I asked her if they did video run gait analysis, because I was interested in seeing if there was anything I could do to improve my running gait. They did, and so I finally got around to making an appointment for today. The appointment wasn't as interesting as the one I had in the US, in that there was no fluoroscopy, just manual examination, but there was the video run gait analysis on the treadmill, which was not something I've had before. I was pleased that my running shoes (New Balance 940's) were deemed correct for my situation. The guy at Sports Basement obviously did a good job of fitting them to me. I was advised to refresh the shoes, though, so I lashed out on a pair of 940 v2's today. My health insurance also paid pretty generously for orthotics, so I'm going to get a pair for them to add to the ankle stability. The results of the run gait analysis were that, surprise surprise, I am rolling inwards a bit more on my right ankle than my left. Interestingly, my left calf was bigger than my right one, so I was advised to stick with the physiotherapy exercises (yay). So with the orthotics, new shoes, and forcing myself to do my physiotherapy exercises, I'm pretty satisfied that my continued running isn't likely to do too much damage to my body. The podiatrist concurred that because I'm currently pain free, everything should be good. I just had to live with the fact that my right ankle is going to always be a bit swollen.

TL;DR: Do not enable net.ipv4.tcp_tw_recycle. The Linux kernel documentation is not very helpful about what net.ipv4.tcp_tw_recycle does:

Enable fast recycling TIME-WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts.

Its sibling, net.ipv4.tcp_tw_reuse is a little bit more documented but the language is about the same:

Allow to reuse TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. Default value is 0. It should not be changed without advice/request of technical experts.

The mere result of this lack of documentation is that we find numerous tuning guides advising to set both these settings to 1 to reduce the number of entries in the TIME-WAIT state. However, as stated by tcp(7) manual page, the net.ipv4.tcp_tw_recycle option is quite problematic for public-facing servers as it won t handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you:

Enable fast recycling of TIME-WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).

I will provide here a more detailed explanation in the hope to teach people who are wrong on the Internet. As a sidenote, despite the use of ipv4 in its name, the net.ipv4.tcp_tw_recycle control also applies to IPv6. Also, keep in mind we are looking at the TCP stack of Linux. This is completely unrelated to Netfilter connection tracking which may be tweaked in other ways¹.

About TIME-WAIT state Let s rewind a bit and have a close look at this TIME-WAIT state. What is it? See the TCP state diagram below²: Only the end closing the connection first will reach the TIME-WAIT state. The other end will follow a path which usually permits to quickly get rid of the connection. You can have a look at the current state of connections with ss -tan:

$ ss -tan   head -5
LISTEN     0  511             *:80              *:*     
SYN-RECV   0  0     192.0.2.145:80    203.0.113.5:35449
SYN-RECV   0  0     192.0.2.145:80   203.0.113.27:53599
ESTAB      0  0     192.0.2.145:80   203.0.113.27:33605
TIME-WAIT  0  0     192.0.2.145:80   203.0.113.47:50685

Purpose There are two purposes for the TIME-WAIT state:

• The most known one is to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows. RFC 1337 explains in details what happens when the TIME-WAIT state is deficient³. Here is an example of what could be avoided if the TIME-WAIT state wasn t shortened:

• The other purpose is to ensure the remote end has closed the connection. When the last ACK is lost, the remote end stays in the LAST-ACK state⁴. Without the TIME-WAIT state, a connection could be reopened while the remote end still thinks the previous connection is valid. When it receives a SYN segment (and the sequence number matches), it will answer with a RST as it is not expecting such a segment. The new connection will be aborted with an error:

RFC 793 requires the TIME-WAIT state to last twice the time of the MSL. On Linux, this duration is not tunable and is defined in include/net/tcp.h as one minute:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
                                  * state, about 60 seconds     */

There have been propositions to turn this into a tunable value but it has been refused on the ground the TIME-WAIT state is a good thing.

Problems Now, let s see why this state can be annoying on a server handling a lot of connections. There are three aspects of the problem:

• the slot taken in the connection table preventing new connections of the same kind,
• the memory occupied by the socket structure in the kernel, and
• the additional CPU usage.

The result of ss -tan state time-wait wc -l is not a problem per se!

Connection table slot A connection in the TIME-WAIT state is kept for one minute in the connection table. This means, another connection with the same quadruplet (source address, source port, destination address, destination port) cannot exist. For a web server, the destination address and the destination port are likely to be constant. If your web server is behind a L7 load-balancer, the source address will also be constant. On Linux, the client port is by default allocated in a port range of about 30,000 ports (this can be changed by tuning net.ipv4.ip_local_port_range). This means that only 30,000 connections can be established between the web server and the load-balancer every minute, so about 500 connections per second. If the TIME-WAIT sockets are on the client side, such a situation is easy to detect. The call to connect() will return EADDRNOTAVAIL and the application will log some error message about that. On the server side, this is more complex as there is no log and no counter to rely on. In doubt, you should just try to come with something sensible to list the number of used quadruplets:

$ ss -tan 'sport = :80'   awk ' print $(NF)" "$(NF-1) '   \
>     sed 's/:[^ ]*//g'   sort   uniq -c
    696 10.24.2.30 10.33.1.64
   1881 10.24.2.30 10.33.1.65
   5314 10.24.2.30 10.33.1.66
   5293 10.24.2.30 10.33.1.67
   3387 10.24.2.30 10.33.1.68
   2663 10.24.2.30 10.33.1.69
   1129 10.24.2.30 10.33.1.70
  10536 10.24.2.30 10.33.1.73

The solution is more quadruplets⁵. This can be done in several ways (in the order of difficulty to setup):

• use more client ports by setting net.ipv4.ip_local_port_range to a wider range,
• use more server ports by asking the web server to listen to several additional ports (81, 82, 83, ),
• use more client IP by configuring additional IP on the load balancer and use them in a round-robin fashion,
• use more server IP by configuring additional IP on the web server⁶.

Of course, a last solution is to tweak net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle. Don t do that yet, we will cover those settings later.

Memory With many connections to handle, leaving a socket open for one additional minute may cost your server some memory. For example, if you want to handle about 10,000 new connections per second, you will have about 600,000 sockets in the TIME-WAIT state. How much memory does it represent? Not that much! First, from the application point of view, a TIME-WAIT socket does not consume any memory: the socket has been closed. In the kernel, a TIME-WAIT socket is present in three structures (for three different purposes):

1. A hash table of connections, named the TCP established hash table (despite containing connections in other states) is used to locate an existing connection, for example when receiving a new segment. Each bucket of this hash table contains both a list of connections in the TIME-WAIT state and a list of regular active connections. The size of the hash table depends on the system memory and is printed at boot:

   $ dmesg   grep "TCP established hash table"
   [    0.169348] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
   

   It is possible to override it by specifying the number of entries on the kernel command line with the thash_entries parameter. Each element of the list of connections in the TIME-WAIT state is a struct tcp_timewait_sock, while the type for other states is struct tcp_sock⁷:

   struct tcp_timewait_sock  
       struct inet_timewait_sock tw_sk;
       u32    tw_rcv_nxt;
       u32    tw_snd_nxt;
       u32    tw_rcv_wnd;
       u32    tw_ts_offset;
       u32    tw_ts_recent;
       long   tw_ts_recent_stamp;
    ;
   struct inet_timewait_sock  
       struct sock_common  __tw_common;
       int                     tw_timeout;
       volatile unsigned char  tw_substate;
       unsigned char           tw_rcv_wscale;
       __be16 tw_sport;
       unsigned int tw_ipv6only     : 1,
                    tw_transparent  : 1,
                    tw_pad          : 6,
                    tw_tos          : 8,
                    tw_ipv6_offset  : 16;
       unsigned long            tw_ttd;
       struct inet_bind_bucket *tw_tb;
       struct hlist_node        tw_death_node;
    ;
   

2. A set of lists of connections, called the death row , is used to expire the connections in the TIME-WAIT state. They are ordered by how much time left before expiration. It uses the same memory space as for the entries in the hash table of connections. This is the struct hlist_node tw_death_node member of struct inet_timewait_sock.
3. A hash table of bound ports, holding the locally bound ports and the associated parameters, is used to determine if it is safe to listen to a given port or to find a free port in the case of dynamic bind. The size of this hash table is the same as the size of the hash table of connections:

   $ dmesg   grep "TCP bind hash table"
   [    0.169962] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
   

   Each element is a struct inet_bind_socket. There is one element for each locally bound port. A TIME-WAIT connection to a web server is locally bound to the port 80 and shares the same entry as its sibling TIME-WAIT connections. On the other hand, a connection to a remote service is locally bound to some random port and does not share its entry.

So, we are only concerned by the space occupied by struct tcp_timewait_sock and struct inet_bind_socket. There is one struct tcp_timewait_sock for each connection in the TIME-WAIT state, inbound or outbound. There is one dedicated struct inet_bind_socket for each outbound connection and none for an inbound connection. A struct tcp_timewait_sock is only 168 bytes while a struct inet_bind_socket is 48 bytes:

$ sudo apt-get install linux-image-$(uname -r)-dbg
[...]
$ gdb /usr/lib/debug/boot/vmlinux-$(uname -r)
(gdb) print sizeof(struct tcp_timewait_sock)
 $1 = 168
(gdb) print sizeof(struct tcp_sock)
 $2 = 1776
(gdb) print sizeof(struct inet_bind_bucket)
 $3 = 48

So, if you have about 40,000 inbound connections in the TIME-WAIT state, it should eat less than 10MB of memory. If you have about 40,000 outbound connections in the TIME-WAIT state, you need to account for 2.5MB of additional memory. Let s check that by looking at the output of slabtop. Here is the result on a server with about 50,000 connections in the TIME-WAIT state, 45,000 of which are outbound connections:

$ sudo slabtop -o   grep -E '(^  OBJS tw_sock_TCP tcp_bind_bucket)'
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
 50955  49725  97%    0.25K   3397       15     13588K tw_sock_TCP            
 44840  36556  81%    0.06K    760       59      3040K tcp_bind_bucket

There is nothing to change here: the memory used by TIME-WAIT connections is really small. If your server need to handle thousands of new connections per second, you need far more memory to be able to efficiently push data to clients. The overhead of TIME-WAIT connections is negligible.

CPU On the CPU side, searching for a free local port can be a bit expensive. The work is done by the inet_csk_get_port() function which uses a lock and iterate on locally bound ports until a free port is found. A large number of entries in this hash table is usually not a problem if you have a lot of outbound connections in the TIME-WAIT state (like ephemeral connections to a memcached server): the connections usually share the same profile, the function will quickly find a free port as it iterates on them sequentially.

Other solutions If you still think you have a problem with TIME-WAIT connections after reading the previous section, there are three additional solutions to solve them:

• disable socket lingering,
• net.ipv4.tcp_tw_reuse, and
• net.ipv4.tcp_tw_recycle.

Socket lingering When close() is called, any remaining data in the kernel buffers will be sent in the background and the socket will eventually transition to the TIME-WAIT state. The application can continue to work immediatly and assume that all data will eventually be safely delivered. However, an application can choose to disable this behaviour, known as socket lingering. There are two flavors:

1. In the first one, any remaining data will be discarded and instead of closing the connection with the normal four-packet connection termination sequence, the connection will be closed with a RST (and therefore, the peer will detect an error) and will be immediatly destroyed. No TIME-WAIT state in this case.
2. With the second flavor, if there is any data still remaining in the socket send buffer, the process will sleep when calling close() until either all the data is sent and acknowledged by the peer or the configured linger timer expires. It is possible for a process to not sleep by setting the socket as non-blocking. In this case, the same process happens in the background. It permits the remaining data to be sent during a configured timeout but if the data is succesfully sent, the normal close sequence is run and you get a TIME-WAIT state. And on the other case, you ll get the connection close with a RST and the remaining data is discarded.

In both cases, disabling socket lingering is not a one-size-fits-all solution. It may be used by some applications like HAProxy or Nginx when it is safe to use from the upper protocol point of view. There are good reasons to not disable it unconditionnaly.

net.ipv4.tcp_tw_reuse The TIME-WAIT state prevents delayed segments to be accepted in an unrelated connection. However, on certain conditions, it is possible to assume a new connection s segment cannot be misinterpreted with an old connection s segment. RFC 1323 presents a set of TCP extensions to improve performance over high-bandwidth paths. Among other things, it defines a new TCP option carrying two four-byte timestamp fields. The first one is the current value of the timestamp clock of the TCP sending the option while the second one is the most recent timestamp received from the remote host. By enabling net.ipv4.tcp_tw_reuse, Linux will reuse an existing connection in the TIME-WAIT state for a new outgoing connection if the new timestamp is strictly bigger than the most recent timestamp recorded for the previous connection: an outgoing connection in the TIME-WAIT state can be reused after just one second. How is it safe? The first purpose of the TIME-WAIT state was to avoid duplicate segments to be accepted in an unrelated connection. Thanks to the use of timestamps, such a duplicate segments will come with an outdated timestamp and therefore be discarded. The second purpose was to ensure the remote end is not in the LAST-ACK state because of the lost of the last ACK. The remote end will retransmit the FIN segment until:

1. it gives up (and tear down the connection), or
2. it receives the ACK it is waiting (and tear down the connection), or
3. it receives a RST (and tear down the connection).

If the FIN segments are received in a timely manner, the local end socket will still be in the TIME-WAIT state and the expected ACK segments will be sent. Once a new connection replaces the TIME-WAIT entry, the SYN segment of the new connection is ignored (thanks to the timestamps) and won t be answered by a RST but only by a retransmission of the FIN segment. The FIN segment will then be answered with a RST (because the local connection is in the SYN-SENT state) which will allow the transition out of the LAST-ACK state. The initial SYN segment will eventually be resent (after one second) because there was no answer and the connection will be established without apparent error, except a slight delay: It should be noted that when a connection is reused, the TWRecycled counter is increased (despite its name).

net.ipv4.tcp_tw_recycle This mechanism also relies on the timestamp option but affects both incoming and outgoing connections which is handy when the server usually closes the connection first⁸. The TIME-WAIT state is scheduled to expire sooner: it will be removed after the retransmission timeout (RTO) interval which is computed from the RTT and its variance. You can spot the appropriate values for a living connection with the ss command:

$ ss --info  sport = :2112 dport = :4057
State      Recv-Q Send-Q    Local Address:Port        Peer Address:Port   
ESTAB      0      1831936   10.47.0.113:2112          10.65.1.42:4057    
         cubic wscale:7,7 rto:564 rtt:352.5/4 ato:40 cwnd:386 ssthresh:200 send 4.5Mbps rcv_space:5792

To keep the same guarantees the TIME-WAIT state was providing, while reducing the expiration timer, when a connection enters the TIME-WAIT state, the latest timestamp is remembered in a dedicated structure containing various metrics for previous known destinations. Then, Linux will drop any segment from the remote host whose timestamp is not strictly bigger than the latest recorded timestamp, unless the TIME-WAIT state would have expired:

if (tmp_opt.saw_tstamp &&
    tcp_death_row.sysctl_tw_recycle &&
    (dst = inet_csk_route_req(sk, &fl4, req, want_cookie)) != NULL &&
    fl4.daddr == saddr &&
    (peer = rt_get_peer((struct rtable *)dst, fl4.daddr)) != NULL)  
        inet_peer_refcheck(peer);
        if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
            (s32)(peer->tcp_ts - req->ts_recent) >
                                        TCP_PAWS_WINDOW)  
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
         
 

When the remote host is in fact a NAT device, the condition on timestamps will forbid allof the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems. The LAST-ACK state is handled in the exact same way as for net.ipv4.tcp_tw_recycle.

Summary The universal solution is to increase the number of possible quadruplets by using, for example, more server ports. This will allow you to not exhaust the possible connections with TIME-WAIT entries. On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are pretty sure you will never have NAT devices in the mix. Enabling net.ipv4.tcp_tw_reuse is useless for incoming connections. On the client side, enabling net.ipv4.tcp_tw_reuse is another almost-safe solution. Enabling net.ipv4.tcp_tw_recycle in addition to net.ipv4.tcp_tw_reuse is mostly useless. And a final quote by W. Richard Stevens, in Unix Network Programming:

The TIME_WAIT state is our friend and is there to help us (i.e., to let old duplicate segments expire in the network). Instead of trying to avoid the state, we should understand it.

---

1. Notably, fiddling with net.netfilter.nf_conntrack_tcp_timeout_time_wait won t change anything on how the TCP stack will handle the TIME-WAIT state.
2. This diagram is licensed under the LaTeX Project Public License 1.3. The original file is available on this page.
3. The first work-around proposed in RFC 1337 is to ignore RST segments in the TIME-WAIT state. This behaviour is controlled by net.ipv4.rfc1337 which is not enabled by default on Linux because this is not a complete solution to the problem described in the RFC.
4. While in the LAST-ACK state, a connection will retransmit the last FIN segment until it gets the expected ACK segment. Therfore, it is unlikely we stay long in this state.
5. On the client side, older kernels also have to find a free local tuple (source address and source port) for each outgoing connection. Increasing the number of server ports or IP won t help in this case. Linux 3.2 is recent enough to be able to share the same local tuple for different destinations. Thanks to Willy Tarreau for his insight on this aspect.
6. This last solution may seem a bit dumb since you could just use more ports but some servers are not able to be configured this way. The before last solution can also be quite cumbersome to setup, depending on the load-balancing software, but uses less IP than the last solution.
7. The use of a dedicated memory structure for sockets in the TIME-WAIT is here since Linux 2.6.14. The struct sock_common structure is a bit more verbose and I won t copy it here.
8. When the server closes the connection first, it gets the TIME-WAIT state while the client will consider the corresponding quadruplet free and hence may reuse it for a new connection.

Piston, an Openstack-in-a-box vendor[1] are a sponsor of the Red Hat[2] Summit this year. Last week they briefly ceased to be for no publicly stated reason, although it's been sugggested that this was in response to Piston winning a contract that Red Hat was also bidding on. This situation didn't last for long - Red Hat's CTO tweeted that this was an error and that Red Hat would pay Piston's sponsorship fee for them.

To Red Hat's credit, having the CTO immediately and publicly accept responsibility and offer reparations seems like the best thing they could possibly do in the situation and demonstrates that there are members of senior management who clearly understand the importance of community collaboration to Red Hat's success. But that leaves open the question of how this happened in the first place.

Red Hat is big on collaboration. Workers get copies of the Red Hat Brand Book, an amazingly well-written description of how Red Hat depends on the wider community. New hire induction sessions stress the importance of open source and collaboration. Red Hat staff are at the heart of many vital free software projects. As far as fundamentally Getting It is concerned, Red Hat are a standard to aspire to.

Which is why something like this is somewhat unexpected. Someone in Red Hat made a deliberate choice to exclude Piston from the Summit. If the suggestion that this was because of commercial concerns is true, it's antithetical to the Red Hat Way. Piston are a contributor to upstream Openstack, just as Red Hat are. If Piston can do a better job of selling that code than Red Hat can, the lesson that Red Hat should take away is that they need to do a better job - not punish someone else for doing so.

However, it's not entirely without precedent. The most obvious example is the change to kernel packaging that happened during the RHEL 6 development cycle. Previous releases had included each individual modification that Red Hat made to the kernel as a separate patch. From RHEL 6 onward, all these patches are merged into one giant patch. This was intended to make it harder for vendors like Oracle to compete with RHEL by taking patches from upcoming RHEL point releases, backporting them to older ones and then selling that to Red Hat customers. It obviously also had the effect of hurting other distributions such as Debian who were shipping 2.6.32-based kernels - bugs that were fixed in RHEL had to be separately fixed in Debian, despite Red Hat continuing to benefit from the work Debian put into the stable 2.6.32 point releases.

It's almost three years since that argument erupted, and by and large the community seems to have accepted that the harm Oracle were doing to Red Hat (while giving almost nothing back in return) justified the change. The parallel argument in the Piston case might be that there's no reason for Red Hat to give advertising space to a company that's doing a better job of selling Red Hat's code than Red Hat are. But the two cases aren't really equal - Oracle are a massively larger vendor who take significantly more from the Linux community than they contribute back. Piston aren't.

Which brings us back to how this could have happened in the first place. The Red Hat company culture is supposed to prevent people from thinking that this kind of thing is acceptable, but in this case someone obviously did. Years of Red Hat already having strong standing in a range of open source communities may have engendered some degree of complacency and allowed some within the company to lose track of how important Red Hat's community interactions are in perpetuating that standing. This specific case may have been resolved without any further fallout, but it should really trigger an examination of whether the reality of the company culture still matches the theory. The alternative is that this kind of event becomes the norm rather than the exception, and it takes far less time to lose community goodwill than it takes to build it in the first place.

[1] And, in the spirit of full disclosure, a competitor to my current employer
[2] Furthering the spirit of full disclosure, a former employer

comments

This morning I noticed some parking bays reserved for car charging in a car park at the corner of Sydney Rd and Glenlyon St in Brunswick (near Aldi). One of the parking spots was occupied by a Plug-in Prius from GoGet [1]. I didn t even realise that you could get a plug-in Prius in Australia. The charging station is run by Charge Point [2]. The charging points are about 1.5m high and the cable is about 3cm thick (about as thick as the pipe used for filling a car with petrol), so it would charge a car much faster than could be done with a regular power point. One big problem with the Charge Point web site is that they don t give any information on pricing. They sell home charge points (which I guess means just an all-weather two-phase power point) but don t give a price for that. They sell charge points that can be used by commercially but don t give a price for them either. Also their infrastructure for billing is apparently based on companies installing charge points and setting a price for the service. Some charge points may offer free service (I guess staff car parks and some government agencies) and others will charge varying rates none of which is available on the web site. Apparently they have an online portal which gives information on such things to registered users so you have to register to discover what it costs. Of course hardly anyone is going to register before discovering the price, not even when registration is free. But while registration is free the web site demands the make and model of the electric car, so presumably one has to spend $40,000 or more on a vehicle before discovering the price and availability of charging it. Charge Point can be used as an example of how not to design a web site that promotes a service, or at least how not to promote a service that is aimed at saving money (electricity is significantly cheaper than petrol so it s of interest to people and organisations that want to save money). The Charge Point site seems to be better suited to showing that the concept can work than convincing people that they should sign up for it. It seems to me that the best thing that they could do would be to prominently display the average cost of all non-free charge points that are open to the public along with an explanation of the price of driving a desirable car (such as a plug-in Prius or a Nissan Leaf) with such an electricity cost. The contact section on the web site only has a link for careers . I don t think it s possible to get widespread use of electric vehicles without getting better information out there. It appears that Charge Point is relying on councils to do the work of promoting their business by installing their stations and reserving car parking as Moreland council has done in this case.

• [1] http://www.goget.com.au/
• [2] http://www.chargepoint.com.au/

I made some changes on http://llvm.org/apt/ for the last 2 months.

• Added trusty, Ubuntu 14.04, as a new supported distribution (on the request of Michael Larabel, Phoronix)
• Support both the stable and development version. Currently, that means that the release_34 branch and the trunk are built. So, for example, clang-3.4 and clang-3.5 can be installed.
  release_34 are only built when a new commit is submitted in this branch. trunk is built twice a day.
• Add a new package llvm- 3.4,3.5 -tools which contains various tools to build software/packages on top of llvm. Contributed by Martin Nowack in the context of Klee.
• Since a C++ 11 compiler is now mandatory, I had to force the usage of a backported gcc/g++ 4.8 (thanks Doko).
  This is the case for Ubuntu Precise (12.04), Quantal (12.10) and raring (13.04).
  The thing is that it triggers a dependency on the libstdc++ 4.8 causing the PPA to be mandatory.
  deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu $DISTRIBUTION main
  
  For now, because of the lack of backport of gcc 4.8, I am not providing support for Debian stable (wheezy).

A new minor release of RcppArmadillo is now on CRAN and in Debian. RcppArmadillo brings the Armadillo templated C++ library for linear algebra to R by means of Rcpp. This release contains both a few minor bugfixes from the 4.000 branch of Armadillo and some enhancements in RcppArmadillo itself that are related to the recent Rcpp 0.11.0 release.

Changes in RcppArmadillo version 0.4.000.4 (2014-02-19)

• Upgraded to Armadillo release Version 4.000.4
  • fix for randi() generating out-of-interval values
  • fix for saving field objects
  • workaround for a bug in the Intel compiler
• Updated for Rcpp (>= 0.11.0) by removing linking step from build process, added appropriate dependency on Rcpp
• Updated RcppArmadillo.package.skeleton function accordingly to use proper NAMESPACE import
• Rewritten rcpparma_hello_world (which is used by the RcppArmadillo.package.skeleton function) to use Rcpp Attributes, and added more examples
• Added two functions to set Armadillo's RNGs (ie the system RNG) from a given value and to randomize it, as suggested by G bor Cs rdi. Note that these do not work within RStudio (which itself also uses the same system RNG).

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Basic configuration There are a few basic things that most admins will already know (and that tiger will warn you about if you forget):

• only allow version 2 of the protocol
• disable root logins
• disable password authentication

This is what /etc/ssh/sshd_config should contain:

Protocol 2
PasswordAuthentication no
PermitRootLogin no

Whitelist approach to giving users ssh access To ensure that only a few users have ssh access to the server and that newly created users don't have it enabled by default, create a new group:

addgroup sshuser

and then add the relevant users to it:

adduser francois sshuser

Finally, add this to /etc/ssh/sshd_config:

AllowGroups sshuser

Deterring brute-force (or dictionary) attacks One way to ban attackers who try to brute-force your ssh server is to install the fail2ban package. It keeps an eye on the ssh log file (/var/log/auth.log) and temporarily blocks IP addresses after a number of failed login attempts. Another approach is to hide the ssh service using Single-Packet Authentication. I have fwknop installed on some of my servers and use small wrapper scripts to connect to them.

Using restricted shells For those users who only need an ssh account on the server in order to transfer files (using scp or rsync), it's a good idea to set their shell (via chsh) to a restricted one like rssh. Should they attempt to log into the server, these users will be greeted with the following error message:

This account is restricted by rssh.
Allowed commands: rsync 
If you believe this is in error, please contact your system administrator.
Connection to server.example.com closed.

Restricting authorized keys to certain IP addresses In addition to listing all of the public keys that are allowed to log into a user account, the ~/.ssh/authorized_keys file also allows (as the man page points out) a user to impose a number of restrictions. Perhaps the most useful option is from which allows a user to restrict the IP addresses which can login using a specific key. Here's what one of my authorized_keys looks like:

from="192.0.2.2" ssh-rsa AAAAB3Nz...zvCn bot@example

You may also want to include the following options to each entry: no-X11-forwarding, no-user-rc, no-pty, no-agent-forwarding and no-port-forwarding.

Increasing the amount of logging The first thing I'd recommend is to increase the level of verbosity in /etc/ssh/sshd_config:

LogLevel VERBOSE

which will, amongst other things, log the fingerprints of keys used to login:

sshd: Connection from 192.0.2.2 port 39671
sshd: Found matching RSA key: de:ad:be:ef:ca:fe
sshd: Postponed publickey for francois from 192.0.2.2 port 39671 ssh2 [preauth]
sshd: Accepted publickey for francois from 192.0.2.2 port 39671 ssh2 

Secondly, if you run logcheck and would like to whitelist the "Accepted publickey" messages on your server, you'll have to start by deleting the first line of /etc/logcheck/ignore.d.server/sshd. Then you can add an entry for all of the usernames and IP addresses that you expect to see. Finally, it is also possible to log all commands issued by a specific user over ssh by enabling the pam_tty_audit module in /etc/pam.d/sshd:

session required pam_tty_audit.so enable=francois

However this module is not included in wheezy and has only recently been re-added to Debian.

Identitying stolen keys One thing I'd love to have is a way to identify a stolen public key. Given the IP restrictions described above, if a public key is stolen and used from a different IP, I will see something like this in /var/log/auth.log:

sshd: Connection from 198.51.100.10 port 39492
sshd: Authentication tried for francois with correct key but not from a permitted host (host=198.51.100.10, ip=198.51.100.10).
sshd: Failed publickey for francois from 198.51.100.10 port 39492 ssh2
sshd: Connection closed by 198.51.100.10 [preauth]

So I can get the IP address of the attacker (likely to be a random VPS or a Tor exit node), but unfortunately, the key fingerprints don't appear for failed connections like they do for successful ones. So I don't know which key to revoke. Is there any way to identify which key was used in a failed login attempt or is the solution to only ever have a single public key in each authorized_keys file and create a separate user account for each user?

In my article Contributing on GitHub I recently explained that I can not use certain social features of GitHub on their website directly. I also explained that I make great use of Hub to circumvent some accessibility issues I have with GitHub. Well, it turns out that adding star, unstar, follow and unfollow to Hub was actually a lot easier then I thought. It is long ago (probably 10 years) that I touched Ruby code the last time. However, since I do learn a lot from examples, I was actually able to add what I needed just by looking at surrounding code. Well, I had to lookup how you add an element to the end of an array (push) but that was basically it. You can have a look at the final outcome in Hub pull request #490. The code resides in the social branch of my hub fork. If you already have an OAuth token for hub, you need to login to GitHub and edit your token to add the user:follow scope if you want to use the follow and unfollow commands. Interestingly, this type of operation is accessible enough to still work with Lynx! I hope this gets merged, I don't particularily like to maintain my own forks of quite obvious functionality. Anyway, problem solved, mostly. All I am missing now is the ability to delete a repository. A dangerous feature, so probably controversial. Anyway, something for another day.

Thomas Habets has blogged about using your TPM (Trusted Platform Module) for SSH authentication a few weeks ago. We worked together to get his package simple-tpm-pk11 into Debian, and it has just arrived in unstable :-). Using simple-tpm-pk11, you can let your TPM generate a key, which you then can use for SSH authentication. This key will never leave the TPM, so it is safer than having your key on the filesystem (e.g. ~/.ssh/id_rsa), since file system access is not enough to steal your key anymore. Instead, you ll need remote code execution. To use this software, first make sure your TPM is enabled in the BIOS. In my ThinkPad X200 from 2008, the TPM is called Security Chip . Afterwards, claim ownership of your TPM using tpm_takeownership -z (from the tpm-tools package) and enter a password. You will not need to enter this password for every SSH authentication later (but you may choose to set a separate password for that). Then, install simple-tpm-pk11, create a key, set it as your PKCS11Provider and install the public key on the host(s) where you want to use it:

mkdir ~/.simple-tpm-pk11
stpm-keygen -o ~/.simple-tpm-pk11/my.key
echo key my.key > ~/.simple-tpm-pk11/config
echo -e "\nHost *\n    PKCS11Provider libsimple-tpm-pk11.so" >> ~/.ssh/config
ssh-keygen -D libsimple-tpm-pk11.so   ssh shell.example.com tee -a .ssh/authorized_keys

You ll now be able to ssh into shell.example.com without having the key for that on your file system :-). In case you have any feedback about/troubles with the software, please feel free to contact Thomas directly.

Writing this because there seems to be no correct documentation on the relevant google websites and it turns out to be non-trivial. Our goal here is to unsubscribe from a ordinary google group. Mails from the google group contain the quoted footer:

-- 
You received this message because you are subscribed to the Google
Groups "FOO" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to FOO+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/FOO
For more options, visit https://groups.google.com/groups/opt_out.

Seems easy enough, so let's send a Mail to this FOO+unsubscribe address. Back comes a E-Mail:

From: FOO <FOO+unsubconfirm@googlegroups.com>
Subject: Unsubscribe request for FOO [ EJzZjpgFhDHd9seTdRA0 ]
To: Christoph Egger <christoph@example.com>
Date: Tue, 18 Feb 2014 18:55:24 +0000 (38 minutes, 53 seconds ago)
 [Leave This Group]
Visit Go 
[Start] your own group, [visit] the help center, or [report]
abuse.

So click on the [Leave This Group] link and be done? Unfortunately not. Looking at the link you notice it's called http://groups.google.com/group/FOO/subscribe -- no token and "subscribe"? I actually want to unsubscribe! And indeed, clicking gets an Interface that offers to "Enter the email address to subscribe:" + Captcha. And whatever it does, it -- of course -- doesn't unsubscribe. (My guess is, it would actually work if you had a real google account associated with that email address and were logged in to that account but there's no way of verifying this as already the first condition is false in this case) Now if you disable HTML completely for the email, a totally different content emerges:

Hello christoph@example.com,
We have received your request to unsubscribe from FOO. In order for us to complete the request, please reply to this email or visit the following confirmation URL:
http://groups.google.com/group/FOO/subscribe
If you have questions related to this or any other Google Group, visit the Help Center at http://groups.google.com/support/.
Thanks,
Google Groups

Still the non-functional link, however it also mentions a different solution: "please reply to this email" which was not present in the HTML mail at all. And it works.

The XDG Basedirectory specification proposes default homedir folders for the categories DATA (~/.local/share), CONFIG (~/.config) and CACHE (~/.cache). One category however is missing: STATE. This category has been requested several times but nothing happened.
Examples for state data are:

• history files of shells, repls, anything that uses libreadline
• logfiles
• state of application windows on exit
• recently opened files
• last time application was run
• emacs: bookmarks, ido last directories, backups, auto-save files, auto-save-list

The missing STATE category is especially annoying if you're managing your dotfiles with a VCS (e.g. via VCSH) and you care to keep your homedir tidy.
If you're as annoyed as me about the missing STATE category, please voice your opinion on the XDG mailing list.
Of course it's a very long way until applications really use such a STATE directory. But without a common standard it will never happen.

I really like to read code written by different people. I've always been the type of learner that can cover a lot of ground by looking at examples. A few of the programming tricks I acquired over time have definitely been learnt by reading programs written by other people. There are of course cases of disagrement with coding style (lets put that very mildly), so not every project is fun to read. But I tend to find projects I like enough to actually look at their source code. Now, if you get into this habit, there is also the occasional spotting of a bug. Many are really cosmetic, like typos in comments or documentation. Others are maybe more serious, who knows what can happen if you read a few lines of code written by someone you have never met in person. This is one of the original arguments of open source and free software. It basically goes like: "People will read your code and report problems." I know that happens, but I also know that there are cases of people finding relatively minor things during a small code review, which they don't report, or which get lost in some developers INBOX. I remember waiting months to see memory corruption fixes for a particular project I submitted by email to actually appear in the development repository. In some cases you really need to track your patches and prod someone to remind them about your contribution if need be. True, some of these issues are probably minor enough to not really hurt if they get lost, but I have a feel that a lot of minor things also sort of add up. This is where GitHub comes into play. We have had project hosting services since at least a decade now, but none of them (to my experience) have ever made it so easy for all involved parties to submit and merge changes into projects. Pull requests are just amazing. Well, at least for me, they are since a few months. I have a sort of mixed hate-love relationship with GitHub when it comes to a topic that is very important to me, Accessibility. The last project hosting site I really liked for their simple web interface (read, still usable with Lynx) was Google Code. That was in the days when Google still seemed to care about such things. These days, their newer services are next to unusable to me (with the tools I prefer). SourceForge was always a catastrophy. Many things were cumbersome to do, because the site is so loaded with links to all sorts of things I never need. However, at the time I was using it, all the things I really needed did work. GitHub is a different beast, in a different era of the internet. While I can read project pages and a few basic things perfectly fine, some parts of the GitHub site are simply unusable for no particular reason. For instance, I can edit (some of my) preferences and even save them. But I can not star a project, nor can I follow other developers. Invoking these links gives me a 404, which seems to indicate that some JavaScript (read, client side code execution) is required which my browser does not perform. However, I really wonder why. Because something like "I want to star this project" will eventually have to end up as a request on the server anyway, so it seems technically unnecessary to rely on Javascript for such an operation to succeed. Luckily, these days, hip projects do have an API. An GitHub is no exception. I want to emphasis that an API alone is not an accessibility fix, since you rarely have the time to implement a clinet for a random API, just to get access to a service you want to use. Sometimes you do, and an API is a great enabler in such situations, but a reference client with a rich feature-set is much better. I have to admit that I haven't evaluated any of the possible alternatives, but one client I found for GitHub's API did really boost my productivity in the last year. I am talking about Hub. It is a simple wrapper around Git that adds a few and enhances some other git commands. Most importantly, hub fork will fork the current checkout on GitHub, and hub pull-request allows you to quickly (and I really mean quickly!) open a pull request for some commits you've just done to your fork of some project. This is very helpful, and very accessible. While it does not solve all my accessibility problems with GitHub, it at least enables me to take part in the GitHub workflow. I can easily fork projects and submit pull reuqests for changes I did. And that is what I did, at least a bit, in roughly the past year. I am not sure exactly, but I think it was spring when someone made me aware of Hub. One approach would be to check my GitHub activity graph, but I probably don't have to mention that it is not accessible. Here is a list of some of my contributions to open source projects on GitHub since I discovered a good client for the GitHub API:

• pelican is the Python based static site generator which runs this blog.
• spirit_x3 is the experimental C++11 implementation of Boost.Spirit.
• music21 is a music notation processing framework for Python from the MIT. They have just recently moved to GitHub. Some of my (small) contributions to the braille music module of music21 predate the move to GitHub.
• teq (and teqqer) aim to provide an API for tracker-style sequencing and a coresponding terminal based client. To me a very exciting project. Clear separation between the Engine and the GUI is rare in the Linux Audio Landscape. Notable good exceptions are SuperCollider. But many end-user oriented tools tend to blend the functionality provided and the user interface into a single program, which makes it very hard to use if there are any accessibility issues. This is not true for teq/teqqer. teq is a tracker engine, and teqqer the coresponding reference user interface. Lets see what comes of it, the project is rather new.

To summarize: I love GitHub for what it does to the free software community. I think the simplicity (and transparency) of contributions is to the benefit of free and open source software. However, I also hate GitHub sometimes when I need to ask a friend to do relatively trivial changes for me. What I actually want, is a command-line client like Hub that makes most or even all functionality provided on the website available. True, some things like graphs are not immediately easy to adapt. But I really shouldn't need to use a modern full-fledged web browser to tell GitHub that I am interested in the activities of a certain user. Or delete a fork. Yes, thats right. I can use Hub to fork a project, but there is no way to delete it again. Sort of strange for casual small-scale contributions. I guess the workflow there is: fork, commit, pull-request, wait for merge, delete. I don't want to have all sorts of forks on GitHub that I actually don't actively use/maintain. So once a pull request was merged, and I don't have any other things I am working on right now, I actually want to get rid of the fork again. It is easy enough to do a fork should I ever need it again.

I really like to read code written by different people. I've always been the type of learner that can cover a lot of ground by looking at examples. A few of the programming tricks I acquired over time have definitely been learnt by reading programs written by other people. There are of course cases of disagrement with coding style (lets put that very mildly), so not every project is fun to read. But I tend to find projects I like enough to actually look at their source code. Now, if you get into this habit, there is also the occasional spotting of a bug. Many are really cosmetic, like typos in comments or documentation. Others are maybe more serious, who knows what can happen if you read a few lines of code written by someone you have never met in person. This is one of the original arguments of open source and free software. It basically goes like: "People will read your code and report problems." I know that happens, but I also know that there are cases of people finding relatively minor things during a small code review, which they don't report, or which get lost in some developers INBOX. I remember waiting months to see memory corruption fixes for a particular project I submitted by email to actually appear in the development repository. In some cases you really need to track your patches and prod someone to remind them about your contribution if need be. True, some of these issues are probably minor enough to not really hurt if they get lost, but I have a feel that a lot of minor things also sort of add up. This is where GitHub comes into play. We have had project hosting services since at least a decade now, but none of them (to my experience) have ever made it so easy for all involved parties to submit and merge changes into projects. Pull requests are just amazing. Well, at least for me, they are since a few months. I have a sort of mixed hate-love relationship with GitHub when it comes to a topic that is very important to me, Accessibility. The last project hosting site I really liked for their simple web interface (read, still usable with Lynx) was Google Code. That was in the days when Google still seemed to care about such things. These days, their newer services are next to unusable to me (with the tools I prefer). SourceForge was always a catastrophy. Many things were cumbersome to do, because the site is so loaded with links to all sorts of things I never need. However, at the time I was using it, all the things I really needed did work. GitHub is a different beast, in a different era of the internet. While I can read project pages and a few basic things perfectly fine, some parts of the GitHub site are simply unusable for no particular reason. For instance, I can edit (some of my) preferences and even save them. But I can not star a project, nor can I follow other developers. Invoking these links gives me a 404, which seems to indicate that some JavaScript (read, client side code execution) is required which my browser does not perform. However, I really wonder why. Because something like "I want to star this project" will eventually have to end up as a request on the server anyway, so it seems technically unnecessary to rely on Javascript for such an operation to succeed. Luckily, these days, hip projects do have an API. An GitHub is no exception. I want to emphasis that an API alone is not an accessibility fix, since you rarely have the time to implement a clinet for a random API, just to get access to a service you want to use. Sometimes you do, and an API is a great enabler in such situations, but a reference client with a rich feature-set is much better. I have to admit that I haven't evaluated any of the possible alternatives, but one client I found for GitHub's API did really boost my productivity in the last year. I am talking about Hub. It is a simple wrapper around Git that adds a few and enhances some other git commands. Most importantly, hub fork will fork the current checkout on GitHub, and hub pull-request allows you to quickly (and I really mean quickly!) open a pull request for some commits you've just done to your fork of some project. This is very helpful, and very accessible. While it does not solve all my accessibility problems with GitHub, it at least enables me to take part in the GitHub workflow. I can easily fork projects and submit pull reuqests for changes I did. And that is what I did, at least a bit, in roughly the past year. I am not sure exactly, but I think it was spring when someone made me aware of Hub. One approach would be to check my GitHub activity graph, but I probably don't have to mention that it is not accessible. Here is a list of some of my contributions to open source projects on GitHub since I discovered a good client for the GitHub API:

• pelican is the Python based static site generator which runs this blog.
• spirit_x3 is the experimental C++11 implementation of Boost.Spirit.
• music21 is a music notation processing framework for Python from the MIT. They have just recently moved to GitHub. Some of my (small) contributions to the braille music module of music21 predate the move to GitHub.
• teq (and teqqer) aim to provide an API for tracker-style sequencing and a coresponding terminal based client. To me a very exciting project. Clear separation between the Engine and the GUI is rare in the Linux Audio Landscape. Notable good exceptions are SuperCollider. But many end-user oriented tools tend to blend the functionality provided and the user interface into a single program, which makes it very hard to use if there are any accessibility issues. This is not true for teq/teqqer. teq is a tracker engine, and teqqer the coresponding reference user interface. Lets see what comes of it, the project is rather new.

To summarize: I love GitHub for what it does to the free software community. I think the simplicity (and transparency) of contributions is to the benefit of free and open source software. However, I also hate GitHub sometimes when I need to ask a friend to do relatively trivial changes for me. What I actually want, is a command-line client like Hub that makes most or even all functionality provided on the website available. True, some things like graphs are not immediately easy to adapt. But I really shouldn't need to use a modern full-fledged web browser to tell GitHub that I am interested in the activities of a certain user. Or delete a fork. Yes, thats right. I can use Hub to fork a project, but there is no way to delete it again. Sort of strange for casual small-scale contributions. I guess the workflow there is: fork, commit, pull-request, wait for merge, delete. I don't want to have all sorts of forks on GitHub that I actually don't actively use/maintain. So once a pull request was merged, and I don't have any other things I am working on right now, I actually want to get rid of the fork again. It is easy enough to do a fork should I ever need it again.

To "pedestal" an aerobar means to elevate the bars above the basebar using risers instead of using headset spacers: You might do this for a few reasons:

• The rider is in a more aerodynamic position when cornering or descending on the basebars.
• Lowers the rider's centre of gravity, improving cornering confidence.
• Less flex when climbing or accelerating under high wattage.
• Teardrop-shaped risers are more aerodynamic than a cylindrical steerer tube:

So, assuming you currently have a bike that uses headset spacers, what would be the effect on your position if you were to replace, say, a 10mm headset spacer with a 10mm armrest riser? First, let us consider the effect of removing the headset spacer. The crucial insight is that removing a 10mm spacer will not lower the stack height ie. the vertical distance relative to the bottom bracket by 10mm. This is because spacers are not oriented perpendicular to the ground; they are stacked along the steerer tube at the head tube angle of the frame. We will assume a head tube angle of 72.5 degrees. We can calculate that removing a 10mm spacer will reduce the stack height by:

sin(72.5 )   10mm = 9.54mm

... and by the same argument it will also increase the effective reach ie. the horizontal distance relative to the bottom bracket by:

cos(72.5 )   10mm = 3.01mm

Next, let us consider the impact of adding the riser. These are oriented perpendicular to the ground, so its addition makes no further change to the reach. However, we can now calculate the overall change in stack height:

  stack = -(sin(72.5 )   10mm) + 10mm = 0.46mm

We can then repeat the calculation for any length of replacement:

Replacement (mm)	stack (mm)	reach (mm)
10	+0.46	+3.01
20	+0.93	+6.01
30	+1.39	+9.02
40	+1.85	+12.0
50	+2.31	+15.0
60	+2.78	+18.0

From this table, I can discover that if I were to replace 30mm of headset spacers with 30mm of risers it would:

1. Increase my effective stack height by 1.39mm (likely neglible).
2. Increase my reach by 9.02mm. This might require me to use a shorter stem to get the same position. If that was undesirable for example, if was already using an extremely short stem I might be forced to abandon the idea altogether.

Three further considerations must be noted:

1. The resultant low height of the base bar could be quite drastic and prevent you breathing properly whilst climbing.
2. Having the steerer tube cut down after to removing headset spacers will reduce the resale value of your bike.
3. You might need to re-cable your brakes as you may have changed the distance the cables must span.

(It may seem odd to provide results for up to 60mm of headset spacers when such a large number of spacers would at the very least void one's warranty. However, I suspect such setups are transiently common within the confines of a fitting studio.)

A couple of days ago, AO3 reached 1 million fanworks in 14,353 fandoms. The AO3 is an archive of fanfiction for multiple fandoms, founded and run by a non-profit organization, Organization for Transformative Works with the aim of "providing access to and preserving the history of fanworks and fan culture in its myriad forms." The code behind AO3 is open source, their servers run Debian and they even have a Diversity Statement! How cool is that? :) And, well, there's also the fact that...

"[...] the Archive is one of a very small number of open-source projects founded and largely staffed by women, another contribution fans have made to the internet as a whole. Many women who were not previously involved in technology have learned new skills in the name of fannish activities, and the AO3 is proud to stand as a shining example of this growth."

So, congratulations AO3!

I wrote in a previous post that Youtube changed their way of delivering videos, with the use of Dynamic Adaptive Streaming over HTTP. On top of that, they started serving both the audio and the video in separate streams, which meant trouble for downloader tools like youtube-dl. As I mentioned in that previous post:

What does this mean in practical terms for users of youtube-dl? Well, if you wanted to download videos in resolutions like the 480p (format 35) that I mentioned, then you will probably have to change your way of doing things, until a more automated solution is in place. You will have to download both the audio and the video and, then, "combine" them (that is, multiplex them) to create one "normal" video file with both the audio and the video.

And latter, I wrote:

Otherwise, to download 480p videos (which I do for lectures and so on with other projects of mine, like edx-dl) I have to call youtube-dl twice: once for format 135 and another for format 140, since the old format 35 files are much smaller than the lower resolution 360p files (due to the former being encoded in High profile vs. the latter being encoded in Constrained Baseline profile).

(...) Well, now, we don't have this problem anymore: The new release of youtube-dl brings us many goodies, including that it is possible to automatically combine/merge/multiplex audio and video formats that Youtube now offers separately (See the previous comments about separate audio and video). Now, if you want a 480p video in H.264 format, High profile, with 128kbps AAC audio (this used to be Youtube's format 35), you can specify format -f 135+140 on the command-line and it will download both the audio, the video and multiplex it with ffmpeg (or avconv, depending on what you have installed). Besides being convenient, this automatic downloading and merging makes it unnecessary to write scripts to, say, retrieve all the videos in a playlist that contains a lot of lectures (see one example here, taught by Benedict Gross). Version 2014.02.17 of youtube-dl will be soon in your favorite mirror of the Debian archives.

Enrico and me have just started a census on Debian Services floating around, and we want YOU to participate!

As an introvert I like - well, I need to - spend time alone. I get my best ideas on my own. But when I truly thrive is when I am connected with others, cocreating. For the last couple of weeks it's been difficult for me to work on the visual design for the MiniDebConf in Barcelona next month. There have been practical reasons, but the real reason was that I was feeling stuck and overwhelmed. Sharing my uncomfortable feelings with my boyfriend finally made me able to see that I have been feeling alone, and feeling that I had to prove myself with my work. I felt disconnected from the people in Debian I want to cooperate with. I'd really benefit from sharing my thoughts and ideas more with others. The unpleasant feeling of "stuckness" dissolved when I said out loud what they were about, and ideas for changing my state appeared. Now I've started hanging out on the irc-channel more, I've asked for help in finding stuff in the huge pile of info, rules and previous examples and experiences at debian.org and my decision to blog and share more is strengthened. Finally the work has begun. Conferences are about meeting up and getting connected. Listening to somebody trying to enlighten you on a subject. Getting inspired. And so my idea of using speech bubbles as a visual identity koncept for the conference has everything to do with my own experience of connection and disconnection. Release, early, release often, I cannot say that I have done that. We really need the peoposals to be ready now. Thankfully I'll have time in the following days to get feedback and finish. I really hope people will accept the idea, but I'm expecting a debate.