Given that I was responsible for putting together the system
architecture for
Github when they
moved away from their previous services provider to
Anchor Systems, I keep an eye
on what's going on there even though I've gotten out of professional
IT.
Recently(ish), Ryan Tomayko blogged about some
experiments
in using Amazon CloudFront as a CDN for static assets. While
discussing as-yet unsolved performance issues, he quoted a tweet from
Shopify's CEO (who I would have thought would have known better):
Protip to make SSL fast: terminate in your load balancer, that makes SSL
session cookies actually work.
No,
no, NO -- a million times
NOOOOOO!
Load balancers are, by their very nature, performance chokepoints. You
can't horizontally scale your load balancer, because if you do (by, say,
putting another load balancer in front of it -- don't laugh, I've seen it
done) you've just re-introduced all the same problems that putting SSL on
the load balancer was supposed to accomplish (that is, getting SSL
session caching to work).
The only thing you accomplish by putting your SSL sessions onto the
load balancer is bring the moment your load balancer melts down
through excessive CPU usage
that much closer. I dunno,
maybe Shopify doesn't actually do that much traffic or something,
but I certainly wouldn't be signing up for that headache if it were
my infrastructure.
(Sidenote: yes, I know that you can get "SSL hardware accelerators",
that put the basic crypto operations into dedicated silicon, but
they're rarely cheap, and you can't horizontally scale those,
either. They're effectively a specialised and outrageously
expensive form of "scaling by bigger hardware" -- which is a valid
method of scaling, but one with fairly terminal limitations that
certainly wouldn't make it practical for Github).
The problem you're trying to solve by centralising SSL operations is
that SSL includes a mechanism whereby a client making a new TCP
connection can say "here's the crypto details from last time I
connected to you" and the server can just pull out all of the
per-connection data that would otherwise take a long time (several
TCP round trips and a smallish chunk of CPU time) to setup. This
can make SSL connections a lot snappier (in Ryan's opinion, "We
spend roughly as much time handshaking as we do generating the
average dynamic request."), so they're a definite win if you can do
it (and yes, you can do it).
Webservers don't typically do this caching by default, but setting
up a simple SSL session cache for a single
nginx
or
Apache
webserver is pretty trivial. Testing it is simple, too: just
install
gnutls-bin and run:
gnutls-cli -V -r HOSTNAME grep 'Session ID'
If all three Session IDs are the same, then you've got SSL session caching
running. (If you're wondering why I'm using
gnutls-cli rather
than
openssl s_client, it's because openssl is living in the
pre-IPv6 dark ages, and I've got IPv6-only services running I wanted to
test).
So that's great for your little single-server hosting setup, but
per-machine caches aren't much use when you're running behind a load
balancer, Github style, because each SSL-enabled TCP connection can
very well end up on a different server, and a per-machine cache
won't help you very much. (If you think the answer to this problem
is "session affinity", you're setting yourself up for a whole other
world of pain and suffering.)
Here's the kicker, though -- the SSL session is just a small, opaque
chunk of bytes (OpenSSL appears to store it internally as an ASN.1
string, but that's an irrelevant implementation detail). An SSL
session cache is just a mapping of the SSL session ID (a string) to
the SSL session data (another string). We know how to do "shared
key-value caches" -- we have the technology. There won't be too
many large-scale sites out there that aren't using memcached (or
something practically equivalent).
So, rather than stuff around trying to run a load balancer with
enough grunt to handle all your SSL communication for all time, you
can continue to horizontally scale the whole site with as many
backend webservers as you need. All that's needed is to teach your
SSL-using server to talk to a shared key-value cache.
Far be it for me to claim any sort of originality in this idea, though --
patches for Apache's mod_ssl to do this have been floating around for a
while, and according to
the
release notes, the 2.4 series will have it all built-in.
However, Github uses nginx, to great effect, and nobody's done the work
to put this feature into nginx (that I could find, anyway).
Until now.
I present, for the edification of all,
a series of patches to nginx
0.8.x to provide memcached-enabled SSL session caching. Porting
them to other versions of nginx hopefully won't be too arduous, and
if you ask nicely I might give it a shot. If you don't use
memcached (gasp!), my work should at least be a template to allow
you to hook into whatever system you prefer. Sharding the memcached
calls wouldn't be hard either, but I'm not a huge fan of that when
other options are available.
I had originally hoped to be able to say "it works at Github!" as a
testimonial, but for some reason it hasn't been deployed there yet
(go forth and pester!), so instead I'm just releasing it out there
for people to try out on a "well, I don't know that it
doesn't work" basis. If it breaks anything, it's not my
fault; feel free to supply patches if you want to fix anything I've
missed.
I've been playing with C++ lately, and it's an interesting language. Some random thoughts go here. First, in which situation does "
Umh, so Google created yet-another-beta-service with yet-another-invite-only stage. Whee! And this time they choose a name yet more stupid to their previous attempts (Wave, Buzz now it's simply Google+).
And yes, some of my friends have mailed me invites (or offered them to me by chat) so I can join the fun and love, and lose the few free time I still have.
Guys, I care about you. I even love you all! But no, I don't care the least for so-called-social-network invites. I am quite a mess managing my time as it stands.
And I cannot reply this to you in person, as Google+ sends mails with a very helpful sender (noreply-929ac25c@plus.google.com or variations of it), I have just added this to my .procmailrc:
Yeah! It's
However, these RAID volumes are in a pretty awkward spot for installers and bootloaders: inside a running Linux environment they look like software RAID which cannot be depended on for booting, but at boot time they look like hard disks which cannot be looked under the hood.
I recently got a new desktop machine which has one of these motherboards, and fortuitously my old desktop I was replacing had the same size disks so I had 4 disks and the option of using a RAID setup. Apparently I m a sucker for punishment because I went for a RAID 10 (that is two RAID volumes made up of two-disk mirrors (the RAID 1 component), and then those two volumes are combined via striping (the RAID 0 component). This has the potential for pretty nice performance: in principle any read can come from one of 2 disks, and every 64KB (the stripe size) of linear data will switch to the other mirror set, giving a nice boost. Writes need to write to 2 disks always, but every 64KB worth of data will alternate mirror sets, also giving a boost.
Sadly we (Ubuntu) aren t ready for this yet: there are two key bugs that make this layout almost impossible to install into. This blog post is for my exo-memory, I want to be able to figure out what I did next time around 
I am involved in developing something coined as the 
For some years, tech media has tried predict when Linux have reached
momentum for ordinary users. That current or next year was to become
I have bought a 
I've been in Europe for the last couple weeks but pretty occupied with
things like attending my brother wedding and a series of outdoor
excursions in Spain.
Today 
I sponsored the upload of fonts-anonymous-pro prepared by Rog rio
Brito. This fixed-width font is especially designed for people who
write code, for instance by making clear differences between
characters such as O and 0, or I, l and 1.
I fixed a few glitches in various l10n scripts related to D-I (those
tthigns that run daily, for which I receive a daily mail). They were
the consequence of Alioth migration and I omitted to fix them until
now....mumbling every morning about the scripts' noisy output..:-)
I worked on only one bug for samba (doh). This was #631118 which I
forwarded upstream in order to get it fixed in 3.5 series (even if 3.6
is due out soon and will be what we target for wheezy).
I tried to follow the flow of translations/reviews in DDTP for
French. The activity is high, here, with about 10 packages translated
every day (most often packages I target for the team...by using
high-popcon stats). We even had a big bump during the week-end when
Georges Khaznadar (who is very active when it comes at educative
software) initiated the translation of many edu-related packages (many
of which he maintains himself).
That's about it. Quiet week, indeed with more time invested in paid
work, including during my spare time..:-). And also a week-end in our
country house in Normandy, which usually means more gardening than hacking.
I haven t blogged for a while, because I ve been doing so many things that I scarcely had time nor will for it. Among all those things, and during the weekends of the last few months, we have been 
In case you want to know more about all of this (sorry, links are in Spanish):
Last week I tried switching a library to Gtk3. The needed changes to the code are available through --with-gtk3. However this is generally not enough. Even if your symbol list doesn't change, the ABI changes implicitly. The library in question had a .symbols file, but that's not enough because the resulting GUI application will bail out at runtime if symbols of both Gtk2 and Gtk3 are found in the same address space. That's mostly because C symbols don't contain any signatures with return types and parameters.
The 
Conrad Sanderson released version 1.99.5 of 
