There was another upsurge in discussion of Android GPL issues last month, triggered by couple of posts by Edward Naughton, followed by another by Florian Mueller. The central thrust is that section 4 of GPLv2 terminates your license on violation, and you need the copyright holders to grant you a new one. If they don't then you don't get to distribute any more copies of the code, even if you've now come into compliance. TLDR; most Android vendors are no longer permitted to distribute Linux.

I'll get to that shortly. There's a few other issues that could do with some clarification. The first is Naughton's insinuation that Google are violating the GPL due to Honeycomb being closed or their "license washing" of some headers. There's no evidence whatsoever that Google have failed to fulfil their GPL obligations in terms of providing source to anyone who received GPL-covered binaries from them. If anyone has some, please do get in touch. Some vendors do appear to be unwilling to hand over code for GPLed bits of Honeycomb. That's an issue with the vendors, not Google.

His second point is more interesting, but the summary is "Google took some GPLed header files and relicensed them under Apache 2.0, and they've taken some other people's GPLv2 code and put it under Apache 2.0 as well". As far as the headers go, there's probably not much to see here. The intent was to produce a set of headers for the C library by taking the kernel headers and removing the kernel-only components. The majority of what's left is just structure definitions and function prototypes, and is almost certainly not copyrightable. And remember that these are the headers that are distributed with the kernel and intended for consumption by userspace. If any of the remaining macros or inline functions are genuinely covered by the GPLv2, any userspace application including them would end up a derived work. This is clearly not the intention of the authors of the code. The risk to Google here is indistinguishable from zero.

How about the repurposing of other code? Naughton's most explicit description is:

For example, Android uses bootcharting logic, which uses the 'bootchartd' script provided by www.bootchart.org, but a C re-implementation that is directly compiled into our init program. The license that appears at www.bootchart.org is the GPLv2, not the Apache 2.0 license that Google claims for its implementation.

, but there's no indication that Google's reimplementation is a derived work of the GPLv2 original.

In summary: No sign that Google's violating the GPL.

Florian's post appears to be pretty much factually correct, other than this bit discussing the SFLC/Best Buy case:

I personally believe that intellectual property rights should usually be enforced against infringing publishers/manufacturers rather than mere resellers, but that's a separate issue.

The case in question was filed against Best Buy because Best Buy were manufacturing infringing devices. It was a set of own-brand Blu Ray players that incorporated Busybox. Best Buy were not a mere reseller.

Anyway. Back to the original point. Nobody appears to disagree that section 4 of the GPLv2 means that violating the license results in total termination of the license. The disagreement is over what happens next. Armijn Hemel, who has done various work on helping companies get back into compliance, believes that simply downloading a new copy of the code will result in a new license being granted, and that he's received legal advice that supports that. Bradley Kuhn disagrees. And the FSF seem to be on his side.

The relevant language in v2 is:

You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License.

The relevant language in v3 is:

You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License

which is awfully similar. However, v3 follows that up with:

However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.

In other words, with v3 you get your license back providing you're in compliance. This doesn't mesh too well with the assumption that you can get a new license by downloading a new copy of the software. It seems pretty clear that the intent of GPLv2 was that the license termination was final and required explicit reinstatement.

So whose interpretation is correct? At this point we really don't know - the only people who've tried to use this aspect of the GPL are the SFLC, and as part of their settlements they've always reinstated permission to distribute Busybox. There's no clear legal precedent. Which makes things a little awkward.

It's not possible to absolutely say that many Android distributors no longer have the right to distribute Linux. But nor is it possible to absolutely say that they haven't lost that right. Any sufficiently motivated kernel copyright holder probably could engage in a pretty effective shakedown racket against Android vendors. Whether they will do remains to be seen, but honestly if I were an Android vendor I'd be worried. There's plenty of people out there who hold copyright over significant parts of the kernel. Would you really bet on all of them being individuals of extreme virtue?

comments

I have a pathological use of browser tabs: I use a lot of them. A lot is probably an understatement. We could say I use them as bookmarks of things I need to track. A couple weeks ago, I was saying I had around two hundred tabs opened. I now actually have much more. It affected startup until I discovered that setting the browser.sessionstore.max_concurrent_tabs pref to 0 was making things much better by only loading tabs when they are selected. This preference has/will become browser.sessionstore.restore_on_demand. However, since I only start my main browser once a day, while other applications start and while I begin to read email, I hadn t noticed that this was still heavily affecting startup time: about:startup tells me reaching the sessionRestored state takes seven seconds, even on a warm startup. It also affects memory usage, because even when tabs are only loaded on demand, there is a quite big overhead for each tab. And more importantly, it gets worse with time. And I think the user interface is actively making it worse. So, to get an idea how bad things were in my session, I wrote a little restartless extension. After installing it, you can go to the about:tabs url to see the damage on your session. Please note that the number of groups is currently wrong until you open the tab grouping interface. This is what the extension has to say about my session 2 days ago, right after a startup:

• 556 tabs across 4 groups in 1 window
• 1 tab has been loaded
• 444 unique addresses
• 105 unique hosts
• 9 empty tabs
• 210 http:
• 319 https:
• 14 ftp:
• 2 about:
• 2 file:
• 55 addresses in more than 1 tab
• 39 hosts in more than 1 tab

The first thing to note is that when I filed the memory bug 4 days earlier, I had a bit more than 470 tabs in that session. You can see 4 days later, I now have 555 tabs (if excluding the about:tabs tab). The second thing to note is something I suspected because it s so easy to get there: a lot of the tabs are opened on the same address. Since Firefox 4.0, if I m not mistaken, there is a great feature in the awesomebar, that allows to jump to an existing tab matching what you type in the urlbar. That is very useful, and I use it a lot. However, there are a lot of cases where it s not as useful as it could be. One of the addresses I visit a lot is http://buildd.debian.org/iceweasel. It gives me the build status of the latest iceweasel package I uploaded to Debian unstable. That url is particularly known in my browsing history, and is the first hit when I type buildd in the urlbar (actually, even typing b brings it first). Unfortunately, that url redirects to https://buildd.debian.org/status/package.php?p=iceweasel through an HTTP redirection. I say unfortunately because when I type buildd in the urlbar, I get 6 suggestions for urls in the form http://buildd.debian.org/package (I also watch other packages build status), and the suggestion to switch to the existing tab for what the first hit would get me to is 7th. Guess what? The suggestion list only shows 6 items ; you have to scroll to see the 7th. The result is that I effectively have fifteen tabs open on that url. I also keep a lot of bugzilla.mozilla.org bugs open in different tabs. The extension tells me there are 255 of them for 166 unique bugs. Largely, the duplicate bug tabs are due to having these bugs open in some tab, but accessing the same bugs from somewhere else, usually a dependent bug or TBPL. I also have 5 tabs opened on my request queue. I usually get there by going to the bugzilla home page and clicking on the My Requests link. And I have several tabs opened on the same bug lists. For the same reason. When I started using tab groups, I splitted in very distinct groups. Basically, one for Mozilla, one for Debian, one for stuff I want to follow (usually blog posts I want to follow comments from), and one for the rest. While I was keeping up with grouping at the beginning, I don t anymore, and the result is that each group is now a real mess. Firefox has hundreds of millions users. It s impossible to create a user experience that works for everyone. One thing is sure, it doesn t work for me. My usage is probably very wrong at different levels, but I don t feel my browser is encouraging me to use it better, except by making my number of opened tabs explode to an unmanageable level (I already have 30 tabs more than when I started writing this post 2 days ago). There are a few other things I would like to know about my usage that my extension hasn t told me yet, either because it doesn t tell, or because I haven t looked:

• How many tabs end up loaded at the end of a typical day?
• How many tabs do I close?
• How many duplicate tabs do I open and close?
• How long has it been since I looked at a given tab?
• How do the number of tabs and duplicates evolve with time?

Reflecting on my usage patterns, I think a few improvements, either in the stock browser, or through extensions, could make my browsing easier:

• Auto-grouping tabs: When I click on a link to an url under mozilla.org, I most likely want it in the Mozilla group. An url under debian.org would most likely go in the Debian group.
• Switch to an existing tab when following a link to an already opened url: That might not be very useful as a general rule, but at least for some domains, it would seem useful for me that the browser switches to an existing tab not only through the urlbar, but also when following links in a page. If I m reading a bug, click on a bug it depends on, and that bug is already opened in another tab, get me there. There would be a history problem to solve, though. (e.g. where do back and forward bring?)

Maybe these exist as extensions, I don t know. It s hard to find very specific things like that through an add-on search (though I haven t searched very hard). [Looks like there is an experiment for the auto tab grouping part] I think it would also be interesting to have something like Test Pilot, but for users that want to know the answer to How do I use my browser? . As I understand it, Test Pilot can show individual user data, but it only can do so if there is such data, and you can t get data for past studies you didn t take. In my case, I m not entirely sure that, apart from the pinned tabs, I use the tab bar a lot. And even for pinned tabs, most of the time I use keyboard shortcuts. I m not using the menu button that much either. I already removed the url and search bar (most of the time) with LessChrome HD. Maybe I could go further and use the full window for web browsing.

I stopped my story in early 1994, after my Big Switch to the mysterious and new world of Linux. My home server, kheops, is now running Debian and is fed with news and mail by FrMug, formerly French Minix Users Group and now run by AFAU, the "Association Fran aise des Amateurs de Usenet". As I'm still console-challenged, I'm reading mail and news on another machine I have at home, named "khephren", which is running an, ahem, beta copy of Microsoft Windows 95. Yeah, I know...:-) These 1994-1999 years were mostly the Usenet years, for me. I gradually became more and more involved in several fr.* newsgroups, either related to the Linux world...or to the Windows world. The latter was professionnally a bonus for me as 1994 is also the year where I switched from the Material Science department at Onera to the Network and Computing Department. My mission there was to accompany the increasing prevalence of personal desktop workstations in the daily life of what was formerly the Big Castle of Computing at Onera, hosting various generations of Cray computers, most of which being in the Top500 world supercomputers (we even had one in the Top10, very briefly, at somepoint). So, I was "the PC and Windows guy" in this world of supercomputing, Unix and network wizards. But my hobby at home helped me keeping and growing skills in both "worlds" which, over the years, has proven to be a not so common profile..:-) Meanwhile, at home, I was spending hours and nights reading those newsgroups...or getting involved in fr.* hierarchy management..or expanding my friends network to many people in the news, Linux, BSD worlds. That lasted for about 3 years with ups and downs (with three young kids at home, you end up having slightly less time...). I was gradually improving my home server use, adding more functionalities to it, for instance setting it up as a file and print server with Samba 1.9.something (this will be the beginning of a long story with samba!). Unfortunately, my dear Ren had passed in the meantime. Ren did so much for Linux in France that he'll be the only link I give in this article. If you happen to go through P re Lachaise cemetery in Paris, go to its Memory garden instead of visiting Jim Morrison's grave and drink a whisky bottle in memory of Ren . Or just install one of the most funnily named Debian packages: le-dico-de-rene-cougnenc and, again, drink something to his memory (whisky is highly preferred). We then reached 1997 and a specific event: my father (who was very sick for years) passed during summer, some time after my grand-father passed too. As it apparently happens for many people, it triggerred me into the interest for family history. After all, those people who are going away are family memories. And, in some way, I wanted to keep track of this memory. As a consequence, I started collecting family memories, interviewing my close relatives (the oldest ones) and ending, of course, in tracking all this with my computers. That was the start of a long genealogy research that sent me on the track of the Perrier family (of course), but also all other branches, on my mother and father side, as well as reconstructing the tree of cousins we have, etc, etc. As anybody who did that knows, this is quite an endless task but I went very committed to it. I quickly ended up in joining research groups that were using the recent growth of personal access to Internet communication. Those groups were, in these 90's, building elaborated systems where people can help together in their research instead of having to travel to places where archives are stored. I participated to many such groups and even ended up animating one who was doing research in archives of the "Yvelines" "d partement", my place of living (even though I have no family branch there). At the same time, other groups were sending me research results for my own family in the French region where my roots are. Collective effort, help others for free, you can recognize things that are a constant all over these years, indeed. It actually helped me in tracking my Perrier ancestors up to about 1680, with 11 generations of men holding that name (from Jean-Baptiste, my son, up to tienne Perrier, who is born about 1640-1650, in the small village named Briant in Sa ne et Loire, in France). Or even climbing up to Laurent Polette, born circa 1510, mentioned in a document date February 1562. He his my oldest known ancestor.Those folks being 15 generations before me. My oldest ancestor is born under Fran ois the First, King of France, protector of Leonardo Da Vinci, at the time La Joconde was painted or the Chambord castle was built. Doh. Unfortunately, genealogy software, at that time, was quite poor for Unix systems. Lifelines was a quite strong development but wasn't that user-friendly and I ended up using a Windows-based "shareware" software written by an individual. Still, my Linux machine was used quite intensively: databases were stored there, and saved there, all important informations were there, my mail was received there, along with dozens of mailing lists I was reading. I even hosted a few mailing lists on my home server. Still, in the meantime also, again in 1997, in October, another event happened. We had a pollution peak in Paris area and October 1st was the first (and as of now the only ever) day where driving restrictions were applied. Up to that moment, I was driving to/from work (30 kilometers one way). I decided to use this opportunity to really test public transport, for one full month. As I knew it would take me much more time than driving, I tried to imagine a way to use that time. How about reading mail and news on a laptop instead of only at home? Done... In a few days, I found a used Compaq Aero laptop (486 16MHz), I installed Debian "bo" on it, switched my mail and news workflow to console programs (mutt and slrn) and here we go! So, finally, in October 1997, I was using Linux nearly all time long, the only exception being genealogy data handling. That laptop was named "mykerinos" (the small pyramid) and....my laptop is still named mykerinos, though it's now the 5th mykerinos I have..:-) Ah, and since then, I nearly always use public transport to go to/from work. Debian addiction is good for the planet! At some point (I don't remember exactly when...about 1999, I think), DSL connections appeared and one of the first DSL FAI emerged on France: Free. I was already using their services for dialup access, but DSL was then a revolution: persistent connection! And nearly immediately, static IP! I became one of their first DSL customers (and I'm still one). Moreover, I had then friends working there...and they had many Debian machines to host their services. So, with the help of my Freenix friends, some DNS hosting on their machines and the still tireless eu.org entirely free domain, perrier.eu.org was born. Immediately, this gave me opportunities for doing many interesting things with my home server: setting up www.perrier.eu.org and use it....for learning web things and sharing genealogical data. For instance, I ended up hosting some data sent by other genealogists such as digital copies of archives for some French regions, all served through my 512/128kbps DSL line (quickly upgraded to 2Mb, IIRC). And, very quickly, I felt the need to share my family tree, of course! And here comes Geneweb. Written (in Ocaml) by Daniel de Rauglaudre, Geneweb was (and still is) by far one of the best tools to share genealogy data over the Net. So, I ended up installing it on my home machine. Guess how I did that? "apt-get install geneweb", of course. Yes, it was packaged. Indeed by Brent Fulgham who did the initial release on July 22nd 2000. Here, my memory is a little bit vague. I think I actually packaged Geneweb for my own use and sharing it on upstream FTP site, forking Brent's package as a start, but updating to latest upstream release. Then I quickly ended up having discussions with Brent about this new upstream release, and that lead to:

geneweb (3.10-3) unstable; urgency=low
  * Merging with unofficial package from Geneweb ftp site
    (in preparation of new maintainer [Christian Perrier] takeing
    over responsibility).  Note that all of this work is from
    Christian.
  * Default database dir is now writable by members of geneweb group
  * New geneweb group added at installation
  * Package tries to properly handle former installations of the
    unofficial Geneweb package available on Geneweb ftp site
  * Debconf dialogs translated in french
  * More languages available
  * Creation of /var/geneweb/etc, /var/geneweb/cnt, /var/geneweb/images
    with proper permission so that authorised users may use these also
  * When de-installed, all databases are exported with gwu (should allow
    smooth upgrades when the databse format changes)
 -- Brent A. Fulgham <bfulgham>  Mon, 22 Jan 2001 19:30:47 -0800

See? Already "debconf translation to French"! "More languages available". Here we are. As discussed with Brent, he propsoed me to take the package over. So, on January 11th 2001, I had applied in the New Maintainer Process. I'm amazed how quickly it went indeed: there was a tool I needed, it needed to be packaged, I could handle the packaging...and then I decided to be involved. Here we are: bubulle is on his way to become one of those magicians who are building the best universal operating system. Yes, me, the clueless material scientist..:-) How did it go? You'll know this in next episode, of course...

User BobH asked on StackOverflow about accelerating path-dependent loops. He provided a simple example in which a vector gets filled conditional on the value of the preceding element. Simple to code, but hard to vectorise. By the time I saw that question yesterday evening, Josh Ulrich had already posted a nice answer suggesting to switch from ifelse to if to escape the overhead of a vectorised expression on simple scalars. I meekly added a comment suggesting that Rcpp would likely do well on this and that someone should volunteer such an answer. Well, it is still August and Mr. Someone is on holiday, so this morning I followed up with such an answer. And as it turns out to work quite well indeed, we will repost it here. Let's start with the general setup, and the two functions supplied by Josh. We also byte-compile these using the compiler package which is the culmination of several years of work by R core member Luke Tierney. This package was added to R during the last major release, and we assessed it in this earlier blog post as well as several Rcpp-related follow-ups. We also load the packages for on-the-fly 'inline' compilation and easy benchmarking:

library(inline)
library(rbenchmark)
library(compiler)
fun1 <- function(z)  
  for(i in 2:NROW(z))  
    z[i] <- ifelse(z[i-1]==1, 1, 0)
   
  z
 
fun1c <- cmpfun(fun1)
fun2 <- function(z)  
  for(i in 2:NROW(z))  
    z[i] <- if(z[i-1]==1) 1 else 0
   
  z
 
fun2c <- cmpfun(fun2)

We see that basic worker just assign to the i-th element based on the preceding element. Function two uses the aforementioned if statement, and both are also byte-compiled. Writing the same code in C++ using both Rcpp (for the R/C++ integration) and inline for the on-the-fly compilation, linking and loading of C++ code into R is pretty straightforward too:

funRcpp <- cxxfunction(signature(zs="numeric"), plugin="Rcpp", body="
  Rcpp::NumericVector z = Rcpp::NumericVector(zs);
  int n = z.size();
  for (int i=1; i<n; i++)  
    z[i] = (z[i-1]==1.0 ? 1.0 : 0.0);
   
  return(z);
")

This single R function call takes the code embedded in the argument to the body variable, builds a complete function in temporary file, and then compiles, links and loads it. We can now call funRcpp() just like the other four functions and do so via the benchmark() function of the rbenchmark package.

R> z <- rep(c(1,1,0,0,0,0), 100)
R> identical(fun1(z),fun2(z),fun1c(z),fun2c(z),funRcpp(z))
[1] TRUE
R> 
R> res <- benchmark(fun1(z), fun2(z),
+                  fun1c(z), fun2c(z),
+                  funRcpp(z),
+                  columns=c("test", "replications", "elapsed", "relative", "user.self", "sys.self"),
+                  order="relative",
+                  replications=1000)
R> print(res)
        test replications elapsed relative user.self sys.self
5 funRcpp(z)         1000   0.005      1.0      0.01        0
4   fun2c(z)         1000   0.482     96.4      0.48        0
2    fun2(z)         1000   1.989    397.8      1.98        0
3   fun1c(z)         1000  11.365   2273.0     11.37        0
1    fun1(z)         1000  13.210   2642.0     13.21        0

We can focus on the columns labelled elapsed and relative. The C++ version is faster by a factor of almost one-hundred compared to the byte-compiled version of funtion2, and almost four-hundred times faster than the plain-R variant of function2. And function1 is even worse, coming at well over twenty-two-hundred times the run-time of the C++ version. Byte-compilation also helps little here. For comparison, we also ran the original example of a very short vector, called more frequently:

R> z <- c(1,1,0,0,0,0)
R> res2 <- benchmark(fun1(z), fun2(z),
+                  fun1c(z), fun2c(z),
+                  funRcpp(z),
+                  columns=c("test", "replications", "elapsed", "relative", "user.self", "sys.self"),
+                  order="relative",
+                  replications=10000)
R> print(res2)
        test replications elapsed relative user.self sys.self
5 funRcpp(z)        10000   0.047  1.00000      0.04        0
4   fun2c(z)        10000   0.134  2.85106      0.13        0
2    fun2(z)        10000   0.328  6.97872      0.32        0
3   fun1c(z)        10000   1.080 22.97872      1.08        0
1    fun1(z)        10000   1.243 26.44681      1.24        0

The qualitative ranking is unchanged: the Rcpp version dominates. Function2 using if instead of the vectorised ifelse is second-best with the byte-compiled version being about twice as fast that the plain R variant, but still almost three times slower than the C++ version. And the relative differences are less pronounced: relatively speaking, the function call overhead matters less here and the actual looping matters more: C++ gets a bigger advantage on the actual loop operations in the longer vectors. That it is an important result as it suggests that on more real-life sized data, the compiled version may reap a larger benefit. All in all a nice demonstration of Rcpp, and a gain of almost one-hundred to the best byte-compiled version is nothing to sneeze at---especially when it is so easy to write and load a five-line C++ function thanks to Rcpp. Before closing, a quick reminder that I will giving two classes on Rcpp in a few weeks. These will be in New York on September 24, and San Franciso on October 8, see this blog post as well as this page at Revolution Analytics (who are a co-organiser of the classes) for details and registration information.

Here is the short version: to get better performance on your SSL terminator, use stud on 64bit system with patch from meric Brun for SSL session reuse with some AES cipher suite (128 or 256, does not really matter), without DHE, on as many cores as needed, a key size of 1024 bits unless more is needed.

Introduction A quick note: when I say SSL, you should read TLS v1 instead. Look at the article for TLS on Wikipedia for some background on this. One year ago, Adam Langley, from Google, stated SSL was not computationally expensive any more:

In January this year (2010), Gmail switched to using HTTPS for everything by default. Previously it had been introduced as an option, but now all of our users use HTTPS to secure their email between their browsers and Google, all the time. In order to do this we had to deploy no additional machines and no special hardware. On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. Many people believe that SSL takes a lot of CPU time and we hope the above numbers (public for the first time) will help to dispel that. If you stop reading now you only need to remember one thing: SSL/TLS is not computationally expensive any more.

This is a very informative post containing tips on how to enhance SSL performance by reducing latency. However, unlike Gmail, you may still be worried by SSL raw performances. Maybe each of your frontends is able to serve 2000 requests per second and CPU overhead for SSL is therefore significative. Maybe you want to terminate SSL in your load balancer (even if you know this is not the best way to scale).

Tuning SSL There are a lot of knobs you can use to get better performances from SSL: choosing the best implementation, using more CPU cores, switching to 64bit system, choosing the right cipher suite and the appropriate key size and enabling a session cache. We will consider three SSL terminators. They all use OpenSSL behind the hood. stunnel is the oldest one and uses a threaded model. stud is a recent attempt to write a simple SSL terminator which is efficient and scalable. It uses the one-process-per-core model. nginx is a web server and it can be used as reverse proxy and therefore act as SSL terminator. It is known to be one of the most efficient web server, hence the choice here. It also features built-in basic load balancing. Since stud and stunnel does not have this feature, we use them with HAProxy, an high performance load-balancer that usually defers the SSL part to stunnel (but stud can act as a drop-in replacement here). Those implementations were already tested by Matt Stancliff. He first concluded nginx sucked at SSL then that nginx did not suck at SSL (in the first case, nginx was the only one to select a DHE cipher suite). The details were very scarce. I hope to bring here more details. This brings the importance of the selected cipher suite. The client and the server must agree on the cipher suite to use for symmetric encryption. The server will select the strongest one it supports from the list proposed by the client. If you enable some expensive cipher suite on your server, it is likely to be selected. The last important knob is SSL session reuse. It matters for both performance and latency. During the first handshake, the server will send a session ID. The client can use this session ID to request an abbreviated handshake. This handshake is shorter (latency improvement by one round-trip) and allows to reuse the previously negociated master secret (performance improvement by skipping the costliest part of the handshake). See the article from Adam Langley for additional information (with helpful drawings). The following table shows what cipher suite is selected by some major web sites. I have also indicated wheter they support session reuse. See this post from Matt Palmer to know how to check this.

Site	Cipher	Session reuse?
www.google.com	RC4-SHA
www.facebook.com	RC4-MD5
twitter.com	AES256-SHA
windowsupdate.microsoft.com	RC4-MD5
www.paypal.com	AES256-SHA
www.cmcicpaiement.fr	DHE-RSA-AES256-SHA

RFC 5077 defines another mechanism for session resumption that does not require any server-side state. I did not investigate much how this works but it does not rely on the session ID.

Benchmarks All those benchmarks have been run on a HP DL 380 G7, with two Xeon L5630 (running at 2.13GHz for a total of 8 cores), without hyperthreading, using a 2.6.39 kernel (HZ is set to 250) and two Intel 82576 NIC. Linux conntrack subsystem has been disabled and file limits have been raised over 100,000. A Spirent Avalanche 2900 appliance is used to run the benchmarks. We are interested in raw SSL performances in term of handshakes per second. Because we want to observe the effects of session reuse, the scenario runs by most tests lets clients doing four successive requests and reusing the SSL session for three of them. There is no HTTP keepalive. There is no compression. The size of the HTTP answer from the server is 1024 bytes. Those choices are done because our primary target is the number of SSL handshakes per second. It should also be noted that HAProxy alone is able to handle 22,000 TPS on one core. During the tests, it was never the bottleneck.

Implementation We run a first bench to compare nginx, stud and stunnel on one core. This bench runs on a 32bit system, with AES128-SHA1 cipher suite and a 1024bit key size. Here is the result for stud (version 0.1, with meric Brun s patch for SSL session reuse): The most important plot is the top one. The blue line is the attempted number of transactions per second (TPS) while the green one is the successful number of TPS. When the two lines start to diverge, this means we reach some kind of maximum: the number of unsuccessful TPS also starts to raise (the red line). There are several noticeable points: the maximum TPS (788 TPS), the maximum TPS with an average response time less than 100 ms (766 TPS), less than 500 ms (776 TPS) and the maximum TPS with less than 0.01% of packet loss (783 TPS). Let s have a look at nginx (version 1.0.5): It is able to get the same performance as stud (763 TPS). However, over 512 TPS, you get more than 100 ms of response time. Over 556 TPS, you even get more than 500 ms! I get this behaviour in every bench with nginx (with or without proxy_buffering, with or without tcp_nopush, with or without tcp_nodelay) and I am unable to explain it. Maybe there is something wrong in the configuration. After hitting this limit, nginx starts to process connections by bursts. Therefore, the number of successful TPS is plotted using a dotted line while the moving average over 20 seconds is plotted with a solid line. On the next plot, you can see the compared performance of the three implementations. On this kind of plot, the number of TPS that we keep is the maximum number of TPS where loss is less than 0.1% and average response time is less than 100 ms. stud achieves a performance of 766 TPS while nginx and stunnel are just above 500 TPS.

Number of cores With multiple cores, we can affect several of them to do SSL hard work. To get better performance, we pin each processes to a dedicated CPU. Here is the repartition used during the tests:

	1 core	2 cores	4 cores	6 cores
CPU 1, Core 1	network	network	network	network + haproxy
CPU 1, Core 2	haproxy	haproxy	haproxy	SSL
CPU 1, Core 3	SSL	SSL	SSL	SSL
CPU 1, Core 4	-	SSL	SSL	SSL
CPU 2, Core 5	-	-	network	SSL
CPU 2, Core 6	-	-	SSL	SSL
CPU 2, Core 7	-	-	SSL	SSL
CPU 2, Core 8	system	system	system	system + haproxy

Remember, we have two CPU, 4 cores on each CPU. Cores on the same CPU share the same L2 cache (on this model). Therefore, the arrangement inside a CPU is not really important. When possible, we try to keep things together on the same CPU. The SSL processes always get exclusive use of a core since they will always be the most busy. The repartition is done with cpuset(7) for userland processes and by setting smp_affinity for network card interrupts. We keep one core for the system. It is quite important to be able to connect and monitor the system correctly even when it is loaded. With so many cores, we can afford to reserve one core for this usage. Beware! There is a trap when pining processes or IRQ to a core. You need to check /proc/cpuinfo to discover the mapping between kernel processors and physical cores. Sometimes, second kernel processor can be the first core of the second physical processor (instead of the second core of the first physical processor). As you can see in the plot above, only stud is able to scale properly. stunnel is not able to take advantage of the cores and its performances are worse than with a single core. I think this may be due to its threaded model and the fact that the userland for this 32bit system is a bit old. nginx is able to achieve the same TPS than stud but is disqualified because of the increased latency. We won t use stunnel in the remaining tests.

32bit vs 64bit The easiest way to achieve a performance boost is to switch to a 64bit system. TPS are doubled. On our 64bit system, thanks to the use of a version of OpenSSL with the appropriate support, AES-NI, a CPU extension to improve the speed of AES encryption/decryption, is enabled. Remaining tests are done with a 64bit system.

Ciphers and key sizes In our tests, the influence of ciphers is minimal. There is almost no difference between AES256 and AES128. Using RC4 adds some latency. The use of AES-NI may have helped to avoid this latency for AES. On the other hand, using a 2048bit key size has a huge performance hit. TPS are divided by 5.

Session cache The last thing to check is the influence of SSL session reuse. Since we are on a local network, we see only one of the positive effects of session reuse: better TPS because of reduced CPU usages. If there was a latency penalty, we should also have seen better TPS thanks to the removed round-trip. This plot also explains why stud performances fall after the maximum: because of the failed transactions, the session cache is not as efficient. This phenomenon does not exist when session reuse is not enabled.

Conclusion Here is a summary of the maximum TPS reached during the benchmark (with an average response time below 100ms). Our control use case is the following: 64bit, 6 cores, AES128, SHA1, 1024bit key, 4 requests into the same SSL session.

Context	nginx 1.0.5	stunnel 4.41	stud 0.1 (patched)
1 core, 32bit	512 TPS	503 TPS	766 TPS
2 cores, 32bit	599 TPS	-	-
32bit	804 TPS	501 TPS	4251 TPS
-	1799 TPS	-	9000 TPS
AES256	-	-	8880 TPS
RC4-MD5	-	-	7370 TPS
2048bit	-	-	1643 TPS
no session reuse	-	-	5844 TPS
80 requests per SSL session	-	-	10797 TPS

Therefore, stud in our control scenario is able to sustain 1500 TPS per core. It seems to be the best current option for a SSL termination. It is not available in Debian yet, but I intend to package it. Here is how stud is invoked:

# ulimit -n 100000
# stud  -n 2 -f 172.31.200.15,443 -b 127.0.0.1,80 -c ALL \
>   -B 1000 -C 20000 --write-proxy \
>   =(cat server1024.crt server1024.key dhe1024)

You may also want to look at HAProxy configuration, nginx configuration and stunnel configuration.

If you were trying to get rid of hiring discrimination (on grounds irrelevant to the ability to do the job), you d think a good way to do it would be to reduce the ability of the hiring manager to discriminate, by restricting their access to irrelevant (but possibly prejudicial) information. It s certainly what I might come up with as an early idea in a brainstorming session. I m not alone: France had this same idea, and gave it a go, by passing a law requiring companies to anonymise resumes before they got to any decision makers. So far, so average. But rather than just coming up with an idea and inflicting it on everyone by a blanket law, they did what should be done with all new ideas: they trialled it (with 50 large corporations, according to the report) before making it universal, to make sure that the theory matched reality. Then, after giving it a good shake, they examined the evidence, and found that the idea had some unintended consequences:

Applicants with foreign names, or who lived in under privileged areas were found to be less likely to be called in for an interview without the listing of their name and address. Researchers reasoned that this was because employers and recruiters made allowances for subpar presentation or limited French speaking if their performance could be explained by deprivation or foreign birth.

The icing on the cake is that now the evidence is in, they re now planning on making it optional (I m not sure how that s different from killing it entirely, but I guess it s worth the same in the end). So we ve got the quinella of decision-making awesome:

• An idea was had
• A trial was run
• The evidence was examined
• When the evidence didn t support the idea, the idea was abandoned

Far too often, we get far too attached to our ideas, and don t let them go when reality doesn t fit our preconceptions. Kudos to the people involved in this idea for not letting their egos get in the way of good government. Let it be an object lesson for us all.

Linux 2.6.24, released more than three years ago, introduced two very interesting features: PID and network namespaces. They allow to create a process that does not share the same PID namespace or the same network namespace than other processes. Those namespaces are then inherited by the children (unless they are assigned new namespaces).

PID & mount namespaces A PID namespace is a view of the processes running on the system. Each PID namespace will have its own numbering of the processes contained in its namespace (including the namespace of the children). Other processes are hidden and therefore, it is not possible to issue any operation requiring a PID on them, like kill(2). Moreover, the process that was created first in a namespace will get PID 1. If it gets killed, the namespace is destroyed and all other processes contained in this namespace are killed as well. systemd, a system and service manager for Linux, targeted as a replacement for the old SysV init, uses could use this feature to launch daemons in their own PID namespace and keep track of them without relying on their cooperation. When it needs to kill a daemon, it just kills the process with PID 1 (which has another PID in the global namespace) and the kernel will do the hard work of killing all processes that were started in this namespace. You can get more details about this namespace in the manual page of clone(2) (search for CLONE_NEWPID). UPDATED: In fact, systemd does not use PID namespaces but cgroups to group processes together. There is no PID numbering here but spawned processes are all enclosed into one cgroup and therefore can be killed easily. For example, here is the result of pstree -p inside a PID namespace:

root@guybrush:/# pstree -p
bash(1) pstree(8)
         sleep(7)

The PID of sleep is 7. In the global namespace, it is 31762 instead. But it is the same processus!

root@guybrush:/# ps -eo pid,command   grep sleep
31762 sleep 1000

Linux have also supported mount namespaces since a long time. Each process can have its own mount points that will be visible only by processes in the same namespace or in children namespaces. When the namespace is destroyed (when no process left), the kernel will do the cleanup itself.

chroot A common way to test experimental software or do packaging work is to build a chroot: debootstrap is used to build a new complete system inside a directory and chroot allows to enter into this new system. Tools like schroot can become handy if you need to do that often. Unfortunately, you only get an isolated filesystem. When you start launching daemons and mounting partitions, you need to keep track of all your actions to undo them. Quitting the chroot will not kill any daemon nor unmount any partition. Extending an existing tool, like schroot to use PID and mount namespaces seems the best way to prevent such problems. I have filed bug #637870 for this purpose. However, this is not as simple as it seems since schroot allows to run a process into an existing session. When the session is a chroot, this is quite easy: just chroot into the same root filesystem and launch the process here. With other kind of namespaces, you need to use a new system call, setns(), to change the current namespace of the running process. However, some bits are still missing. First, the GNU C Library does not implement the appropriate wrapper. Second, Linux (as of 3.0) does not contain the support to change the PID namespace of a running process yet. Other tools exist, like lxc (aka Linux Containers). However, it is more targeted at providing a complete container. I have reported that lxc-execute is not able to spawn a standalone shell but got no luck. UPDATED: systemd ships with systemd-nspawn which was exactly what I needed. However, it requires systemd. Otherwise, it seems less basic than the alternative that I describe below (some sensible mount points are setup, the console is correctly configured, signals are correctly transmitted, ). If you use systemd, look at it!

jchroot Building a chroot clone using namespaces is quite easy. The only difficulty is the use of clone(2). Here is a sketch:

int main()  
  pid_t pid;
  int ret;
  long stack_size = sysconf(_SC_PAGESIZE);
  void *stack = alloca(stack_size) + stack_size;
  /* New process with its own PID/IPC/NS namespace */
  pid = clone(step2,
              stack,
              SIGCHLD   CLONE_NEWPID   CLONE_NEWIPC  
              CLONE_NEWNS   CLONE_FILES,
              NULL);
  /* Wait for the child to terminate */
  while (waitpid(pid, &ret, 0) < 0 && errno == EINTR)
    continue;
  return WIFEXITED(ret)?WEXITSTATUS(ret):EXIT_FAILURE;
 
static int step2(void *a)  
  /* Mount /proc as an example */
  mount("proc", "/proc", "proc", 0, "");
  /* Chroot */
  chroot("/srv/debian");
  chdir("/");
  /* Drop privileges */
  setgid(1000);
  setgroups(0, NULL);
  setuid(1000);
  /* Exec shell */
  execl("/bin/bash", "/bin/bash", NULL);
 

Add command flags, error handling and fstab parsing for the definition of mount points and you get jchroot. Here is an example of use:

$ cat fstab
proc     /proc  proc    defaults                  0  0
sys      /sys   sysfs   defaults                  0  0
/home    /home  none    bind,rw                   0  0
/dev/pts /dev/pts none  bind,rw                   0  0
/var/run /var/run tmpfs rw,nosuid,noexec,mode=755 0  0
/etc/resolv.conf /etc/resolv.conf none bind,ro    0  0
$ sudo jchroot -f fstab -n test /srv/debian/squeeze /bin/bash
# hostname
test
# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19480  1888 pts/1    S    15:18   0:00 /bin/bash
root         3  0.0  0.0  16536  1184 pts/1    R+   15:18   0:00 ps auxww
# /etc/init.d/postgresql start
Starting PostgreSQL 9.0 database server: main.
# exit
$ ps auxwwc   grep postgres
$

I hope that jchroot won t live for too long and that its features will be integrated into regular chroot tools. I was thinking at proposing a patch for chroot(8). Anyone knows if this would have a chance to be accepted?

This year without our friends from Grml, but The MirOS Project (all two active developers and our Booth Babe gecko2@) will of course attend FrOSCon, nicknamed Froschkon, again. We ll have a pre-event meal time at my favourite Jugoslawian Restaurant on Friday (20:00 CEST) contact me privately for the co rdinates if interested. On Saturday and Sunday we ll staff a booth and answer questions about the many projects we have (more or less) running, including but not limited to paxmirabilis (aka MirCPIO), The MirBSD Korn Shell aka mksh(1), jupp the editor, and developers private projects such as slowly undermining Debian or Google-Go. While slow we are still working on World Domination. And teaching people good shell programming by example code. We might even bring CDs, but I m still working on the ISO last night s build aborted because the OS grew a bit making the floppy image not fit any more. (Solution, drop ping(8) and rtsol(8), but re-add sf(4) and bce(4) now that they fit again.)

A very popular refrain on tech sites is that the high street cannot support multiple competing phone ecosystems. It s a reasonable position to take. Do phone stores want to train their employees on six or seven different OSes? Do consumers understand the differences, and should they need to? Do app developers want to rewrite their app six or seven times? The obvious answer to all of these is no . Stores only want to train their employees once, developers want to write their app once. Choice is bad, because choice is complicated. You can have it in any colour, so long as it s black. The problem is, we need those competing efforts, for the entire market to increase in quality. Until the iPhone appeared on the market, Google s nascent mobile phone OS looked more like a poor man s Blackberry OS than what ended up shipping on the HTC Dream and later devices. Windows Phone 7 didn t ship with any multitasking at all and now the Mango update will be incorporating a multitasking model largely thieved from WebOS. iOS only just got support for updating the phone s firmware without being plugged into a host PC via USB, something which a few Android devices and all WebOS devices have always supported. Consider, for comparison, the web browser market. It would be easiest if there was only one web browser but that browser would quickly stagnate and cause pain, the way Internet Explorer 6 did when it had almost the entire market. It wasn t until upstarts like Firefox and Opera and Chrome and more started showing off their unique ideas that the entire pack started improving including IE, with IE9 supporting most of the features that the competition introduced. Every smartphone shipping today has something to offer that other devices don t and every smartphone OS shipping today has something to offer that other OSes don t. iOS has the widest app catalog. Android is Open Source (FSVO Open ). WebOS has the best multitasking, and Just Type . Windows Phone 7 offers a drastic new user interface paradigm. Blackberry OS is built for messaging tasks, and has the best sysadmin control. Symbian is well, okay, sometimes there s a time to let go but the same applies for web browsers too. Smartphones are communication devices that people use every day of their lives, and every consumer has different needs we shouldn t try to push them into an ill-fitting category, just to satisfy ourselves that it s probably only possible for three mobile platforms to succeed in the mass-market . For some people, Blackberry OS really *is* the best choice, and no matter how much you pretend, an iPhone would enhance their lives. And as for the app question tough shit. Plenty of app developers only target iOS even though Android is overtaking in the market, and if they really want to reach as many people as possible, then use a cross-platform framework like PhoneGap or some of Xamarin s products. Single-ecosystem apps are a low-effort push to reach as many people possible with minimal investment, and the only way to satisfy that class of developer is to eliminate ALL competition in the marketplace iOS-only devs are this decade s IE6-only devs. Everyone is entitled to their opinion, but I for one am sick of all the tales of doom and gloom that immediately surround any effort which isn t iOS or pure Android. We have a marketplace of ideas, which we should be celebrating, and not dumping on. I m eagerly awaiting delivery of my HP Pre 3, with all the unique possibilities it offers me which simply another Android device wouldn t have. And the big difference between WebOS and Nokia s Maemo/MeeGo efforts, for those who are still doubtful, is HP haven t spent years trying to deliberately sabotage the platform. Give it time. If I m wrong, the worst-case scenario is being lumbered with an awesome phone.

You might not know who Peter is because he s not very visible on Debian mailing lists. He s very active however and in particular on IRC. He was an admin of the OFTC IRC network at the time Debian switched from Freenode to OFTC. Nowadays he s a member of the Debian System Administration team who runs all the debian.org servers. If you went to a Debconf you probably met him since he s always looking for new signatures of his GPG key. He owns the best connected key in the PGP web of trust. He also wrote caff a popular GPG key signing tool. Raphael: Who are you? Peter: I m Peter Palfrader, also known as weasel. I m in my early 30s, born and raised in Innsbruck, Austria and am now living and working in Salzburg, Austria. In my copious free time, other than help running Debian s servers I also help maintaining the Tor project s infrastructure. Away from the computer I enjoy reading fiction (mostly English language Science Fiction and Fantasy), playing board games and going to the movies. Weather permitting, I also occasionally do some cycling. Raphael: How did you start contributing to Debian? Peter: I installed my first Debian the week slink came out. That was Debian 2.1 for the youngsters, in early 1999. The one thing I immediately liked about slink was that Debian s pppd supported RAS authentication which my university s dial-up system required. No way I d go back to SuSE 5.3 when I had working Internet with my Debian box. During that year I started getting involved in the German language Debian channel on IRCnet which got me in contact with some DDs. Christian Kurz (<shorty>) was working on Debian QA at the time and he asked my help in writing a couple of scripts. Some of that work, debcheck, still produces parts of the qa.d.o website, tho the relevance of that nowadays is probably negligible. While trying to learn more Perl earlier, I had written a program to produce syntax highlighted HTML for code snippets in various languages. I didn t really know what I was doing but it kinda worked, and probably still does since I still get mail from users every now and then. I figured that it would be really nice if people could just get my software together with Debian. According to code2html s Debian changelog the initial release of the package was done on a weekday at 2:30 in the morning early in 2000, and if my memory serves me correctly, shorty uploaded it shortly afterwards. I started packaging a couple of other piece of software and in the same year I sent my mail to the debian account managers to register my intent to become a DD. No new developers where being accepted at that time since the DAMs wanted to overhaul the entire process so I wasn t surprised to not get any immediate reply. Of course what the silence also meant was that the mail had been lost, but I only learned of that later when I took all my courage to ask DAM about the status of application a couple months later. Once that was sorted out I was assigned an AM, did the usual dance, and got my account late in November 2000. Raphael: Four years ago, the Debian System Administration team was a real bottleneck for the project and personal conflicts made it almost impossible to find solutions. You were eager to help and at some point you got dropped as a new member in that team. Can you share your story and how you managed the transition in the difficult climate at that time? Peter: Ah, that was quite the surprise for an awful lot of people, me included. Branden Robinson, who was our DPL for the 2005-2006 term, tried to get some new blood added to DSA who were at the time quite divided. He briefly talked to me on IRC some time in summer 2005, telling me I had come recommended for a role on the sysadmin team . In the course of these 15 minutes he outlined some of the issues he thought a new member of DSA would face and asked me if I thought I could help. My reply was cautiously positive, saying that I didn t want to step on anybody s toes but maybe I could be of some assistance. And that was the first and last of it, until some fine November day two years later I got an email from Phil Hands saying I ve just added you to the adm group, and added you to the debian-admin@d.o alias. and welcome on board . *blink* What!? My teammates at the time were James Troup (elmo), Phil Hands (fil), Martin Joey Schulze and Ryan Murray (neuro). The old team, while apparently not on good terms with one another, was however still around to do heavy lifting when required. I still remember when on my first or second day on the team two disks failed in the raid5 of ftp-master.debian.org aka ries. Neuro did the reinstall once new disks had arrived at Brown University. I m sure I d have been way out of my league had this job fallen to me. Fortunately my teammates were all willing and able to help me find whatever pieces of information existed that might help me learn how debian.org does its stuff. Unfortunately a lot of it only existed in various heads, or when lucky, in one of the huge mbox archives of the debian-admin alias or list. Anyway, soon I was able to get my hands dirty with upgrading from sarge to etch, which had been released about half a year earlier. Raphael: I know the DSA team has accomplished a lot over the last few years. Can you share some interesting figures? Peter: Indeed we have accomplished a lot. In my opinion the most important of these accomplishment is that we re actually once again a team nowadays. A team where people talk to one another and where nobody should be a SPoF. Since this year s debconf we are six people in the admin team: Tollef Fog Heen (Mithrandir) and Faidon Liambotis (paravoid) joined the existing members: Luca Filipozzi, Stephen Gran, Martin Zobel-Helas, and myself. Growing a core team, especially one where membership comes with uid0 on all machines, is not easy and that s why I m very glad we managed to actually do this step. I also think the infrastructure and our workflows have matured well over the last four years. We now have essential monitoring as a matter of course: Nagios not only checks whether all daemons that should be running are in fact running, but it also monitors hardware health of disks, fans, etc. where possible. We are alerted of outstanding security updates that need to be installed and of changes made to our systems that weren t then explicitly acked by one of us. We have set up a centralized configuration system, puppet, for some of our configuration that is the same, or at least similar, on all our machines. Most, if not all, pieces of software, scripts and helpers that we use on debian.org infrastructure is in publicly accessible git repositories. We have good communication with other teams in Debian that need our support, like the ftp folks or the buildd people. As for figures, I don t think there s anything spectacular. As of the time of our BoF at this year s DebConf, we take care of approximately 135 systems, about 100 of them being real iron, the other virtual machines (KVM). They are hosted at over 30 different locations, tho we are trying to cut down on that number, but that s a long and difficult process. We don t really collect a lot of other figures like web hits on www.debian.org or downloads from the ftp archive. The web team might do the former and the latter is pretty much impossible due to the distributed nature of our mirrors, as you well know. Raphael: The DSA team has a policy of eating its own dog food, i.e. you re trying to rely only on what s available in Debian. How does that work out and what are the remaining gaps? Peter: Mostly Debian, the OS, just meets our needs. Sure, the update frequency is a bit high, we probably wouldn t mind a longer release cycle. But on the other hand most software is recent enough. And when it s not, that s easy to fix with backports. If they aren t on backports.debian.org already, we ll just put them there (or ask somebody else to prepare a backport for us) and so everybody else benefits from that work too. Some things we need just don t, and probably won t, exist in Debian. These are mainly proprietary hardware health checks like HP s tools for their servers, or various vendors programs to query their raid controller. HP actually makes packages for their stuff which is very nice, but other things we just put into /usr/local, or if we really need it on a number of machines, package ourselves. The push to cripple our installers and kernels by removing firmware was quite annoying, since it made installing from the official media next to impossible in some cases. Support for working around these limitations has improved with squeeze so that s probably ok now. One of the other problems is that especially on embedded platforms most of the buildd work happens on some variation of development boards, usually due to increased memory and hard disk requirements than the intended market audience. This often implies that the kernel shipped with Debian won t be usable on our own debian.org machines. This makes keeping up with security and other kernel fixes way more error prone and time intensive. We keep annoying the right people in Debian to add kernel flavors that actually boot on our machines, and things are getting better, so maybe in the future this will no longer be a problem. Raphael: If you could spend all your time on Debian, what would you work on? Peter: One of the things that I think is a bit annoying for admins that maintain machines all over the globe is mirror selection. I shouldn t have to care where my packages come from, apt-get should just fetch them from a mirror, any mirror, that is close by, fast and recent. I don t need to know which one it was. We have deployed geodns for security.debian.org a while ago, and it seems to work quite well for the coarse granularity we desired for that setup, but geodns is an ugly hack (I think it is a layer violation), it might not scale to hundreds or thousands of mirrors, and it doesn t play well with DNSSEC. What I d really like to see is Debian support apt s mirror method that I think (and I apologize if I m wronging somebody) Michael Vogt implemented recently. The basic idea is that you simply add deb mirror://mirror.debian.org/ or something like that to your sources.list, and apt goes and asks that server for a list of mirrors it should use right now. The client code exists, but I don t know how well tested it is. What is missing is the server part. One that gives clients a mirror, or list of mirrors, that are close to them, current, and carry their architecture. It s probably not a huge amount of work, but at the same time it s also not entirely trivial. If I had more time on my hands this is something that I d try to do. Hopefully somebody will pick it up. Raphael: What motivates you to continue to contribute year after year? Peter: It s fun, mostly. Sure, there are things that need to be done regularly that are boring or become so after a while, but as a sysadmin you tend to do things once or twice and then seek to automate it. DSA s users, i.e. DDs, constantly want to play with new services or approaches to make Debian better and often they need our support or help in their endeavors. So that s a constant flow of interesting challenges. Another reason is that Debian is simply where some of my friends are. Working on Debian with them is interacting with friends. I not only use Debian at debian.org. I use it at work, I use it on my own machines, on the servers of the Tor project. When I was with OFTC Debian is what we put on our machines. Being a part of Debian is one way to ensure what Debian releases is actually usable to me, professionally and with other projects. Raphael: Is there someone in Debian that you admire for their contributions? Peter: That s a hard one. There are certainly people who I respect greatly for their technical or other contributions to Debian, but I don t want to single anybody out in particular. I think we all, everyone who ever contributed to Debian with code, support or a bug report, can be very proud of what we are producing one of the best operating systems out there.

---

Thank you to Peter for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

No comment Liked this article? Click here. My blog is Flattr-enabled.

The Border Gateway Protocol is what network operators use to exchange the various routes forming the Internet. It's a pretty critical piece of infrastructure. Most providers are probably using proprietary solutions using dedicated hardware, but free software on comodity hardware can also be an option for some networks. In Debian, we already have Quagga and Bird that both implement BGP on top of Linux for quite a while. Back in time, FOSDEM feb. 2006, Ging is being demonstrated. Immediately my mind races with something like: waow, we could port OpenBGPD on that! OpenBGPD is another BGP routing daemon. This new effort was started in 2003 by Henning Brauer for OpenBSD. To sum up its goal, it was meant to be secure, reliable, lean, easy to configure, fast and memory efficient. This is quite the opposite of Quagga codebase, which is inherited from an older project, Zebra which was mostly documented in Japanese. Another issue I have seen hapenning with Quagga on Linux is that its blocking implementation and the time that the kernel needs to insert/remove routes made the daemon unresponsive enough to created cascading failures. The IPv4 Internet is nowdays made of more than 350 000 prefixes, so when a network peer fails, the router needs to process those changes while continuing to answer other peers. Otherwise, they will kill the session meaning yet another 350 000 prefixes to process on both sides. And again when the session is restored. Leading to more downtime, flaps, and failures. OpenBGPD was ported to FreeBSD. The port is maintained by Hiroki Sato. Starting with this port and using libbsd made the creation of a package for Debian GNU/kFreeBSD pretty doable while at DebCamp11. It probably still has some pretty rough edges and the manpage needs to be reviewed (and some unsupported features trimmed, maybe), but the package should be usable. So OpenBGPD is now in Debian, and it could be one of the killer features of a port that some describe as a toy OS. Unfortunately I did not yet have access to proper network equipment to test the package on real production scenario (with 2 or 3 full BGP views). If you have a lab (or feel adventurous) and have access to such gears, please have a try!

About 24 hours ago I rebooted the system that runs the secondary DNS for my zone and a few other zones. I d upgraded a few things and the system had been running for almost 200 days without a reboot so it was time for it. Unfortunately it didn t come back up. Even more unfortunately the other DNS server for my zone is ns.sws.net.au which is also the only other server for the sws.net.au zone. Normally this will work because the servers for the net.au zone have a glue record containing the server IP address. So when asked for the NS records for the sws.net.au domain the reply will include the IP address of ns.sws.net.au. The unfortunate part was that the IP address was the old IP address from before the sws.net.au servers changed to a new IP address range, I wonder whether this was due to the recovery process after the Distribute IT hack [1], as forgetting to change a glue record is not something that I or the other guy who runs that network would forget. But it is possible that we both stuffed up. The DNS secondary was an IBM P3-1GHz desktop system with two IDE disks in a RAID-1 array. It s been quite reliable, it s been running in the same hardware configuration for about four years now with only one disk replacement. It turned out that the cooling fan in the front of the case had seized up due to a lot of dirt and the BIOS wouldn t let the system boot in that state. Also one of the disks was reporting serious SMART problems and needed to be replaced poor cooling tends to cause disk errors. It seems that Compaq systems are good at informing the user of SMART problems, two different Compaq desktop systems (one from before the HP buyout and one from after) made very forceful recommendations that I replace the disk, it s a pity that the BIOS doesn t allow a normal boot process after the warning as following the recommendation to backup the data is difficult when the system won t boot. I have a temporary server running now, but my plan is to install a P3-866 system and use a 5400rpm disk to replace the 7200rpm that s currently in the second position in the RAID array. I ve done some tests on power use and an old P3 system uses a lot less than most new systems [2]. Power use directly maps to heat dissipation and a full size desktop system with big fans that dissipates less than 50W is more likely to survive a poorly cooled room in summer. Laptops dissipate less heat but as their vents are smaller (thus less effective at the best of times and more likely to get blocked) this doesn t provide a great benefit. Also my past experience of laptops as servers is that they don t want to boot up when the lid is closed and getting RAID-1 and multiple ethernet ports on a laptop is difficult. Finally I am going to create a third DNS server for the sws.net.au domain. While it is more pain to run extra servers, for some zones it s just worth it.

• [1] http://www.zdnet.com.au/distributeit-data-unrecoverable-post-hack-339317149.htm
• [2] http://doc.coker.com.au/environment/computer-power-use/

Using a very customizable email user agent (namely, Mutt), I am able to use several signatures depending on almost whatever criterion I want. So I spent some time drawing several thematic ones in ASCII Art, sometimes taking inspiration on existing models, sometimes entirely by myself (in bold). Hoping that it may be useful, here are some of them:

Bee (a logo of my engineering school, Centrale Paris)	___ (___\ ,- <(((((>0 Tanguy Ortolo (___/ -
Fishes (for April 1st)	(\_ _/) )\/__( > Tanguy < )__\/( )/ '
Angel (for Christmas)	(\ _ ( \o /\/ Tanguy _)>
Cyclist	__o _<_\,_ Tanguy (_) '(_)
Fencer	< o /--)---- Tanguy / >
Cypherpunk	____ Tanguy Ortolo /_ \______________ ((_) _ .--^---^^-^-' \____/ ' urn:pgp:240BBA15B694DD00E38030D8D6EFA6AC4B10D847:
Hacker	. o . . . o Tanguy Ortolo o o o
Debianist	,--. : / ) Tanguy Ortolo -' Debian Maintainer \_

If you like them, do not hesitate to copy them. I am not sure whether or not this can seriously be an object of copyright, so in case it would be necessary: I put them, at least those that are of my own (those labelled in bold), under CC0 (unfortunately my country law do not allow me to put my works under public domain): Copying is an act of love. Please copy.

---

PS: Sorry, but my blog does not allow comments with fixed-width fonts

During Debconf 11, I got access to a fast s390 machine, and I have started to work on a Debian s390x port, the 64-bit version of the s390 port. One of my goal was to help the SPARC64 port, as some of the issues are the same: both are 64-bit big-endian, don t support unaligned access and behave differently between -fpic and -fPIC. Why such a port? When talking about 64-bit ports, we usually hear: 4GB is enough, handling 64-bit takes more memory . This really sounds like 640K ought to be enough for anybody . The s390 port is actually 31-bit from the address point of view (one bit is reserved for address space extension from 24 to 31 bits), so each process is limited to 2GB only. Nowadays applications which need more than 2GB are not that uncommon, especially on mainframes. Actually the 2GB limit already causes some problem in Debian: in some cases it s not possible to build haskell applications or even C applications using GCC. On the other hand, we already require a 64-bit kernel on the s390 port (only the userland is 32-bit), and applications are handling more and more 64-bit or greater values (files offset, time counters, uid, etc.). What is the status? Bootstrapping the architecture was not really easy (as for any other new architectures), due to a huge amount of dependencies and build-dependencies loops, as explained by Wookey during Debconf11. Now that this part is mostly done, an autobuilder has been started and currently more than 65% of the packages are built. The s390x port is hosted on debian-ports.org. Unfortunately it is not yet deboostrapable, though that should happen in the next few days (only a few packages are missing). The main issues are currently packages which fail to build from source due to linker, gcc-4.6 and curl changes, or due to the libjpeg and multiarch transitions, and thus are not directly related to s390x. If your package is in this case, it would be a good idea to fix it. Otherwise if it has a lot of reverse dependencies and the bug is opened for a while, just expect an NMU (as allowed by the 0-day NMU policy). Of course for a few packages s390x specific fixes are needed, some of them are already in the BTS. How you can help? The list of bugs blocking the s390x port is available through the s390x usertag, fixing these bugs (a lot of them are general FTBFS) would help a lot. Alternatively if you have access to an s390x machine you can take a look at the packages failing to build. Update: Fixed the explanation about the 32th bit, thanks to Bastian Blank for the comment.

Documentation! That is what the next version of Lintian will have. In 2.5.1, Lintian
had support for adding options in the lintianrc. Unfortunately, the
user had to guess the syntax, which makes the feature less useful.
With the next release, the manpage and the example config file[1]
will contain documentation and examples. The User Manual[2] also saw some improvements. Particularly, it will
mention that lintian can process a .changes file and that it does
cross-package checks. Of course, it also documents the format of the
new Vendor Profiles. Though even with these improvements, the current
documentation still leaves much to be wanted. The last unpack script (unpack-srcpkg-l1) was finally removed and
was replaced by a smarter version of index collection. This means
that Lintian can now schedule index in parallel with a few other
collections (namely unpacked, which is the current bottleneck). The speed improvement is probably not noticeable, but it was also a
good time to upgrade our index collection to properly handle
multi-tarball source packages. Previously Lintian only extracted the
index from the base tarball. The fields collection also saw some attention. Previously it
extracted fields from the control file (in the .deb, etc.) into a
directory. Each field became a file and the value of the field would
be written in the field. Today, the collection basically looks like
this:

[...]
if (-d 'fields') 
    system('rm', '-rf', 'fields') == 0
        or die 'Could not remove old fields directory';
 
[...]

It has been deprecated the minute Lintian::Collect API saw a little
upgrade to fetch these fields itself. This change should remove some
unnecessary file I/O in a normal run[3]. If you have a friend with his/her own little home-grown Lintian check
(or collection)[4], they will have to update it to A) not depend on
fields, B) use the Lintian::Collect API for accessing fields and C)
depend on index for source packages if it uses the index information.
For the latter, Lintian::Collect has grown support for source package
index. Lintian has also seen a lot of decrufting in the 2.5.2 development
cycle. The new test suite now features skeletons ( standard base
environments ) and template support for all package test suites[5]. This means it takes a lot less of writing (or copy/waste) to generate
a simple specialized test in the low-level test suites. As a part
of cleaning up the test-suites, I got to run:

$ git rm t/ source,debs /*/copyright

without breaking a single test. I have to admit that commit 8584996
is among my favorite ones in the 2.5.2 development cycle. Finally, if you are one of the few that have ever used the
--packages-file option, then I am pleased to inform you that it will
be deprecated in 2.5.2 and replaced by --packages-from-file. The
old option had some specialized format, whereas the new option parses
the file (or stdin) as one package per line . The latter being far
more "find -name '*.deb' "-friendly. In case you are wondering why --packages-file is deprecated, here is
the packages-file parser from lintian 2.5.1:

    while (my $line = <$pkgin>)  
	chomp($line);
	my (undef, undef, undef, $file) = split(/\s+/, $line, 4);
	$pool->add_file($file);
     

- [1] Installed as /etc/lintianrc [2] Installed as /usr/share/doc/lintian/lintian. html/,txt.gz [3] Namely, 2 opens + 2 closes + 1 read and one write for each field
in every package (the fields were cached, once they were read). Not
to mention a directory listing and having to remove the files again
later. [4] Of course, you do not have such a thing Because we all know
that no one (except this not too close friend of yours) messes with
Internal APIs of other projects [5] The exception being that the changes suite does not have a
standard base environment, since it would be completely unused.

Wondering what is going on with the port of Debian to the Minix 3 microkernal? Here's an update. dpkg Ported Well, the big news as I mentioned in my last post is that dpkg has been successfully ported. I haven't sent the patches to the maintainers just yet as they need review and clean up but I will do that ASAP. The architecture name is defined as minix_i386. (There is no k because I am using the Minix libc as well as the kernel.) I am currently targetting squeeze. A Maze of Twisty Dependencies All Different Armed with my new dpkg, I have begun building packages which has been an interesting experience. The Debian packaging systems greatest asset dependencies is also its most frustrating during initial bootstrapping. Take debhelper for instance; nowadays most packages depend on it and I personally wouldn't think of creating a .deb without it but it build-depends e.g. po4a which build-depends on...debhelper. Also some programs which are assumed to be present on linux such as ldconfig are simply not available on Minix. So currently theres a lot of dpkg --force-depends and even --force-all going on. Of course this is only a temporary problem until I get a full set of base packages built. The equivs package is coming in handy too. In lieu of working .debs, I have been using Minix native packages to fill in the gaps. They use NetBSD pkgsrc which is a new experience for me. I can't say I'm that impressed but they work and, being uptodate, are a good source of patches. Works in Progress The two big packages I need to work on next are perl 5.10.1 and gcc 4.4.5. I haven't touched the latter yet. Perl is failing in its second-stage bootstrap for reasons I haven't gotten to the bottom of just yet. I suspect it has more to do with deficiencies in my build environment than a problem in perl itself. libc Deficiencies As I mentioned last time, the situation has improved tremendously since Minix switched over to using the NetBSD libc but I am still running into problems with unimplemented APIs. Take getpwent(3) and getpwuid(3) for instance. Being able to map between user ids and names is kind of important you know? Even worse instead of being completely absent some of these exist as stubs. So they compile and link without warning but always return ENOFILE and so on. I was surprised to find that there is no standardized API for getting information on mounted filesystems. (Which e.g. df(1) needs.) Linux uses getmnt(3) similar but incompatible with System V. The BSDs use getmntinfo so probably this is what Minix will implement. GNU programs are good about portability but a lot of Free software is unfortunately Linux-centric. Hopefully the Debian FreeBSD port already has patches for these cases. There are several annoyances like this but I suppose I shouldn't complain too much. I am using an unreleased version of Minix which is under heavy development and I must say the upstream developers have been good about responding to my concerns. Looking forward: fakeroot, grub2, ext2 fakeroot is a very handy development tool that frees a developer from having to create .debs as root. Alas it won't work under Minix as it depends on LD_PRELOAD and Minix doesn't even support shared libraries. fakeroot-ng sounds a bit more promising as it uses the ptrace(2) system call which is implemented in the Minix kernel though I don't know how well. Minix is now multiboot compliant so it is possible to use grub2 instead of the Minix boot loader. Definitely something I'm going to look at soon. Minix now supports ext2 (though not ext3 or ext4) in addition to its native file system. For compatability this should become the default in the Debian port. While the kernel support is part of the standard Minix system, the userland support (e2fsprogs) is not for some reason. So I'll have to see about porting that. Can I Help the Porting Effort? Sorry not yet. Though if you are determined I'll try and help you get set up. Hopefully in a month or less I will be able to distribute a pre-alpha version so I suggest waiting for that. I am definitely going to be looking for outside help.

Sometime it happens, that there are school holidays. As my lovely girl used to be a teacher, she has some time off at that period. So she left me with our kids yesterday to visit the grandparents.
It was two hours after leaving, when I was really surprized that I was sitting around and thinking about what to do next, as normaly the weekend are family time.
So I decided to take my bike and a paper map to get of for a trip. Since some years, I usually use my bike to escort the girls or when going to work, which is just a 5 minutes ride, so I thought that would be a great opportunitiy.
I took a backpack with raining clothes, mounted the bike helmet on it, in case I needed to go cross country, some water and my music player.
It was a great ride, beside that I realized I m not in the same condition as some years ago. Unfortunately this will likely not change in the next 12 mounth, as we have some projects in the queue we need to deal with. Hopefully we will solve that satisfactorily and have a bit more spare time afterwards.

Anyways .. 27,65km in 1h40 is not so bad, as it was hot and humid.
Ride summary:

I got a call from Mum today to inform me that my grandmother had passed away last night, Brisbane time, at 94 years of age. If you want to get all genealogically correct, she's technically my step-maternal grandmother, but she's the grandmother I spent the most time with, growing up, so I'm not going to split hairs. It's come as a bit of a shock, because despite having had progressively worse dementia for probably over a decade, she's been in incredibly good physical condition. I think it was in her 70's that she had a heart valve replaced, and I for one had been expecting that to be the thing to wear out and take her down slowly. I last saw her in the flesh in January, when the above photo was taken, and more recently a few weeks ago on Skype. She was recently moved from an aged-care hostel in Toowoomba to a nursing home in Brisbane, as her dementia had progressed beyond the point that she could be cared for in the previous facility. Mum had visited her earlier in the day yesterday, and she was fine, but apparently the staff found her unconscious at around 5pm when they went to get her for dinner. She was taken to the Royal Brisbane and Women's Hospital where it was determined she had suffered from a stroke, and she passed away later that evening at around 11:30pm. So all in all, she had a very good innings, and of all the ways to leave this world, this was one of the quicker and more painless ways to do it. That's the problem with a quick exit - it's usually completely unexpected. It's looking like the funeral will be on Friday, and I've got a flight tentatively booked to leave on Wednesday and get in on Friday morning and return again on Monday. At the time I arranged this, I'd completely forgotten that I was flying to Calgary on Monday night to babysit the son of friends, while they go to the US Consulate for a visa appointment. Fortunately this travel doesn't conflict with that travel, it just means I'll be away from work for longer, and spending a lot of time inside pressurized tubes. I'm fairly confident with all of the short international trips, I'm going to look like a drug mule or something, and someone's going to thoroughly cavity search me.

FreedomBox activities at Debconf11 I spent the last two weeks of July 2011 in Banja Luka. The occasion was the annual Debian developer's conference, Debconf11 and preceding work week known as Debcamp. This was my tenth successive year attending Debconf, and I had a very productive and pleasant time! The facilities were good, the local team was friendly, enthusiastic, and very helpful, and in addition to giving three talks and hosting a couple panel discussions, I managed to put a burst of energy into work on FreedomBox. Several other developers working on FreedomBox were also present, and a good number of Sheeva and Dream plugs were evident in the hacklabs sporting new FreedomBox stickers. Working together in the same place for several days, we made good progress on several projects, and also had some great discussions about what we want to do going forward. image building tools For some time, I've been working towards a light-weight tool set to build FreedomBox software images. Shortly before Debconf started, I chose the name 'freedom-maker' for this tool and shared a link to a readable copy of my git repository with other developers I expected to work with in Banja Luka. With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker went from almost useful to actually useful. It still deserves work to be more useful to others, but I have now pushed a copy of the git repository to git.debian.org so that we can take advantage of the tools supported there to enable others to more easily contribute to the code. Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius' vmdebootstrap to build x86 images suitable for testing in a virtualized environment. At the same time, we plan to refactor the existing code slightly to enable lists of desired packages for the various image flavors we expect to produce independently of the configuration for each specific image building tool. Jonas continued in parallel to work on his alternate packaging toolset boxer. It offers some potentially interesting features for the future, and we may eventually merge some or all of it into freedom-maker, but for now it remains a separate utility. uAP user space tools Several weeks ago, we received from Marvell the source code to two user space programs that are necessary for configuring and monitoring the binary firmware provided for the uAP wireless chip used in the DreamPlug. Early during my stay in Banja Luka, I packaged these for Debian as uaputl and uapevent, and I am pleased to note that they were quickly accepted into the archive and are now present in Debian mirrors. u-boot Another bit of code received very shortly before Debconf started was the source for the version of u-boot shipped by Globalscale in the DreamPlug units we're working with. During Debcamp, Clint Adams passed a copy of this source to Jason Cooper, who was already trying to add support for the DreamPlug to upstream u-boot, but had stalled due to a lack of information. Jason has now merged his own work with the sources we got from the manufacturer, and is making good progress towards merging DreamPlug support into upstream u-boot. Once that happens, we should be able to flash our Sheeva and Dream plug devices with a u-boot image built from the source in the Debian u-boot package, in the process enabling things that matter to us like the ability to boot from an ext2 partition, and hopefully the ability to execute command scripts from that partition instead of having to hard-code kernel filenames in flash. This will allow us to support the ongoing effort in Debian to move away from the need for kernel symlinks. DreamPlug kernels With respect to kernels, another work stream at Debconf primarily involving H ctor Or n and Nick Bane was to analyze the current state of the patches from Marvell and Globalscale used to support the DreamPlug against both upstream and current Debian kernel sources. To my surprise and our collective pleasure, the remaining patch set required against current upstream kernels is much smaller than we previously believed! There are still several patches critical to us that are not merged upstream, but the work remaining to be able to build images for our devices from mainline and Debian kernel source trees now seems like something we might be able to complete before Debian's next stable release. One of our discoveries during the u-boot and kernel work during Debconf was that Globalscale did not obtain a new machine id for the DreamPlug, but instead re-used the one for the GuruPlug series, despite there being some differences in the hardware that require at least one additional driver. After much discussion, we plan to continue using the existing machine id instead of requesting another, particularly because the ARM kernel community has apparently stopped issuing new ids for the moment. We will add a new kernel config option for the DreamPlug, however, and are likely to build distinct Sheeva and Dream kernel packages that do not require initrd for use in FreedomBox images, even if doing so is not strictly necessary. This will allow us to optimize both the in-memory footprint and boot times for our devices. software configuration Another area of investigation in Banja Luka was technology for package configuration. Mirsal Ennaime performed various tests using debconf and Config::Model, with some results reflected in this commit relating to configuring the bitcoind daemon in the bitcoin package for Debian. identity and trust management While we did not actually do any FreedomBox specific work on the trust management layer we know is necessary, after several rounds of conversation, I am now more convinced than ever that the right path forward is to base our trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting software elements. Our thinking to date is captured on an Identity Management page in the wiki. communication services Another thing that became fairly clear to me during discussions at Debconf is that in the near term, planning to build communication services around XMPP is the approach most likely to give good results. Investigating the software choices available to build an interesting XMPP infrastructure is now a high priority for me. Jonas has done some work towards configuring and integrating ejabberd or Prosody, I've started studying yate as a possible call manager and VoIP server choice with XMPP/jingle support, and we await with great interest a release from the Buddycloud developers to evaluate as a possible basis for deploying social network services. Some of these software choices will lead us to use Apache as our web services base technology because of the need for features that only it supports well among daemons that are Free Software. Jonas completed packaging GNU Sip Witch for Debian, and it is now available in the mirror network. Tzafrir Cohen and Jonas did some initial testing on its use. documentation A number of new wiki pages were written (or at least started) in order to sum up ideas, design various aspects of FreedomBox, and reflect discussions that happened during DebConf11. A lot of work is needed to complete these pages though, as well as others to capture more of the current state of the project. press coverage Finally, while in Banja Luka I got some great press coverage for FreedomBox! On Sunday the 24th, I was interviewed by the main television network serving the Republika Srpska. This led to a couple of minutes of coverage near the top of the national news program that night, immediately following the lead story about the President and several ministers appearing at Debian Day that morning to help open the conference. This interview was later re-used in another TV program that summarized Debconf11. On the morning of Thursday the 28th, I was part of a small group that spent more than an hour meeting with the Minister of Science and Technology in his office, and the relationship between Debian and our work on FreedomBox was one of the items of discussion in that meeting and the associated press conference. I'm told this resulted in more press coverage, but if true I have not seen it yet. summary On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf! All in all, we got lots of work done on FreedomBox in Banja Luka, enough that I think at least the next few steps along the road towards an eventual "1.0" release of a reference implementation are now much clearer than they were two weeks ago!

Hello, this is the fourth report for the project. First, I d like to
apologize for being late, I took a short trip over the weekend and
missed the deadline. This time I ve good news for my project. As
expected I ve made the necesary changes to have Eucalyptus support ARM
images. The approach was extending and exposing an arch field that
will be used by libvirt s XML domain file. I m working currently on
having the involved functions working with the new extension and I
expect to have this done over the middle of the next week. What was done so far

- Extended the ncInstance_t struct on util/data.h adding the archId
parameter. This struct keeps the information regarding a node (nc),
say RAM, Kernel, etc. Currently it has no way to set an arch in order
to be used later. typdef struct ncInstance_t
...
char reservationId[CHAR_BUFFER_SIZE];
char userId[CHAR_BUFFER_SIZE];
char archId[CHAR_BUFFER_SIZE]; // arch field added
int retries;
...

The Eucalyptus team was OK with this approach that I ve proposed few
weeks ago. For all my changes the approach is maintain compatibility
with existing features.
We still have to define if we are setting a default value, say amd64,
or keep it unset until the user sets one. For now we will be supporting
amd64 and arm as valid fields. - Modified the allocate_intance() function to support the archId
field. This involved adding a new parameter and detecting whether its
present and setting de value accordingly. This function is later used
by the doRunInstance() function in node/handlers_kvm.c - Modified the involved functions that are called by the KVM handlers
and result in passing parameters to the XML file generator for being
used by libvirt s and KVM and have the image running under
Eucalyptus. The new parameters and field are not required and only
being used if they are set with a value. The archId parameter has been
added as the last one, so existing function calls can work. This could
be rearranged later, but will involve a more extensive revision with
Eucalyptus developers. - I ve set a git local repo in order to keep track of the changes to
Eucalyptus. Since this part is a sort of a Eucalyptus branch I took
this approach. Steffen and I are discussing whether to release this as
a pkg-eucalyptus branch (as dpatch, using existing packaging). Either
way I m putting the diff files form my git repo over the wiki and will
put them on my site too. I plan to use git-svn in order to send them
later to the pkg-eucalyptus repo once we are settled with this. - Updated the project s wiki[1] page with information regarding all that s
involved on my work, so it can be re-played or resumed later. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog What I plan to to over the next weeks
-
- Finish the modifications to the relevant functions. Expected dealine:
next week.
- Do integration tests:
Write a test program, there s an example under node/ that could be
helpful.
Generate the XML domain definition from Eucalyptus functions. The
full cycle: handler_kvm -> allocate_instance -> get_instance_xml ->
have the XML file properly generated -> qemu-kvm runs the instance
using the proper emulator (arm in our case).
- Test runnning the image I ve worked on the project s begining.
- Fixing bits and cooperate with the pkg-eucalyptus team on the
packaging bits.
- Talk to Eucalyptus so they adopt our patches.
- Rock and roll :) What would be nice to have after the project ends
-
- The Debian packaging *needs* work. I ve already discussed this on the
pkg-eucalyptus mailing list. I ve also committed myself to help on
that side even after the SoC ends.
- Talk with upstream and Ubuntu to coordinate the packaging efforts and
modifications to the software.
- Get adopters to use our work :) By now, few weeks about to finish the SoC, I can say that it s been a
good experience and as I stated I m planning to keep maintaing/working
on the software after that. I ve already joined the pkg-eucalyptus team
and plan to contribute actively. I m sad I didn t managed to get Debconf
this year, time was a constraint again, hopefully we can meet in 2012! I
guess that all by now! Best regards,
Rud