This is my fifth and final report about dextools for the 2011 Google Summer of Code program. During the summer, I worked with Matt Zimmerman and Stefano Zacchiroli to create a replacement web dashboard for the Debian Derivatives Exchange (DEX). The dashboard displays a list of projects and their respective tasks and then allows users to easily update the status of these tasks. The dashboard also contains two graphs for tracking the progress of projects and providing instant recognition of all contributions to a project. At the start of the summer, I knew that I wanted to to work on improving the DEX infrastructure. I had worked with the team on the ancient-merges project, and while a lot of good work was accomplished, the process for tracking our progress was a bit clunky and difficult to interact with. Anyone who read my initial proposal for the Summer of Code would probably agree that it was quite vague. I really did not have a solid plan for what I wanted to accomplish. This began to change after my initial phone call with my mentor, Matt Zimmerman. We decided that the first thing I would work on would be a basic dashboard. Our plan was to have all of the data stored on the Debian BTS. This would allow Debian Maintainers to benefit from DEX without needing to learn about DEX s tools and workflows. The dashboard would support multiple derivatives and multiple projects for each derivative. Each project would be made up of a list of tasks, where each task is linked to a BTS bug. I prepared a mockup and some initial code based on that plan. The dashboard was able to successfully display a list of BTS bugs and their information. It generated the lists by assuming that bugs would be usertagged with a user of debian-derivatives@lists.debian.org and a tag in the format of dex
. At this point, we started trying to make sure that the dashboard would work for things like the ancient-patches, python2.7, and the upcoming large-merges projects. It did not take long to determine that it would not work for all of these things. The ancient-patches project started with a simple list of patch names. While most of these patches eventually ended up as bugs on the BTS, they did not start this way. The dashboard would need to support specifying a list of task names. This meant that it would also need to store data itself and that not all data could be located on the BTS. This is when we first started to define what a task is. The dashboard tracks tasks. It does not directly track bugs or patches. A task is nothing more than a title and status with an optional assignee, note, and bug. This made it simple to convert a list of patch names or a list of bug numbers to a list of tasks. During the early early stages of the project, the dashboard moved around a lot. We hoped to have it end up on dex.alioth.debian.org, but we were not sure whether Alioth would meet all of our needs. This was right around the time of the Alioth migration. Thanks to some tips, I figured out that I could use a simple cronjob on wagner to pull the git repository for vasks. This would allow the running instance of the dashboard to always be up-to-date. The Alioth admins have also been quite supportive. They have installed several additional packages for me that were necessary to keep the dashboard running. From the start, Matt felt it was important to have some public documentation about the project. This would allow us to point people to something other than my blog posts. I decided to put this information on the wiki. At one point, I also had a copy of the wiki page stored in the git repository. This file was updated via a cronjob, and I then committed and pushed it when I had code changes. I ultimately decided that the best approach would be to maintain the page on the wiki. A basic README file is included in the git repository that simply links to the wiki page. On wagner, I have a cronjob running to pull a copy of the wiki page so that it can be accessed via the web. Since Matt was traveling/moving this Summer, he arranged for Stefano Zacchiroli to fill in for him as my mentor temporarily. It was at this point in the summer that I began working on the first graph script. The goal was to have a script to parse the list of tasks and generate a graph showing the number of open tasks versus time. This would allow us to easily track our progress, estimate when a project will be complete, and detect periods of slowing down. We decided that while we liked the Ubuntu burndown charts, they were a bit more complex than we needed, so we did not reuse their code. A dashboard that can t be modified is not that useful. An early goal of this project was to allow users to update the dashboard via the web. This proved to be a bit more difficult than I initially thought. First, all of the html files that make up the dashboard are generated by a script. This means that if you modify one of the html files directly, all changes will be lost the next time the script runs. When you make a change on a web form and hit submit, you usually expect to see the changes the next time you visit the page. In order to accomplish this, I made the form processing script directly modify the html files. However, at this point, we did not have a way to locally store extended information about tasks, causing the python script to delete all changes made via the web form. This issue was rectified fairly quickly. There is now a changes file that stores all information submitted via the web form. The python script reads this file and applies the changes when it is generating the html files. A second problem concerned the text inputs that were being used in most of the cells on the table. By default, some fields would have text cut-off if the string was too long. Text inputs have a size attribute that is meant to specify the number of visible characters. I attempted to set this attribute in a script to the length of the input s value. Strangely, the inputs ended up with a lot of extra whitespace following the text. This resulted in the table being stretched horizontally. After many hours of research and testing, I was unable to find a solution to this problem. We ultimately decided to simply remove the size attribute and accept that text will be cut-off. My code got some early testing from Allison Randal and other people working on the dex-ubuntu-python2.7 project. They were a huge help in providing feedback and finding bugs in the code. While the dashboard was unable to meet all of their needs (due to being in the early stages of development), I hope that it at least helped to make it easier to track the project s status. Whenever a script is accepting input from a user, it is important to validate it. Although the dashboard uses a select box to limit the choices for the status , it is still trivial to submit an arbitrary status for a task. That is why I added some validation to the form processing script. It will ignore any unknown values, ensuring that all tasks have a valid status. The other fields are a bit more tricky. They were designed to be arbitrary text fields. As a result, almost anything can be entered. There is no way to tell if something is a title or a person. We thought of a few ways we could change this, but most of them involved locking down the dashboard and restricting changes. The old dashboard treated the fields as arbitrary text, so we decided to not include any validation in the new one. One popular feature request was the ability to link directly to a project. Early versions of the dashboard had one main dex.html page that used javascript to pull in the various projects. To get around this, I first trying using the query string to allow users to specify a distribution and project. While this worked, it resulted in long and ugly URLs. It took a while for me to be able to implement a cleaner solution. However, I eventually split each project into its own static HTML file. This meant that people could simply copy the URL from the address bar to link to a particular project. The main reason for doing this was that we felt the typical person would be interested in working on a specific project; most people won t be interested in navigating between the different projects. This change also allowed links that utilized the #graph anchor to function properly. Before, due to the way the page went about loading the project, trying to specify a specific project and #graph in the URL did not work properly. There is a plugin for jquery that allows tables to be sorted by particular columns. For tables that just contain text, this works fine without any issues. However, once I started using select boxes and text inputs, things got a bit messed up. After some research, I finally figured out how to use addParser to instruct the tablesorter plugin in how to sort each column in the table. In some of the earlier versions, I used some zebra stripes to make the table easy to read. Thanks to a suggestion from Paul Wise (pabs), I got rid of this striping and modified the code to color the rows based on the task s status. Complete tasks are green, in progress tasks are yellow, and incomplete tasks are red. This makes it very easy to find tasks that still need work as well as measure the overall status of a project. Sometimes, a large task list can feel very intimidating. That is why I added the ability to hide completed tasks with the check of a box. Matt wanted to take this one step farther; he wanted the ability to also hide tasks with a closed bug. This is why you will find 2 checkboxes at the top of the dashboard that allow you to custimize which tasks are visible. Eventually, I might add support for specifying this in the URL to allow groups to link to a particular view of the dashboard. In order to make it as easy as possible for new contributors to get involved with DEX projects, I wanted to have a way to document what a project is about and how to handle tasks. I decided to use the wiki for this. All of the per-project documentation lives at http://wiki.debian.org/DEX//
. You can even take advantage of wiki markup to format the documentation. The script will parse the HTML files generated by the wiki, do some minor cleanup, and then display the documentation at the top of the page. My original plan was to use the ?action=raw output, but the lack of formatting proved too restrictive. My second plan was to try and use something like BeautifulSoup or a regex to filter non-whitelisted tags out of the wiki-generated html. This failed and proved to be unnecessary. The wiki does a pretty decent job of preventing malicious markup from ending up on the dashboard. There is a second graph that is displayed on the dashboard. This graph shows the number of tasks each person has completed. The graph is sorted so that the tallest bars on the left, and the goal is to provide some immediate recognition of work done by contributors. Since we essentially allow anyone to go ahead and modify the dashboard, we knew it was important to have a method in place for dealing with abuse. Any malicious edits made on the wiki can easily be reverted. This idea of reverting to an earlier revision is what inspired me to use a second local git repository to store the projects/ directory. This directory stores all of the changes made via the web interface. Whenever the form is submitted or the script runs to update the list of tasks, any changes made to the projects/ directory are committed. This means that if a malicious user decides to delete everything on the dashboard, we can quickly and easily revert the changes. The revision history might also be interesting to analyze at a future date. When multiple people are collaborating on a project, an issue tracker can be quite useful. For some reason, we did not set one up until quite late into the project. Despite that, we are still using it to track known bugs and feature requests. The issue tracker will become even more useful once the dashboard starts to be utilized by more people. I spent a fair bit of time working on having the dashboard immediately respond to changes. For example, if you change the status of a task, the row color will instantly update. The form will also be submitted in the background, eliminating the need to hit submit to save the change. I also use ajax in some new dialogs that allow the user to create new projects and/or tasks via the web. While these forms are submitted in the background via ajax, I am unable to have the changes show up on the dashboard immediately. The generation of the task list is a bit slow, so having the script run more often is not really a good idea. The next step for the dashboard is to really open it up for testing. We hope to start up a new DEX project shortly. This will allow us to see which features work and which need fixing. It will also help us find some of the bugs that are probably hiding in the code. Finally, if you are interested in helping out with DEX or dextools, my code is available in a git repository (http://anonscm.debian.org/gitweb/?p=dex/gsoc2011.git) on alioth. There is also the issue tracker (https://alioth.debian.org/tracker/index.php?group_id=100600&atid=413120) where you can report bugs and request new features. Finally, you can join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler). I have really enjoyed working on dextools this summer. I would like to thank Matt Zimmerman, Stefano Zacchiroli, and everyone else who helped with the project. I look forward to working with all of you more in the future.

Here is the short version: to get better performance on your SSL terminator, use stud on 64bit system with patch from meric Brun for SSL session reuse with some AES cipher suite (128 or 256, does not really matter), without DHE, on as many cores as needed, a key size of 1024 bits unless more is needed.

Introduction A quick note: when I say SSL, you should read TLS v1 instead. Look at the article for TLS on Wikipedia for some background on this. One year ago, Adam Langley, from Google, stated SSL was not computationally expensive any more:

In January this year (2010), Gmail switched to using HTTPS for everything by default. Previously it had been introduced as an option, but now all of our users use HTTPS to secure their email between their browsers and Google, all the time. In order to do this we had to deploy no additional machines and no special hardware. On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. Many people believe that SSL takes a lot of CPU time and we hope the above numbers (public for the first time) will help to dispel that. If you stop reading now you only need to remember one thing: SSL/TLS is not computationally expensive any more.

This is a very informative post containing tips on how to enhance SSL performance by reducing latency. However, unlike Gmail, you may still be worried by SSL raw performances. Maybe each of your frontends is able to serve 2000 requests per second and CPU overhead for SSL is therefore significative. Maybe you want to terminate SSL in your load balancer (even if you know this is not the best way to scale).

Tuning SSL There are a lot of knobs you can use to get better performances from SSL: choosing the best implementation, using more CPU cores, switching to 64bit system, choosing the right cipher suite and the appropriate key size and enabling a session cache. We will consider three SSL terminators. They all use OpenSSL behind the hood. stunnel is the oldest one and uses a threaded model. stud is a recent attempt to write a simple SSL terminator which is efficient and scalable. It uses the one-process-per-core model. nginx is a web server and it can be used as reverse proxy and therefore act as SSL terminator. It is known to be one of the most efficient web server, hence the choice here. It also features built-in basic load balancing. Since stud and stunnel does not have this feature, we use them with HAProxy, an high performance load-balancer that usually defers the SSL part to stunnel (but stud can act as a drop-in replacement here). Those implementations were already tested by Matt Stancliff. He first concluded nginx sucked at SSL then that nginx did not suck at SSL (in the first case, nginx was the only one to select a DHE cipher suite). The details were very scarce. I hope to bring here more details. This brings the importance of the selected cipher suite. The client and the server must agree on the cipher suite to use for symmetric encryption. The server will select the strongest one it supports from the list proposed by the client. If you enable some expensive cipher suite on your server, it is likely to be selected. The last important knob is SSL session reuse. It matters for both performance and latency. During the first handshake, the server will send a session ID. The client can use this session ID to request an abbreviated handshake. This handshake is shorter (latency improvement by one round-trip) and allows to reuse the previously negociated master secret (performance improvement by skipping the costliest part of the handshake). See the article from Adam Langley for additional information (with helpful drawings). The following table shows what cipher suite is selected by some major web sites. I have also indicated wheter they support session reuse. See this post from Matt Palmer to know how to check this.

Site	Cipher	Session reuse?
www.google.com	RC4-SHA
www.facebook.com	RC4-MD5
twitter.com	AES256-SHA
windowsupdate.microsoft.com	RC4-MD5
www.paypal.com	AES256-SHA
www.cmcicpaiement.fr	DHE-RSA-AES256-SHA

RFC 5077 defines another mechanism for session resumption that does not require any server-side state. I did not investigate much how this works but it does not rely on the session ID.

Benchmarks All those benchmarks have been run on a HP DL 380 G7, with two Xeon L5630 (running at 2.13GHz for a total of 8 cores), without hyperthreading, using a 2.6.39 kernel (HZ is set to 250) and two Intel 82576 NIC. Linux conntrack subsystem has been disabled and file limits have been raised over 100,000. A Spirent Avalanche 2900 appliance is used to run the benchmarks. We are interested in raw SSL performances in term of handshakes per second. Because we want to observe the effects of session reuse, the scenario runs by most tests lets clients doing four successive requests and reusing the SSL session for three of them. There is no HTTP keepalive. There is no compression. The size of the HTTP answer from the server is 1024 bytes. Those choices are done because our primary target is the number of SSL handshakes per second. It should also be noted that HAProxy alone is able to handle 22,000 TPS on one core. During the tests, it was never the bottleneck.

Implementation We run a first bench to compare nginx, stud and stunnel on one core. This bench runs on a 32bit system, with AES128-SHA1 cipher suite and a 1024bit key size. Here is the result for stud (version 0.1, with meric Brun s patch for SSL session reuse): The most important plot is the top one. The blue line is the attempted number of transactions per second (TPS) while the green one is the successful number of TPS. When the two lines start to diverge, this means we reach some kind of maximum: the number of unsuccessful TPS also starts to raise (the red line). There are several noticeable points: the maximum TPS (788 TPS), the maximum TPS with an average response time less than 100 ms (766 TPS), less than 500 ms (776 TPS) and the maximum TPS with less than 0.01% of packet loss (783 TPS). Let s have a look at nginx (version 1.0.5): It is able to get the same performance as stud (763 TPS). However, over 512 TPS, you get more than 100 ms of response time. Over 556 TPS, you even get more than 500 ms! I get this behaviour in every bench with nginx (with or without proxy_buffering, with or without tcp_nopush, with or without tcp_nodelay) and I am unable to explain it. Maybe there is something wrong in the configuration. After hitting this limit, nginx starts to process connections by bursts. Therefore, the number of successful TPS is plotted using a dotted line while the moving average over 20 seconds is plotted with a solid line. On the next plot, you can see the compared performance of the three implementations. On this kind of plot, the number of TPS that we keep is the maximum number of TPS where loss is less than 0.1% and average response time is less than 100 ms. stud achieves a performance of 766 TPS while nginx and stunnel are just above 500 TPS.

Number of cores With multiple cores, we can affect several of them to do SSL hard work. To get better performance, we pin each processes to a dedicated CPU. Here is the repartition used during the tests:

	1 core	2 cores	4 cores	6 cores
CPU 1, Core 1	network	network	network	network + haproxy
CPU 1, Core 2	haproxy	haproxy	haproxy	SSL
CPU 1, Core 3	SSL	SSL	SSL	SSL
CPU 1, Core 4	-	SSL	SSL	SSL
CPU 2, Core 5	-	-	network	SSL
CPU 2, Core 6	-	-	SSL	SSL
CPU 2, Core 7	-	-	SSL	SSL
CPU 2, Core 8	system	system	system	system + haproxy

Remember, we have two CPU, 4 cores on each CPU. Cores on the same CPU share the same L2 cache (on this model). Therefore, the arrangement inside a CPU is not really important. When possible, we try to keep things together on the same CPU. The SSL processes always get exclusive use of a core since they will always be the most busy. The repartition is done with cpuset(7) for userland processes and by setting smp_affinity for network card interrupts. We keep one core for the system. It is quite important to be able to connect and monitor the system correctly even when it is loaded. With so many cores, we can afford to reserve one core for this usage. Beware! There is a trap when pining processes or IRQ to a core. You need to check /proc/cpuinfo to discover the mapping between kernel processors and physical cores. Sometimes, second kernel processor can be the first core of the second physical processor (instead of the second core of the first physical processor). As you can see in the plot above, only stud is able to scale properly. stunnel is not able to take advantage of the cores and its performances are worse than with a single core. I think this may be due to its threaded model and the fact that the userland for this 32bit system is a bit old. nginx is able to achieve the same TPS than stud but is disqualified because of the increased latency. We won t use stunnel in the remaining tests.

32bit vs 64bit The easiest way to achieve a performance boost is to switch to a 64bit system. TPS are doubled. On our 64bit system, thanks to the use of a version of OpenSSL with the appropriate support, AES-NI, a CPU extension to improve the speed of AES encryption/decryption, is enabled. Remaining tests are done with a 64bit system.

Ciphers and key sizes In our tests, the influence of ciphers is minimal. There is almost no difference between AES256 and AES128. Using RC4 adds some latency. The use of AES-NI may have helped to avoid this latency for AES. On the other hand, using a 2048bit key size has a huge performance hit. TPS are divided by 5.

Session cache The last thing to check is the influence of SSL session reuse. Since we are on a local network, we see only one of the positive effects of session reuse: better TPS because of reduced CPU usages. If there was a latency penalty, we should also have seen better TPS thanks to the removed round-trip. This plot also explains why stud performances fall after the maximum: because of the failed transactions, the session cache is not as efficient. This phenomenon does not exist when session reuse is not enabled.

Conclusion Here is a summary of the maximum TPS reached during the benchmark (with an average response time below 100ms). Our control use case is the following: 64bit, 6 cores, AES128, SHA1, 1024bit key, 4 requests into the same SSL session.

Context	nginx 1.0.5	stunnel 4.41	stud 0.1 (patched)
1 core, 32bit	512 TPS	503 TPS	766 TPS
2 cores, 32bit	599 TPS	-	-
32bit	804 TPS	501 TPS	4251 TPS
-	1799 TPS	-	9000 TPS
AES256	-	-	8880 TPS
RC4-MD5	-	-	7370 TPS
2048bit	-	-	1643 TPS
no session reuse	-	-	5844 TPS
80 requests per SSL session	-	-	10797 TPS

Therefore, stud in our control scenario is able to sustain 1500 TPS per core. It seems to be the best current option for a SSL termination. It is not available in Debian yet, but I intend to package it. Here is how stud is invoked:

# ulimit -n 100000
# stud  -n 2 -f 172.31.200.15,443 -b 127.0.0.1,80 -c ALL \
>   -B 1000 -C 20000 --write-proxy \
>   =(cat server1024.crt server1024.key dhe1024)

You may also want to look at HAProxy configuration, nginx configuration and stunnel configuration.

(This was going to be a comment on this blog post, but it s a Turdpress site that wants JS and cookies to comment. Bugger that for a game of skittles. Rimuhosting s recent extended outage due to power problems was apparently caused by a transfer switch failure at their colo provider. This has led people to wonder if putting UPSes in individual racks is a wise move. The theory is that in the event of a small outage, the UPS can keep things humming, and in an extended outage you can gracefully shut things down rather than having a hard thump. I happen to think this theory is bunkum. Your UPS is a newly instituted single point of failure. I d be willing to bet that the cost of purchasing, installing, and maintaining the UPSes, as well as the cost of the outages that inevitably result from their occasional failure, would be far greater than the cost of the occasional power outage you get in a well-managed facility. Good facilities don t have small outages. They don t have squirrels in the roof cavities, and they don t have people dropping spanners across busbars. The only outages they have are the big ones, when some piece of overengineered equipment turns out to be not so overengineered the multi-hour (or multi-day) ones where your UPS isn t going to stop you from going down. Your SLA credit and customer goodwill is already toast, so all you re saving is the incremental cost of a little bit more downtime while you get fscks run. If you want the best possible power reliability, get yourself into a really well engineered facility, and run dual-power on everything. Definitely run the numbers before you go down the UPS road; I ll bet you find they re not worth it.

<meta content="text/html; charset=utf-8" equiv="content-type">

Hi everybody,

This is the 5th and the final report of my Google Summer of Code project for Debian,

Python Multi-Build for Python Extensions Packaging.

During the Google's Summer of Code 2011 period, I coded a tool to packaging Python extensions which can be built by distutils, autotools, cmake and scons. I finished tool for Python3.X (py3build), its CDBS files and working on integrating it with debhelper's sequencer and the tool for Python2.X (pybuild) and its BDS/DebHelper files. I will surely continue after GSoC finished and will continue improving the tool. The tool's git repository is here[1].

I'm a bit behind of my schedule. I finished all plugins for Python3.X except packaging module. Instead of that I worked on CDBS and DebHelper files. I finished CDBS files, also write a DebHelper but we decided to do DH file a different way. So I need to change it. I will port the tool to Python2.X.

In git repository there is contrib/ directory to contain CDBS and DebHelper files. CDBS files are in contrib/cdbs/ directory and debhelper files are in contrib/debhelper/ directory. I put original files if they exists (with .orig suffix) and modified files. I also created a new file for CDBS, called python-py3build.mk.in for handling all build systems. And modified python-distutils.mk.in and pyt<wbr>hon-autotools.mk.in files. All they are in git repository.

I'm still working on DebHelper file. After that I'll work on pybuild. And surely testing all of them and if any bug or problem occurs I will fix it.

Also, I need to improve my build system detection algorithm. At first, I thought that if there is multiple build system matches then show which are usable and then exit but I realized that in CDBS and DebHelper I need one build system. So even if there is multiple matches, it should return something. I'm also working on that.

For now that's all. Thanks everybody. It is a pleasure to contribute to Debian and participating from Google's Summer of Code. I will be in touch with Debian Project as a contributor.

[1] git://git.debian.org/~<wbr>mesutcan-guest/py3multibuild.<wbr>git

So, on July 31st 2011, it was exactly 10 years since I am a Debian developer. What happened in the meantime? What lead me to this? What turned this unskilled dude into a sometimes quite visible contributor of one of the major free software projects? If you're interested in that, please read on. Otherwise, well, this is just yet another "bubulle talks about self" post and you can skip it. Well, first of all, how did I end up being a DD? And first of firsts, how did I end up not being a random user of Windows, playing games on his desktop computer at work? I am not a computer scientist, an "informaticien" as we sometimes say in French (most often, what people who are not in computing stuff say...thinking that all folks working more or less closely to computers know everything about them). I graduated with a PhD in Materials science, in 1989 after conducting a research on "Influence of Yttrium Oxide dispersion on the strength of titanium alloys", at Onera, a French public research institution for aerospace and defence. I was then hired, still at Onera (where I'm still working, 25 years after starting my PhD work) to lead the Mechanical Testing Laboratory in the Materials Science department. So, my lab had many big machines designed to conduct creep and fatigue tests often at interestingly high temperatures such as 2000 C, on samples of various materials that are used in aircrafts structures, engines, etc. or in various space thingies (remember Herm s, the european shuttle?) or in various "things that fly but just one way and you shouldn't be there when they land". So, we had computers handling data acquisition for these tests. So I became involved with designing data acquisition setups, or even programming data acquisition programs (one of mine, written in Forth, ran all Onera creep tests until 2004 and successfully passed Y2K because I knew that 2000 was a leap year. It could even pass 2100 as I knew this is not a leap year..:-) One day, I had to buy a modem in order to communicate with out italian counterpart (CIRA) and exchange tests results. Then I had to learn how modems work in MS-Dos(yeah...). Then, I discovered "online" resources...indeed more that strange world that was then called "BBS" (Bulletin Board Systems), those mysterious things ran by some happy few who were using modems at their home place to communicate in "forums" and everything related to technology. PC-Board, Remote Access, Fidonet, etc. became familiar to me these days, back in 1988-1990. So familiar that I ended up running my own BBS at home, killing our phone bills and using very sophisticated techniques such as US Robotics "High Speed Transfer" modems that could be used at 19200bps asymetrical for very high speed transfers of kilobytes and kilobytes of useless MS-Dos "freeware" and "shareware". Indeed, my very first BBS didn't even have one of these: it was using a cheaper V.22bis modem operating at 2400bps. I was waiting for my order of a sophisticated HST modem to arrive from USA through obscure import channels meant to circumvent the French telephone company regulations that were forbidding the use of "unapproved" systems, in order to protect the famous French Telephone System from interferences brought by Bad American Material. Bubulle System was born. During those years, I discovered very interesting things: computers can run together once you draw a wire between them. That's called Local Area Network and you can even transfer data at 1Mbps between two computers, assuming you don't forget to put terminating resistors at the end of the line on these funny 10-baseT connectors at the back of your home-assembled PC that was using 2MB of RAM (bought for very cheap through the help of an American friend who was in touch with some Chinese folks who were selling 256kb RAM sticks for half the retail price....assuming you want to drive in a mysterious storing place close to Charles de Gaulle airport or Eurodisney). Hello Gordon. Yes, I know, we're still friends on Facebook and you're probably still using that weird programming language which you were, IIRC, the only person in the world to use. 2MB, that should be enough for barely anything, including running *multitasking* software where, miracle, I could run two tasks at the same time on my one and only PC at home. Miracle, I don't have to shutdown my BBS if I want to read forums on my friends BBS. Yay for Desqview/386! Multitasking for dummies Still, I have to build one of these "networks" at home. Elizabeth won't like that as it means one more box (home-made of course) in our living room and some more wires. And why the hell is it running 24/7? So that friends can visit the BBS, darling... And I can even communicate with them: I write a message, I get an answer the day after and so on. In one full week, we can have a great conversation that would have taken *minutes* to have in the real life. Isn't this the miracle of technlogy? And, yes, this is a good reason for having phone bills raising up to 500 Francs/month: people can "exchange" programs through my BBS, that horrible white box running in the living room. Often, these programs are written for free and some of them are even given with "source code", which allows people to *modify* them. And that even makes friends, you know? Imagine that some day we have to move from one house to another: then I can just call out "who wants to help bubulle moving?", and probably a dozen of (sometimes scary but always nice and polite...and sometimes even showered) geeks will pop up and happily carry boxes full of my vinyl LP collection all day long. An entire world of friends. "bubulle", you say, dear? What's that? That's my nickname. It was invented by one of these friends, a really strange guy named Ren Cougnenc who wrote this "free software" program anmed BBTH, which allows you to use modems to connect to BBS, and even to those many "Minitel" BBS we have in France, thanks to our wonderful world-leading technology using V.23 communication. Many people know me as "bubulle" because, you know, Perrier water has bubbles and Ren likes Gaston Lagaffe fish companion who he named "bubulle". Ren , I love you. You chose to leave this world back in 1998. We'll no longer have our "p'tits midis" in Antony where you were showing me the marvels of what's coming in next episode. All this was around 1988 and about 1992, doing all these mysterious things at home (between Jean-Baptiste and Sophie's diapers) while still working with data acquisition MS-Dos machines at work. How did this end up being a Debian Developer? You'll know in the next episode..:-)

It took some time but it finally happened! Rule of thumb: if you're happy with Squeeze s X, stick to it. If you need a newer driver, for example for Intel Sandy Bridge, then give squeeze-backports a shot, and cross fingers! As usual, x.debian.net contains the relevant documentation. Hopefully, the short squeeze-backports instructions will be sufficient. If you encounter some issues, feel free to contact debian-x@.

Linux 2.6.24, released more than three years ago, introduced two very interesting features: PID and network namespaces. They allow to create a process that does not share the same PID namespace or the same network namespace than other processes. Those namespaces are then inherited by the children (unless they are assigned new namespaces).

PID & mount namespaces A PID namespace is a view of the processes running on the system. Each PID namespace will have its own numbering of the processes contained in its namespace (including the namespace of the children). Other processes are hidden and therefore, it is not possible to issue any operation requiring a PID on them, like kill(2). Moreover, the process that was created first in a namespace will get PID 1. If it gets killed, the namespace is destroyed and all other processes contained in this namespace are killed as well. systemd, a system and service manager for Linux, targeted as a replacement for the old SysV init, uses could use this feature to launch daemons in their own PID namespace and keep track of them without relying on their cooperation. When it needs to kill a daemon, it just kills the process with PID 1 (which has another PID in the global namespace) and the kernel will do the hard work of killing all processes that were started in this namespace. You can get more details about this namespace in the manual page of clone(2) (search for CLONE_NEWPID). UPDATED: In fact, systemd does not use PID namespaces but cgroups to group processes together. There is no PID numbering here but spawned processes are all enclosed into one cgroup and therefore can be killed easily. For example, here is the result of pstree -p inside a PID namespace:

root@guybrush:/# pstree -p
bash(1) pstree(8)
         sleep(7)

The PID of sleep is 7. In the global namespace, it is 31762 instead. But it is the same processus!

root@guybrush:/# ps -eo pid,command   grep sleep
31762 sleep 1000

Linux have also supported mount namespaces since a long time. Each process can have its own mount points that will be visible only by processes in the same namespace or in children namespaces. When the namespace is destroyed (when no process left), the kernel will do the cleanup itself.

chroot A common way to test experimental software or do packaging work is to build a chroot: debootstrap is used to build a new complete system inside a directory and chroot allows to enter into this new system. Tools like schroot can become handy if you need to do that often. Unfortunately, you only get an isolated filesystem. When you start launching daemons and mounting partitions, you need to keep track of all your actions to undo them. Quitting the chroot will not kill any daemon nor unmount any partition. Extending an existing tool, like schroot to use PID and mount namespaces seems the best way to prevent such problems. I have filed bug #637870 for this purpose. However, this is not as simple as it seems since schroot allows to run a process into an existing session. When the session is a chroot, this is quite easy: just chroot into the same root filesystem and launch the process here. With other kind of namespaces, you need to use a new system call, setns(), to change the current namespace of the running process. However, some bits are still missing. First, the GNU C Library does not implement the appropriate wrapper. Second, Linux (as of 3.0) does not contain the support to change the PID namespace of a running process yet. Other tools exist, like lxc (aka Linux Containers). However, it is more targeted at providing a complete container. I have reported that lxc-execute is not able to spawn a standalone shell but got no luck. UPDATED: systemd ships with systemd-nspawn which was exactly what I needed. However, it requires systemd. Otherwise, it seems less basic than the alternative that I describe below (some sensible mount points are setup, the console is correctly configured, signals are correctly transmitted, ). If you use systemd, look at it!

jchroot Building a chroot clone using namespaces is quite easy. The only difficulty is the use of clone(2). Here is a sketch:

int main()  
  pid_t pid;
  int ret;
  long stack_size = sysconf(_SC_PAGESIZE);
  void *stack = alloca(stack_size) + stack_size;
  /* New process with its own PID/IPC/NS namespace */
  pid = clone(step2,
              stack,
              SIGCHLD   CLONE_NEWPID   CLONE_NEWIPC  
              CLONE_NEWNS   CLONE_FILES,
              NULL);
  /* Wait for the child to terminate */
  while (waitpid(pid, &ret, 0) < 0 && errno == EINTR)
    continue;
  return WIFEXITED(ret)?WEXITSTATUS(ret):EXIT_FAILURE;
 
static int step2(void *a)  
  /* Mount /proc as an example */
  mount("proc", "/proc", "proc", 0, "");
  /* Chroot */
  chroot("/srv/debian");
  chdir("/");
  /* Drop privileges */
  setgid(1000);
  setgroups(0, NULL);
  setuid(1000);
  /* Exec shell */
  execl("/bin/bash", "/bin/bash", NULL);
 

Add command flags, error handling and fstab parsing for the definition of mount points and you get jchroot. Here is an example of use:

$ cat fstab
proc     /proc  proc    defaults                  0  0
sys      /sys   sysfs   defaults                  0  0
/home    /home  none    bind,rw                   0  0
/dev/pts /dev/pts none  bind,rw                   0  0
/var/run /var/run tmpfs rw,nosuid,noexec,mode=755 0  0
/etc/resolv.conf /etc/resolv.conf none bind,ro    0  0
$ sudo jchroot -f fstab -n test /srv/debian/squeeze /bin/bash
# hostname
test
# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19480  1888 pts/1    S    15:18   0:00 /bin/bash
root         3  0.0  0.0  16536  1184 pts/1    R+   15:18   0:00 ps auxww
# /etc/init.d/postgresql start
Starting PostgreSQL 9.0 database server: main.
# exit
$ ps auxwwc   grep postgres
$

I hope that jchroot won t live for too long and that its features will be integrated into regular chroot tools. I was thinking at proposing a patch for chroot(8). Anyone knows if this would have a chance to be accepted?

This year without our friends from Grml, but The MirOS Project (all two active developers and our Booth Babe gecko2@) will of course attend FrOSCon, nicknamed Froschkon, again. We ll have a pre-event meal time at my favourite Jugoslawian Restaurant on Friday (20:00 CEST) contact me privately for the co rdinates if interested. On Saturday and Sunday we ll staff a booth and answer questions about the many projects we have (more or less) running, including but not limited to paxmirabilis (aka MirCPIO), The MirBSD Korn Shell aka mksh(1), jupp the editor, and developers private projects such as slowly undermining Debian or Google-Go. While slow we are still working on World Domination. And teaching people good shell programming by example code. We might even bring CDs, but I m still working on the ISO last night s build aborted because the OS grew a bit making the floppy image not fit any more. (Solution, drop ping(8) and rtsol(8), but re-add sf(4) and bce(4) now that they fit again.)

A very popular refrain on tech sites is that the high street cannot support multiple competing phone ecosystems. It s a reasonable position to take. Do phone stores want to train their employees on six or seven different OSes? Do consumers understand the differences, and should they need to? Do app developers want to rewrite their app six or seven times? The obvious answer to all of these is no . Stores only want to train their employees once, developers want to write their app once. Choice is bad, because choice is complicated. You can have it in any colour, so long as it s black. The problem is, we need those competing efforts, for the entire market to increase in quality. Until the iPhone appeared on the market, Google s nascent mobile phone OS looked more like a poor man s Blackberry OS than what ended up shipping on the HTC Dream and later devices. Windows Phone 7 didn t ship with any multitasking at all and now the Mango update will be incorporating a multitasking model largely thieved from WebOS. iOS only just got support for updating the phone s firmware without being plugged into a host PC via USB, something which a few Android devices and all WebOS devices have always supported. Consider, for comparison, the web browser market. It would be easiest if there was only one web browser but that browser would quickly stagnate and cause pain, the way Internet Explorer 6 did when it had almost the entire market. It wasn t until upstarts like Firefox and Opera and Chrome and more started showing off their unique ideas that the entire pack started improving including IE, with IE9 supporting most of the features that the competition introduced. Every smartphone shipping today has something to offer that other devices don t and every smartphone OS shipping today has something to offer that other OSes don t. iOS has the widest app catalog. Android is Open Source (FSVO Open ). WebOS has the best multitasking, and Just Type . Windows Phone 7 offers a drastic new user interface paradigm. Blackberry OS is built for messaging tasks, and has the best sysadmin control. Symbian is well, okay, sometimes there s a time to let go but the same applies for web browsers too. Smartphones are communication devices that people use every day of their lives, and every consumer has different needs we shouldn t try to push them into an ill-fitting category, just to satisfy ourselves that it s probably only possible for three mobile platforms to succeed in the mass-market . For some people, Blackberry OS really *is* the best choice, and no matter how much you pretend, an iPhone would enhance their lives. And as for the app question tough shit. Plenty of app developers only target iOS even though Android is overtaking in the market, and if they really want to reach as many people as possible, then use a cross-platform framework like PhoneGap or some of Xamarin s products. Single-ecosystem apps are a low-effort push to reach as many people possible with minimal investment, and the only way to satisfy that class of developer is to eliminate ALL competition in the marketplace iOS-only devs are this decade s IE6-only devs. Everyone is entitled to their opinion, but I for one am sick of all the tales of doom and gloom that immediately surround any effort which isn t iOS or pure Android. We have a marketplace of ideas, which we should be celebrating, and not dumping on. I m eagerly awaiting delivery of my HP Pre 3, with all the unique possibilities it offers me which simply another Android device wouldn t have. And the big difference between WebOS and Nokia s Maemo/MeeGo efforts, for those who are still doubtful, is HP haven t spent years trying to deliberately sabotage the platform. Give it time. If I m wrong, the worst-case scenario is being lumbered with an awesome phone.

You might not know who Peter is because he s not very visible on Debian mailing lists. He s very active however and in particular on IRC. He was an admin of the OFTC IRC network at the time Debian switched from Freenode to OFTC. Nowadays he s a member of the Debian System Administration team who runs all the debian.org servers. If you went to a Debconf you probably met him since he s always looking for new signatures of his GPG key. He owns the best connected key in the PGP web of trust. He also wrote caff a popular GPG key signing tool. Raphael: Who are you? Peter: I m Peter Palfrader, also known as weasel. I m in my early 30s, born and raised in Innsbruck, Austria and am now living and working in Salzburg, Austria. In my copious free time, other than help running Debian s servers I also help maintaining the Tor project s infrastructure. Away from the computer I enjoy reading fiction (mostly English language Science Fiction and Fantasy), playing board games and going to the movies. Weather permitting, I also occasionally do some cycling. Raphael: How did you start contributing to Debian? Peter: I installed my first Debian the week slink came out. That was Debian 2.1 for the youngsters, in early 1999. The one thing I immediately liked about slink was that Debian s pppd supported RAS authentication which my university s dial-up system required. No way I d go back to SuSE 5.3 when I had working Internet with my Debian box. During that year I started getting involved in the German language Debian channel on IRCnet which got me in contact with some DDs. Christian Kurz (<shorty>) was working on Debian QA at the time and he asked my help in writing a couple of scripts. Some of that work, debcheck, still produces parts of the qa.d.o website, tho the relevance of that nowadays is probably negligible. While trying to learn more Perl earlier, I had written a program to produce syntax highlighted HTML for code snippets in various languages. I didn t really know what I was doing but it kinda worked, and probably still does since I still get mail from users every now and then. I figured that it would be really nice if people could just get my software together with Debian. According to code2html s Debian changelog the initial release of the package was done on a weekday at 2:30 in the morning early in 2000, and if my memory serves me correctly, shorty uploaded it shortly afterwards. I started packaging a couple of other piece of software and in the same year I sent my mail to the debian account managers to register my intent to become a DD. No new developers where being accepted at that time since the DAMs wanted to overhaul the entire process so I wasn t surprised to not get any immediate reply. Of course what the silence also meant was that the mail had been lost, but I only learned of that later when I took all my courage to ask DAM about the status of application a couple months later. Once that was sorted out I was assigned an AM, did the usual dance, and got my account late in November 2000. Raphael: Four years ago, the Debian System Administration team was a real bottleneck for the project and personal conflicts made it almost impossible to find solutions. You were eager to help and at some point you got dropped as a new member in that team. Can you share your story and how you managed the transition in the difficult climate at that time? Peter: Ah, that was quite the surprise for an awful lot of people, me included. Branden Robinson, who was our DPL for the 2005-2006 term, tried to get some new blood added to DSA who were at the time quite divided. He briefly talked to me on IRC some time in summer 2005, telling me I had come recommended for a role on the sysadmin team . In the course of these 15 minutes he outlined some of the issues he thought a new member of DSA would face and asked me if I thought I could help. My reply was cautiously positive, saying that I didn t want to step on anybody s toes but maybe I could be of some assistance. And that was the first and last of it, until some fine November day two years later I got an email from Phil Hands saying I ve just added you to the adm group, and added you to the debian-admin@d.o alias. and welcome on board . *blink* What!? My teammates at the time were James Troup (elmo), Phil Hands (fil), Martin Joey Schulze and Ryan Murray (neuro). The old team, while apparently not on good terms with one another, was however still around to do heavy lifting when required. I still remember when on my first or second day on the team two disks failed in the raid5 of ftp-master.debian.org aka ries. Neuro did the reinstall once new disks had arrived at Brown University. I m sure I d have been way out of my league had this job fallen to me. Fortunately my teammates were all willing and able to help me find whatever pieces of information existed that might help me learn how debian.org does its stuff. Unfortunately a lot of it only existed in various heads, or when lucky, in one of the huge mbox archives of the debian-admin alias or list. Anyway, soon I was able to get my hands dirty with upgrading from sarge to etch, which had been released about half a year earlier. Raphael: I know the DSA team has accomplished a lot over the last few years. Can you share some interesting figures? Peter: Indeed we have accomplished a lot. In my opinion the most important of these accomplishment is that we re actually once again a team nowadays. A team where people talk to one another and where nobody should be a SPoF. Since this year s debconf we are six people in the admin team: Tollef Fog Heen (Mithrandir) and Faidon Liambotis (paravoid) joined the existing members: Luca Filipozzi, Stephen Gran, Martin Zobel-Helas, and myself. Growing a core team, especially one where membership comes with uid0 on all machines, is not easy and that s why I m very glad we managed to actually do this step. I also think the infrastructure and our workflows have matured well over the last four years. We now have essential monitoring as a matter of course: Nagios not only checks whether all daemons that should be running are in fact running, but it also monitors hardware health of disks, fans, etc. where possible. We are alerted of outstanding security updates that need to be installed and of changes made to our systems that weren t then explicitly acked by one of us. We have set up a centralized configuration system, puppet, for some of our configuration that is the same, or at least similar, on all our machines. Most, if not all, pieces of software, scripts and helpers that we use on debian.org infrastructure is in publicly accessible git repositories. We have good communication with other teams in Debian that need our support, like the ftp folks or the buildd people. As for figures, I don t think there s anything spectacular. As of the time of our BoF at this year s DebConf, we take care of approximately 135 systems, about 100 of them being real iron, the other virtual machines (KVM). They are hosted at over 30 different locations, tho we are trying to cut down on that number, but that s a long and difficult process. We don t really collect a lot of other figures like web hits on www.debian.org or downloads from the ftp archive. The web team might do the former and the latter is pretty much impossible due to the distributed nature of our mirrors, as you well know. Raphael: The DSA team has a policy of eating its own dog food, i.e. you re trying to rely only on what s available in Debian. How does that work out and what are the remaining gaps? Peter: Mostly Debian, the OS, just meets our needs. Sure, the update frequency is a bit high, we probably wouldn t mind a longer release cycle. But on the other hand most software is recent enough. And when it s not, that s easy to fix with backports. If they aren t on backports.debian.org already, we ll just put them there (or ask somebody else to prepare a backport for us) and so everybody else benefits from that work too. Some things we need just don t, and probably won t, exist in Debian. These are mainly proprietary hardware health checks like HP s tools for their servers, or various vendors programs to query their raid controller. HP actually makes packages for their stuff which is very nice, but other things we just put into /usr/local, or if we really need it on a number of machines, package ourselves. The push to cripple our installers and kernels by removing firmware was quite annoying, since it made installing from the official media next to impossible in some cases. Support for working around these limitations has improved with squeeze so that s probably ok now. One of the other problems is that especially on embedded platforms most of the buildd work happens on some variation of development boards, usually due to increased memory and hard disk requirements than the intended market audience. This often implies that the kernel shipped with Debian won t be usable on our own debian.org machines. This makes keeping up with security and other kernel fixes way more error prone and time intensive. We keep annoying the right people in Debian to add kernel flavors that actually boot on our machines, and things are getting better, so maybe in the future this will no longer be a problem. Raphael: If you could spend all your time on Debian, what would you work on? Peter: One of the things that I think is a bit annoying for admins that maintain machines all over the globe is mirror selection. I shouldn t have to care where my packages come from, apt-get should just fetch them from a mirror, any mirror, that is close by, fast and recent. I don t need to know which one it was. We have deployed geodns for security.debian.org a while ago, and it seems to work quite well for the coarse granularity we desired for that setup, but geodns is an ugly hack (I think it is a layer violation), it might not scale to hundreds or thousands of mirrors, and it doesn t play well with DNSSEC. What I d really like to see is Debian support apt s mirror method that I think (and I apologize if I m wronging somebody) Michael Vogt implemented recently. The basic idea is that you simply add deb mirror://mirror.debian.org/ or something like that to your sources.list, and apt goes and asks that server for a list of mirrors it should use right now. The client code exists, but I don t know how well tested it is. What is missing is the server part. One that gives clients a mirror, or list of mirrors, that are close to them, current, and carry their architecture. It s probably not a huge amount of work, but at the same time it s also not entirely trivial. If I had more time on my hands this is something that I d try to do. Hopefully somebody will pick it up. Raphael: What motivates you to continue to contribute year after year? Peter: It s fun, mostly. Sure, there are things that need to be done regularly that are boring or become so after a while, but as a sysadmin you tend to do things once or twice and then seek to automate it. DSA s users, i.e. DDs, constantly want to play with new services or approaches to make Debian better and often they need our support or help in their endeavors. So that s a constant flow of interesting challenges. Another reason is that Debian is simply where some of my friends are. Working on Debian with them is interacting with friends. I not only use Debian at debian.org. I use it at work, I use it on my own machines, on the servers of the Tor project. When I was with OFTC Debian is what we put on our machines. Being a part of Debian is one way to ensure what Debian releases is actually usable to me, professionally and with other projects. Raphael: Is there someone in Debian that you admire for their contributions? Peter: That s a hard one. There are certainly people who I respect greatly for their technical or other contributions to Debian, but I don t want to single anybody out in particular. I think we all, everyone who ever contributed to Debian with code, support or a bug report, can be very proud of what we are producing one of the best operating systems out there.

---

Thank you to Peter for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

No comment Liked this article? Click here. My blog is Flattr-enabled.

I have just bought a new motherboard, a ASRock E350M1/USB3 with and embedded AMD E350 processor. I was mostly interested in trying the embedded GPU and the UEFI firmware (UEFI is simply the name of EFI 2). According to ASRock's marketing, the main benefit of that UEFI is its graphical interface that allows to use the mouse to configure the board. I cannot agree with that, because I think it is perfectly possible to implement that with a BIOS (and I think some BIOSes actually do it), and it is pointless anyway. For me, the main benefits are a faster and cleaner boot (no idea why it should be faster, but it is).The EFI boot process EFI seems to suffer from some (many?) design errors, but I find its boot process more clean and flexible than the old BIOS one:

1. It starts with a boot manager which is similar to the temporary boot device selection menu of most recent BIOSes.
2. When you have selected the device to boot from, it looks into its EFI System partition, identified by its type GUID, as FAT file system, and executes a boot loader from a file at a specific path: /efi/boot/bootx64.efi (on amd64 platforms) by default, specially for removable devices.
3. Once the boot loader is loaded, the boot process continues as usual and is no longer specific to EFI (although the ABI used by the boot loader is specific, of course).

It is worth mentioning that the boot manager configuration, which lies in the EFI NVRAM, can be accessed and modified from an operating system, to add random entries for loading a boot loader at any path accessible from the boot manager (you will not be able to load a file from an Ext4 file system, for instance). GRUB on EFI To be able to install GRUB on EFI you need an EFI System partition, formatted in FAT and mounted on /boot/efi. GRUB will simply install itself on /boot/efi/efi/debian/grubx64.efi (on Debian) according to the EFI Specification, and use efibootmgr to create a new boot manager entry called debian and pointing to it. Tricks Well, for me, it did not work. The debian boot option failed, saying that this was not a bootable disk. I managed to get an EFI Shell, by putting Tianocore's shell into a FAT-formatted USB stick as /boot/efi/efi/boot/bootx64.efi. Using this DOS-smelling shell, I found that the directory /efi was unusable: it was listed, but any command trying to access to it failed. The reason was quite tricky, caused by a broken FAT implementation. With FAT, the file system label is stored both in the file system header and as a bogus root directory entry (yes, this sounds completely crazy). As I labelled my EFI System partition as EFI , there was such a bogus entry for this name: I think ASUS' implementation of FAT was taking it for the /efi directory. It took me hours to finally find the cause of this problem and fix it be re-formatting the partition with a new label (re-labelling may have been sufficient but I wanted to be safe).

Every year I worry that DebConf might not be as good as I hope, or not as good as previous years. Well, I've yet to be let down! The road trip down to Banja Luka was good fun, even if it took a little longer than planned. We ended up travelling down through Germany on the same Friday as much of the country finished work/school for their summer vacation, so there was a lot of traffic. Meh, we got there in the end. I made my usual mistake of planning some things to hack on during the DebConf week; by now I should know better... :-) I made a start on one small project, but then got so distracted by so many talks and side meetings with people that it's still waiting. As always, my own personal TODO list picked up huge amounts of extra stuff from those discussions. Catching up with, and socialising with, Debian friends from all over the world was also fun as always! Lots of highlights of the week for me, both for technical and social reasons:

• Multi-arch is finally with us! Great talk from Steve Langasek, and a very useful meeting afterwards to unpick what work is still needed. Quite a bit, it seems... :-)
• Useful meetings with Otavio to discuss future changes to how debian-installer is going to work, and how the debian-cd and debian-installer teams can work together better on that.
• Catching up with Martin to work out what's needed to get pancutanfinally integrated with our regular CD builds.
• Talking with Jimmy about where to go next for regular debian-live builds on our core CD build machine.
• The ARM BoF, where things have moved on from "why should I care about ARM?" to "when will you have armhf ready for going in the archive?". W00t!
• Doing our own day trip in the cars from the UK. We got to meet some very friendly and helpful local people, plus we found some incredible roads to drive on. Even if not all of them were technically roads yet... :-)
• Playing poker in a real casino for the first time, along with a bunch of friends. Nice atmosphere, and very good fun.

Banja Luka was a lovely place to visit, and a great host city for DebConf. Looking forwards to Managua now!

Dear lazyweb, In a couple of weeks I'm likely to be going cycle touring around the North East of England. This will involve first taking a train from Nottingham. While looking at train options just now, I was reminded of a little survey that I wanted to conduct. Taking bicycles on trains in the UK can be a hit-or-miss experience. The UK's train routes are operated by a variety of train operating companies who each have different policies on taking bicycles on their trains. Some require a reservation, some do not. Some do not even allow reservations at all. There's no official database of these policies. It is up to you to check all of the companies' websites individually. As far as I can tell it is impossible to make a cycle reservation online. If you don't make a reservation, chances are that you'll be OK, but there are no guarantees you won't find yourself stranded in Bolton when none of your connections will take you. The number of spaces reserved for cycles is extermely limited. Two seems to be the most common number that a train can officially carry (you may be able to stash it in the vestibule). Even given this, it is usual to find that the cycle spaces are taken up by prams, luggage and other crap when you actually board the train, so the effective number may actually be 0. However, it is thankfully not uncommon to find staff and other passengers who are more than willing to go out of their way to help you if they see you are trying to take a bicycle onto a train. Anyway, I'm interested in how this situation is in other non-UK countries. Better or worse? A lot of countries certainly have a more cycle-friendly culture I'd like to know how this kind of thing works there so that I can write a Strongly Worded Letter about how we Should Be Doing Better. (Image by onohiroki, CC-by-nd.)

These days I briefly discussed with a fellow Debian developer about how to maintain upstream patches in Debian packages with Git, what brought me to rethink a little about my current practices. What I usually do is pretty much like point 4 in Raphael's post "4 tips to maintain a 3.0 (quilt) Debian source package in a VCS": I make commits in the Debian packaging branch, or in a separate branch that is merged into the Debian packaging branch. Then I add the single-debian-patch option to debian/source/options so that a single Debian patch is generated, and include a patch header that points people interested in the individual changes to the public Git repository where they were originally done. My reasoning for doing so was the following: most upstream developers will hardly care enough to come check the patches applied against their source in Debian, so it's not so important to have a clean source package with separated and explained patches. But then there is the people who will actually care about the patches: other distribution developers. Not imposing a specific VCS on them to review the patches applied in Debian is a nice thing to do. Then I wrote a script called git-export-debian-patches (download, manpage), which was partly inspired by David Bremner's script. It exports all commits in the Debian packaging branch that do not touch files under debian/ and were not applied upstream to debian/patches. The script also creates an appropriate debian/patches/series files. The script is even smart enough to detect patches that were later reverted in the Debian branch and exclude them (and the commit that reverted them) from the patch list. The advantage I see over gbp-pq is that I don't need to rebase (and thus lose history) to have a clean set of patches. The advantage over the gitpkg quilt-patches-deb-export-hook hook is that I don't need to explicitly say which ranges I want: every change that is merged in master, was not applied upstream and was not reverted gets listed as a patch. To be honest I don't have any experience with either gbp-pq or gitpkg and these advantages were based on what I read, so please leave a (nice ;-)) comment if I said something stupid. I am looking forward to receive feedback about the tool, specially about potential corner cases in which it would break. For now I have tested it in a package with simple changes agains upstream source, and it seems fine.

OK, this one is meant for Planet Debian...

One of many, many changes in the latest lintian is vendor profiles and, thanks to a heads-up by Niels Thykier, Emdebian will have working profile support in the next upload of emdebian-grip. (The only reason it's not already in Debian is my own fault for not uploading when I thought I had the time to upload.)

$ lintian --profile emdebian-grip drivel_3.0.2-1em1_amd64.deb 

$ lintian --profile debian drivel_3.0.2-1em1_amd64.deb 

E: drivel: debian-changelog-file-missing

E: drivel: copyright-file-compressed

W: drivel: copyright-without-copyright-notice

E: drivel: description-contains-invalid-control-statement

W: drivel: binary-without-manpage usr/bin/drivel

So the em1 version implements Emdebian Policy for Emdebian Grip. Clean for Emdebian Grip, just as the Debian package is clean prior to the changes.

I expect this to dramatically improve the processing of Emdebian packages, both for Grip and for the cross-built flavour, Crush, once that actually starts up.

Thanks to the lintian team for this support. Now if there was some way of backporting this version of lintian to Squeeze, I could also use this at work to suppress really annoying lintian warnings about non-standard suite names. (The whole point of using a non-standard suite name is to keep our stuff separate from the normal Debian/Emdebian stuff for licensing reasons etc.) Update: of course, I didn't check the PTS for lintian before writing that, so hence didn't notice that the backport already exists! Thanks again to Niels for the prompt. I've now got another package to create for work. Update2: Thanks to Joerg Jaspert for the tip that all lintian versions get backported directly from unstable as an exception on ~bpo. The work package is ready, so this is going to make things a lot easier when building stuff on stable.

In other news, the same version of emdebian-grip will include support for integrating Emdebian Grip into Debian itself. This too will use vendor-specific support, this time an internal vendor name which just needs to work on the "buildd".

(It's not quite a buildd, Emdebian Grip doesn't build anything, it's all post-processing. It's just that the processing of Debian packages for Emdebian Grip will look a bit like having a second buildd working on the packages uploaded by the existing buildd. The process itself is still developing...)

Multistrap runtime message translations need updating but the manpages aren't being done this time around. More changes are likely before those get done.Current runtime translation status:

language        translated     fuzzy     untranslated

-----------------------------------------------------

  da               103           4             4

  fr                54          17            40

  pt                54          17            40

  vi                54          17            40

http://lists.debian.org/debian-i18n/2011/08/msg00105.html
Check with the relevant debian-i18n team for your language, this is just a prompter in case people miss the mailing list post.Update: Hmm, that went to the wrong blog. It got tagged i18n but was meant for the i18n blog. Ah well, doesn't hurt for Planet Debian to see what is going on elsewhere....

The Border Gateway Protocol is what network operators use to exchange the various routes forming the Internet. It's a pretty critical piece of infrastructure. Most providers are probably using proprietary solutions using dedicated hardware, but free software on comodity hardware can also be an option for some networks. In Debian, we already have Quagga and Bird that both implement BGP on top of Linux for quite a while. Back in time, FOSDEM feb. 2006, Ging is being demonstrated. Immediately my mind races with something like: waow, we could port OpenBGPD on that! OpenBGPD is another BGP routing daemon. This new effort was started in 2003 by Henning Brauer for OpenBSD. To sum up its goal, it was meant to be secure, reliable, lean, easy to configure, fast and memory efficient. This is quite the opposite of Quagga codebase, which is inherited from an older project, Zebra which was mostly documented in Japanese. Another issue I have seen hapenning with Quagga on Linux is that its blocking implementation and the time that the kernel needs to insert/remove routes made the daemon unresponsive enough to created cascading failures. The IPv4 Internet is nowdays made of more than 350 000 prefixes, so when a network peer fails, the router needs to process those changes while continuing to answer other peers. Otherwise, they will kill the session meaning yet another 350 000 prefixes to process on both sides. And again when the session is restored. Leading to more downtime, flaps, and failures. OpenBGPD was ported to FreeBSD. The port is maintained by Hiroki Sato. Starting with this port and using libbsd made the creation of a package for Debian GNU/kFreeBSD pretty doable while at DebCamp11. It probably still has some pretty rough edges and the manpage needs to be reviewed (and some unsupported features trimmed, maybe), but the package should be usable. So OpenBGPD is now in Debian, and it could be one of the killer features of a port that some describe as a toy OS. Unfortunately I did not yet have access to proper network equipment to test the package on real production scenario (with 2 or 3 full BGP views). If you have a lab (or feel adventurous) and have access to such gears, please have a try!

Welcome Planet Debian viewers If things are working right, Planet Debian will now be getting my blog posts tagged Debian. The main reason I had not done this before is not much on my blog relates to Debian, and it tends to be much less technical than most of the blogs there. Viewers from there might want to check out my recent post on debconf11 and may check out the rest of my blog archives, where I'm trying to get rid of my house and move in to a motorhome.

About 24 hours ago I rebooted the system that runs the secondary DNS for my zone and a few other zones. I d upgraded a few things and the system had been running for almost 200 days without a reboot so it was time for it. Unfortunately it didn t come back up. Even more unfortunately the other DNS server for my zone is ns.sws.net.au which is also the only other server for the sws.net.au zone. Normally this will work because the servers for the net.au zone have a glue record containing the server IP address. So when asked for the NS records for the sws.net.au domain the reply will include the IP address of ns.sws.net.au. The unfortunate part was that the IP address was the old IP address from before the sws.net.au servers changed to a new IP address range, I wonder whether this was due to the recovery process after the Distribute IT hack [1], as forgetting to change a glue record is not something that I or the other guy who runs that network would forget. But it is possible that we both stuffed up. The DNS secondary was an IBM P3-1GHz desktop system with two IDE disks in a RAID-1 array. It s been quite reliable, it s been running in the same hardware configuration for about four years now with only one disk replacement. It turned out that the cooling fan in the front of the case had seized up due to a lot of dirt and the BIOS wouldn t let the system boot in that state. Also one of the disks was reporting serious SMART problems and needed to be replaced poor cooling tends to cause disk errors. It seems that Compaq systems are good at informing the user of SMART problems, two different Compaq desktop systems (one from before the HP buyout and one from after) made very forceful recommendations that I replace the disk, it s a pity that the BIOS doesn t allow a normal boot process after the warning as following the recommendation to backup the data is difficult when the system won t boot. I have a temporary server running now, but my plan is to install a P3-866 system and use a 5400rpm disk to replace the 7200rpm that s currently in the second position in the RAID array. I ve done some tests on power use and an old P3 system uses a lot less than most new systems [2]. Power use directly maps to heat dissipation and a full size desktop system with big fans that dissipates less than 50W is more likely to survive a poorly cooled room in summer. Laptops dissipate less heat but as their vents are smaller (thus less effective at the best of times and more likely to get blocked) this doesn t provide a great benefit. Also my past experience of laptops as servers is that they don t want to boot up when the lid is closed and getting RAID-1 and multiple ethernet ports on a laptop is difficult. Finally I am going to create a third DNS server for the sws.net.au domain. While it is more pain to run extra servers, for some zones it s just worth it.

• [1] http://www.zdnet.com.au/distributeit-data-unrecoverable-post-hack-339317149.htm
• [2] http://doc.coker.com.au/environment/computer-power-use/

As our GSoC students are approaching quite well, we've already numerous features included in current master, which are worth mentioning. You can try all bellow mentioned features on our demo server. Grid editing provides really simple way of editing data, by simply clicking on it while browsing table. Just try it on some table. Performance suggestions are part of completely reworked server status and can give you valuable information how to tune your server. You can see it on demo server. Another nice addition to server status page is live monitor, which charts basic server information in real time. Again, try it on demo server (you need to press start for now, this will probably go away soon). Other big, thought not that fancy change is complete support for routines. This includes stored procedures/functions, triggers and events. For example see routines in sakilla database. Of course there is much more to mention, like improving of AJAX in lot of places, GIS support. On the other side, we've also recently decided to remove support for Microsoft Excel import and export as used PHPExcel library seems to suffer some licensing issues, which prevented phpMyAdmin from being included in Fedora. You can still use CSV or OpenDocument formats though.

Filed under: English Phpmyadmin 0 comments Flattr this!