Hello,

This is the fourth report of my GSoC project for Debian. It has been a long time
since I wrote third report. So past four weeks a lot of things are implemented.

Firstly, I've changed tool's main script's name to py3build (removed .py suffix)
and make it executable so after you cloned my git repository you can use tool
with "./py3build".

I fixed a bug which occurs when the tool is invoked from somewhere except
its directory. If you invoke it like "./path/to/py3build/py3build" then the tool give
errors (f.e. could not find plugins etc.). Now this is fixed.

When I'm designing the API, I think if there is given build system (with
--system parameter) there is no need to invoke detect method but when
I'm writing autotools plugin, I realized that I need to learn which files are
exists and according to that information I'll invoke right thing (f.e. autoconf
or directly configure). So even if there is a given build system with --system
parameter its detect method invokes.

When I'm implementing plugins to tool, I need clean args (f.e. parameters
which are using with make clean) and install args (f.e. parameters with
make install) and I added it.

I added --dest-dir parameter this is passing to distutils plugin's root parameter.
In distutils plugin --install-dir parameter is passing as --install-platlib,
--install-purelib and --install-scripts so that maintainer can override it
with py3build's --install-args parameter.

Now, I implemented distutils, autotools, cmake plugins and working on
scons and packaging plugins. During the writing period of plugins I'm
testing them. After I finished these plugins I'm planning to write a patch
to dh and cdbs. After that I will write the tool and plugins for Python2.X.

OK, I'll also blog about other DebConf activities, but I was just working on my sports software of choice and just summarized my running activities here in Banja Luka (it's 22h45 last day of Debconf). I finally broke my record of last year (where I ran 112 kilometers) by running 115 kilometers. However, that was in 12 days while I was only 10 full days in New-York. So that record is only partially broken. These 115km were run in 12h09', which leads to a pace of 9.5km/h or 6'18"/km or 10'13"/mile (hello, Dirk). Fairly slow, indeed. However, this or that may better explain..:-). The cumulated positive slope for these 115km has been around 2000 meters! Specifically, I had great fun today when No l and I decided to make a final round in the hills close to the Banj Brdo. For that 12km run, we shared a taxi to the start of the Banj Brdo road, to avoid 2km of boring run in Banja Luka streets, then made our way to nearly each and every path we could find. Of course, that should soon lead to a very well mapped area on Openstreetmap. As usual, when a DebConf ends, the area is much better maped than it was before it. Here, the city was too well mapped by local OSMers, so we had to go to the mountains..:-). On this sunny Saturday, doing what I call a real trail run, with mud, slopes, scratching vegetation, rocks.....was an incredibly great pleasure. Sharing this with No l was also great. Hopefully, running in Managua next year will be as much fun (but maybe really tricky because of the heat). See you there!

OK, I'll also blog about other DebConf activities, but I was just working on my sports software of choice and just summarized my running activities here in Banja Luka (it's 22h45 last day of Debconf). I finally broke my record of last year (where I ran 112 kilometers) by running 115 kilometers. However, that was in 12 days while I was only 10 full days in New-York. So that record is only partially broken. These 115km were run in 12h09', which leads to a pace of 9.5km/h or 6'18"/km or 10'13"/mile (hello, Dirk). Fairly slow, indeed. However, this or that may better explain..:-). The cumulated positive slope for these 115km has been around 2000 meters! Specifically, I had great fun today when No l and I decided to make a final round in the hills close to the . For that 12km run, we shared a taxi to the start of the Banj Brdo road, to avoid 2km of boring run in Banja Luka streets, then made our way to nearly each and every path we could find. Of course, that should soon lead to a very well mapped area on Openstreetmap. As usual, when a DebConf ends, the area is much better maped than it was before it. Here, the city was too well mapped by local OSMers, so we had to go to the mountains..:-). On this sunny Saturday, doing what I call a real trail run, with mud, slopes, scratching vegetation, rocks.....was an incredibly great pleasure. Sharing this with No l was also great. Hopefully, running in Managua next year will be as much fun (but maybe really tricky because of the heat). See you there!

In the past week, I ran a lot of benchmarks using a Spirent Avalanche which is an appliance providing performance testing of network related products (a load balancer, a router, a web server, ). The reporting module does not provide a lot of flexibility and the plots are not the most beautiful ones. Fortunately, the results are also exported as CSV. matplotlib is a python plotting library which produces great figures without making things hard when they should be easy. It is a good replacement for GNU Plot and you don t need a lot of Python knowledge to use it. The documentation includes a fine user s guide that should get you started in less than 20 minutes.

Quick introduction You can use matplotlib from IPython to experiment:

$ ipython -pylab
Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) 
Type "copyright", "credits" or "license" for more information.
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
> In [1]: plot([1,1,4,5,10,11])

When you are ready, you can build a simple Python script:

#!/usr/bin/env python
from matplotlib.pylab import *
plot([1,1,4,5,10,11])
savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.

from matplotlib.pylab import *
import sys, os
import gzip
skip = 0
for line in gzip.open(sys.argv[1]):
    if line.startswith("Seconds Elapsed,"):
        break
    skip = skip + 1
ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)

Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].

General structure I need 4 plots:

• successful, unsuccesful and attempted transactions per second,
• minimum, average and maximum response time per page,
• Avalanche CPU usage,
• incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.

# Create the figure (A4 format)
figure(num=None, figsize=(8.27, 11.69), dpi=100)
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
# [...]
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
# [...]
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
# [...]
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
# [...]
# Save to PDF
savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.

# Plot 4: Bandwidth
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000.,
     'b-', label='Incoming traffic')
plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000.,
     'r-', label='Outgoing traffic')
grid(True, which="both", linestyle="dotted")
ylabel("Mbps", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.

CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.

# CPU
max = np.max(ava['average_cpu_utilization'])
order = 10**np.floor(np.log10(max))
max = np.ceil(max/order)*order
cpu = (max - ava['average_cpu_utilization'])*100/max
# Plot 3: CPU
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
plot(ava['seconds_elapsed'],
     cpu,
     'r-', label="Avalanche CPU")
grid(True, which="both", linestyle="dotted")
ylabel("Avalanche CPU%", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:

# Plot 2: response time
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'],
     'b-', label="Minimum response time")
plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'],
     'r-', label="Maximum response time")
plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'],
     'g-', linewidth=2, label="Average response time")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8))
grid(True, which="major", linestyle="dotted")
yscale("log")
ylabel("Response time (msec)", fontsize=9)
xticks(fontsize=9)
yticks(fontsize=9)

This is also the first graphic with a legend.

Transactions per second The most important plot is the number of transactions per second.

# Plot 1: TPS
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'],
     '-', color='0.7', label="Desired Load")
plot(ava['seconds_elapsed'], ava['successful_transactionssecond'],
     'g:', label="Successful")
plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']),
     'g-', linewidth=2)
plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'],
     'b-', label="Attempted")
plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'],
     'k-', label="Aborted")
plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1],
     'r-', label="Unsuccessful")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10))
grid(True, which="both", linestyle="dotted")
ylabel("Transactions/s")

The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:

import numpy as np
def smooth(x, win=4):
    s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]]
    w = np.ones(win, 'd')
    y = np.convolve(w/w.sum(),s,mode='valid')
    return y[(win-1)/2:-(win-1)/2]

np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:

# Noticeable points
count = 0
def highlight(index, reason):
    global count
    if index and index > 0:
        x,y = (ava['seconds_elapsed'][index],
               smooth(ava['successful_transactionssecond'])[index])
        plot([x], [y], 'ko')
        annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y),
                 xytext=(20, -(count+4.7)*22), textcoords='axes points',
                 arrowprops=dict(arrowstyle="-",
                 connectionstyle="angle,angleA=0,angleB=80,rad=10"),
                 horizontalalignment='left',
                 verticalalignment='bottom',
                 fontsize=8)
        count = count + 1
highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS")
highlight(np.argmax(cpu > 99), "CPU>99%")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")

np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

Recently, I was working with Daf and Rob on a little offline wiki project -- more on that soon -- and we realized that we needed to parse some dates in ISO 8601 format. One of us wondered out loud if there was a Python module that could help us. I offered to take a look. Turns out, less than two months before, someone had uploaded just such a module into Debian. The maintainer? Me.

It is half-time for me at DebCamp/DebConf11 in Banja Luka, Bosnia-Herzegovina, and that is a good time to write about my work so far. I started with some finger exercises, such as the HTML copying for gnome-terminal (as blogged previously) and cosmetic changes to the buildd.debian.org/status website. After that, I began my main project: A SAT-based solver for the testing transition problem. In Debian, new software first enters a repository called unstable, where it is tested. Once some requirements are fulfilled (package has been in unstable for usually at least 10 days and has no new release critical bugs), it is entitled to enter the repository called testing, which will eventually form a new stable Debian release. But some packages need to migrate together, usually because either requires the new version of the other. Also, no packages in testing ought to become uninstallable by some seemingly unrelated change. The software that decides these things is called britney. It is sufficiently good in making sure nothing bad happens, but not smart enough to figure out what to do in case more than two packages need to migrate simultaneously. My goal is to improve this. Now, the the various requirements can all be expressed as predicative formulas, and a lot of research has been going into writing good solvers for such problems, called SAT-solvers. Hence my plan was to only specify what we expect from a testing migration, but leave the search for a solution to such a general purpose and highly optimized program. My initial progress was good and I had some result after two days of hacking, and the approach is promising. Because the data sets are pretty large (1.5GB of input data, the final SAT problem has 1.8 million clauses with 250.000 variables), so I learned quite a bit about profiling and optimizing Haskell programs, and that parsec is slower than working with BS.lines and BS.split to parse simple input. I also used the FFI to use dpkg code directly to compare Debian version numbers. At some point I noticed that I actually want to solve PMAX-SAT problems: Given two set of clauses, hard clauses and soft clauses, find a variable assignment that fulfills the hard clauses and as many of the soft clauses as possible. Unfortunately, there are no fast Free PMAX-SAT solvers around. The ones that I found and that were fast enough that I could use them, msuncore by Joao Marques-Silva at the University College Dublin and MiniMaxSat by Federico Heras at the Universitat Polit cnica de Catalunya, are only available as statically linked binaries. It is a shame that this is acceptable by the academic community; imagine mathematicians would stop including the proofs in their papers, and only share their theorems. So if you happen to have written a PMAX-SAT solver and can solve this instance (in weighted DIMACS format) in less than five minutes, and want to brag that the Debian project is using your code, then please release it under a Free license (e.g. GPL or BSD) and tell me about it! The code of my SAT-based solver is, of course, available, though slightly unpolished. Parts of it (notably the interface to picosat and the PMAX-SAT solvers) might be of general interest and I plan to put them on Hackage as a separate license. Update: I have contacted the authors of the SAT solvers mentioned above, and they have reconfirmed that they have no intention of releasing the source. Now I put my hope in maxsatz2009 by Chumin LI, which is GPL but was not able to cope with my large instance directly; I think I need to change the memory management.

Flattr this post

If a single man can display so much hatred
think only of how much love we all can display together.

Stine Renate H heim

I wrote a Norwegian post explaining my experience at Ut ya. I had taken this blog for dead, and had entirely forgotten that it was syndicated on Planet Debian. I don t want to let Google Translate make this disaster any worse than it is the translation of bullets into balls being particularly bad so the international attention the massacre has garnered in consideration, I am writing an English translation of my experiences. I feel somehow duty-bound to make people aware of what happened, but I don t want to get into anything else but a sober description of the events and some very brief reflections. There are many details I have chosen to omit. Others have written their experiences of the events at Ut ya. I wanted to write mine down as well, and get it out there . Partly, I want to write this down because I m unsure if I will remember all the details at a later point in time, although I think I d prefer it if I couldn t. I m also writing this because people are asking about my experiences and it s much better to have an URL to give them, lest I have to keep going through the same spiel over and over again. Our former Prime Minister and current labour movement demigod Gro Harlem Brundtland had recently left the island. I had been the cameraman for a video interview of her talking about Ut ya, and I was in the media group room encoding the video into a file suitable for YouTube, when someone else in the room startled and said that Twitter was full of messages about a loud explosion in Oslo. As the newspapers brought us information about the extent of the damages, a consensus arose that an informational meeting was in order. As soon as the current round of talks finished, we were gathered into the main hall. The meeting was duly held, and after the statement was made that a TV feed would be made available, I took it upon myself as the local alpha geek to make it happen. Of course, the situation caused both the wireless network and the GPRS networks to become totally unusable. As I was waiting for someone to set up a password, I took the opportunity to face the consequences of having eaten two bits of a microwavable dish called Hold-It the local equivalent of a Hot Pocket and went to the toilet. As I was in there, I first heard agitated shouting, then screams, then gunshots coming from just outside the toilets. More than anything else, it sounded like a toy gun. I was convinced that someone was making a joke in incredibly bad taste and I stormed out of the booth with the intent of halting it. As I tore the door open, I saw two of my comrades hiding in a recessed corner. Their facial expressions left absolutely no doubt that this was no toy. They signalled for me to get back in the booth. I closed the door, did a mental double-take in utter, complete confusion, and opened it again. They were still signalling. Had they not stood there, I would have run straight into the gunman; they saved my life. I looked out into the hallway, and I made eye contact with a young boy lying in a pool of blood. He was motioning for me to help him. I heard more gunshots from inside the building and retreated back inside. As I was trying to think through my next move, I realized that the decidedly insubstantial wood-fiber door would not resist any kind of bullets. I made my way out into the hallway, with the intent of escaping outside. At that point, I was of course not aware that there was an intention to kill as many as possible, so I thought that the open spaces outside would be a place of relative safety. Of course, this proved to be wrong and my life was probably saved a second time by one of the caf volunteers taking me into a hard-to-spot employee s bathroom. We sat there for ninety minutes. Always ready to make a run for it, ready for just about anything. A peculiar group dynamic arose with these two people with whom I had barely previously spoken. We came to share a strange sense of common destiny and gallows humour. One of them had seen the shooter and described the police uniform. I perceived it to be realistic that we were the only ones aware of the wounded outside the toilet. I tried to reach the emergency services, but all their lines were busy; the terror attack in Oslo had probably clogged their lines. I finally got through to the fire services, who could inform me that the police did know about the situation and were on their way. This was to take 90 minutes and by the time we evacuated, the young boy outside my door had perished. The despair I first saw in his eyes as I passed him, fleeing from one room to the other and the empty, blank stare as we left, are burned into me and they are images I will never in my life forget. Finally, the real police arrived. We walked out. I chose the path through the minor conference hall something I now regret. The sight was simply beyond my capacity to describe fully, and so terrifying that I barely remember the sight only the terror it struck in me. There were several people bunched up in a corner, a big amorphous heap of bodies. Some were conscious and yelled at me not to do anything that could startle the police, others lay still. Their bodies were all covered in blood, and a thick pool of blood extended at least a half-metre in all directions around them. The policeman across the hall was screaming orders at me, but he was screaming so loudly that I couldn t make out his words at first. We were first moved into the camp newspaper s offices. There were about eight of us there, I think, in addition to one girl who lay wounded. Towards the end she was drifting in and out of consciousness. We covered her with sweaters to keep her warm and one of us tried to at least temper her bleeding. The bullet had missed her heart, but by the entry wound it was clear that it was not by far. I do not know who this girl was or how she is now. I sat behind and never saw her face. The wounded were evacuated first. I don t remember how long we remained; I had lost all concept of time. In spite of protests from the group who knew him, one kid was put in handcuffs. At the time I didn t understand why, and the policeman seemed to say something almost to the effect that there was no reason for it at all. I didn t see when they undid his cuffs, but I remember thinking that this treatment made a terrible experience even worse for him. I tried my best to comfort him but knew it would be little help. Later, when things stabilised a little, we were told that he was handcuffed because he had come from an unsecured area. The police was extremely good at carefully explaining what was happening and why; this was a big help and I am grateful for it. Eventually we were moved out into the main corridor of the building, where we joined up into a group of about fifty. Only when I saw the two people who saved my life did any emotion other than mild confusion arise. I broke down shivering in tears in one of their arms. After a few seconds, I came back to my senses and realised that this was not the time. I quickly gathered myself, got the shaking under control, and sat down. We were given some chocolate and soda from the kiosk. I remember making an offhand remark that an inability to find joy in free candy was a sure sign of a bad situation. We all laughed out loud. Gallows humour is a coping mechanism, but in retrospect one almost feels guilty for it. We were shown out in a single file with hands above our heads. I remember an intense concern that someone would slip in the steep, muddy slope and create a misunderstanding. Outside, there were more bodies. Some under improvised covers a tarpaulin from the waffle stand, the deflated bouncy castle but some simply lay there. Everyone I met displayed a courage, a mental discipline and unity of purpose far beyond anything one would ever wish to expect from people this young. Everyone conducted themselves with an attitude that could almost be described as stiff upper lip . Safely across the fjord we were offered blankets. I was asked if I was aware of any injuries, and asked to lift my shirt and show my abdominal region. We were shown into the bus which took us to the hotel used by the survivors and their family. I simply cannot describe in any words the relief I felt when I was able to embrace my living comrades. It was completely unlike anything I had ever felt before in my life. The euphoric feeling was tempered only by the realisation that there would be many I could never see again, comrades whom I had taken great pride in calling my friends, with futures in the service of all mankind, futures I had previously found such great joy in pondering and guessing about. The feeling which continues to upset me the most, is the feeling that so many of my comrades left behind grieving families and friends. Torn away senselessly. I do not know how much more than this relatively sober account of the events on Ut ya I can muster. I would, however, like to offer some reflections. First of all, from the bottom of my heart, I want to thank the police who saved the lives of so many still on the island, the holidaymakers who took aboard swimmers into their boats and the rescue services staffed primarily by volunteers who have spared no effort in trying to soften the blow as much as they can. The opportunity to spend time with those comrades who underwent the same experience as myself has also been an immeasurable aid. I was also so relieved to find my very closest friend among the survivors, which has also been an indescribable help. If I can name a single positive in this tragedy: Had he arrived with his automatic weapon fifteen or twenty minutes prior, he would have arrived during the informational meeting, at a time when the major hall was absolutely jam-packed the death toll would be many times what it ended up being. I am agonisingly aware of the meager comfort this provides to those who have been bereft of their closest, but I do find some solace in this. We cannot sweep under a rug that this was without question! a political attack on the labour movement. But it is thankfully also an attack which has been perceived by everyone as an attack on the Norwegian society, and on a symbol of the wide recruitment to the participatory democracy which lies at our very national soul. I cannot thank the Norwegian people, and indeed the people in other nations who have offered their condolences, enough for their shows of support and shared grief. It really has been a tremendous help to me knowing that so many people feel with us. I also want to thank from the bottom of my heart the rock-steadiness of everyone in both the national and local wings of the Labour Party and Labour Youth in supporting us, and the political milieu in general for their resolute steadfastness saving me from losing yet more that I cherish; our freedoms in a participatory democracy. Our Party has lost many of its very brightest youngsters. Personally I feel an angry spite, a deep restless urge to get the wheels of society going again. I want to show his kind that we will not be broken. We re stronger than that. I will not be frightened into silence and passivity. I want to remember the dead, and honor them by carrying on our common work. I want to end this with a request to everyone who reads this, echoing a statement I read by one of my good friends and comrades: Please, don t let me see any messages of hatred, wishes for the death penalty, anything like that. If anyone should be of the belief that anything will improve by murdering this sad little person, they would be profoundly wrong. All attention now should be plowed into caring for those victims and their relatives who did not share my luck, and not giving an audience to a perpetrator who wants one. Tore Sinding Bekkedal