The last versions of OpenSSH brought an interesting feature: sharing multiple sessions over a single connection. When enabled, this is how it works:

1. The first time you open an SSH connection to a server (including anything that works on top of SSH, like SCP, SFTP, rsync or Git), it opens a network connection as usual. It also opens a local Unix socket and listens to it for later use.
2. The next times you open an SSH connection to that server while the first one is still open, instead of opening a new network connection, it connects to that local Unix socket and lets the first SSH client carry its new session.

Advantages

• By reusing an existing connection, it saves file descriptors, thus power, baby seals and kittens, and most of all it is faster than opening new ones.
• It does not require a new authentication, so it saves you the time to type your password or you key passphrase, and the time for the challenge-response to take place.
• Since you do not have to authenticate again, you can take advantage of you shell's power features to auto-complete remote paths for SCP, SFTP or rsync over SSH. That is, if you use a powerful shell, of course.

Disadvantages Just like SSH agents, there is one major side-effect to take into account: new remote sessions can be opened without a password. Never use such a feature on a public computer if you are used to temporarily leave running sessions: here, locking them would not protect other people to get a shell access to your remote account! Configuration In ~/.ssh/config or /etc/ssh/ssh_config:

ControlPath ~/.ssh/control-%r@%h:%p
ControlMaster auto
ControlPersist 1

ControlPath

The name template for the session sharing Unix sockets. See the ssh_config(5) manpage for the syntax.

ControlMaster

Lets the SSH client use the session sharing socket if it exists or create it and listen otherwise.

ControlPersist

Usually, the first client to a given server is in charge of the connection sharing. It will then stay in foreground even if you terminate the session it is used for, until every other clients using its socket has finished, and forcefully terminating it will have the effect of terminating these clients! With that option, the master connection is put in background and will only terminate when no client needs it any more (one second after that, in fact), and you will be able to terminate the first session without kicking the other ones.

Try it! Apply this configuration and connect several times to a single host, to see the difference

Sociological Images has an interesting article about the attempts to apply the word Camping to OWS and framing the issues [1]. Lester Macgurdy wrote an insightful article about the snake , a new technique for OWS protesters to beat riot police [2]. Ron Barassi suggests that Australia Day be celebrated on the 27th of May to commemorate the day in 1967 when the Australian constitution was amended to not be racist [3]. The current Australia Day is often referred to as Invasion Day . IMHO Ron deserves another Best and Fairest award. Stefon Harris gave an entertaining TED talk about improv Jazz music titled There Are No Mistakes on the Bandstand [4]. It seems that his concepts can apply to some extent to many collaborative projects. John Robb wrote an interesting article about the future of drone (UAV) warfare [5]. He suggests that having one person control each drone is a temporary thing and that the future is to have a cloud of cheap autonomous drones taking strategic control from one person. His comparison of Starcraft players to future drone fighters is interesting. The OWS movement is branching out into other related areas, OccupyYourHomes.org is one of the latest ones [6]. When banks try to forclose on homes without good cause the OWS people are protesting. Cory Doctorow wrote an important article for The Guardian about corporations using the Youtube ContentID system to pirate works that other people have uploaded [7]. Matt Taibbi s description of Goldman Sachs as a great vampire squid wrapped around the face of humanity, relentlessly jamming its blood funnel into anything that smells like money will never die [8]. It has spawned many other creative descriptions of the evil and greed of Goldman Sachs and even Lloyd Blankfein of Goldman Sachs describes his company as having burned down the Reichstag, shot the Archduke Ferdinand and fired on Fort Sumter he was trying to use satire, but I don t think that Goldman Sachs people would act differently to Fritz Thyssen. Keith Packard wrote an interesting article about the Calypso CalDAV system which he uses with Android [9]. He makes lots of good points about how to improve calendaring and contacts on Android, unfortunately I lack time to fiddle with such things at the moment so I ll stick with Google in spite of the risks. Asheesh Laroia wrote a great article about the problems with short (32bit) GPG keys [10]. It seems that creating keys with matching ID numbers isn t particularly difficult and that GPG doesn t handle them as well as we would like giving the possibility of at best annoying DoS attacks and at worse security problems due to using the wrong key. Sociological Images has an interesting article about when game show audiences are trustworthy [11]. It seems that French people don t want an undeserving person to win so they will intentionally advocate the wrong answer if the contestant should know it. Paul Wayper gave a great lecture titled SE Linux for Everyone [12]. He covers the basics of SE Linux in a user-friendly way and explains some simple solutions to common problems which don t involve compromising system security. Paul Tassi wrote an insightful article for Forbes about piracy [13]. His conclusion is that the media companies should make it cheaper and easier to be a customer and not spend insane amounts of money on low quality products. The Reid Report has an interesting article about Ron Paul s racism [14]. Ron Paul is generally well regarded outside the US because he wants the US government to stop meddling in the affairs of other countries, but while he s less bad than other US politicians in terms of foreign policy that doesn t make him a good person. Anonymous hacked some mailboxes belonging to a neo-Nazi group and found links to Ron Paul [15]. I ve always been suspicious of the way Ron Paul wanted to avoid anti-racism legislation on supposed Libertarian principles. The Reid Report has an interesting summary of Ron Paul news plus some criticism of Glenn Greenwald and others who associate with him [16].

• [1] http://thesocietypages.org/socimages/2011/12/06/on-camping-and-the-framing-of-occupy-movement-strategies/
• [2] http://www.portlandoccupier.org/2011/12/15/occupy-portland-outsmarts-police-creating-blueprint-for-other-occupations/
• [3] http://tinyurl.com/6mr95hu
• [4] http://www.ted.com/talks/stefon_harris_there_are_no_mistakes_on_the_bandstand.html
• [5] http://globalguerrillas.typepad.com/globalguerrillas/2011/12/drone-bonjwas.html
• [6] http://occupyourhomes.org/
• [7] http://www.guardian.co.uk/technology/2011/dec/12/pirates-of-youtube-cory-doctorow
• [8] http://dealbook.nytimes.com/2011/12/13/the-long-life-of-the-vampire-squid-metaphor/
• [9] http://keithp.com/blogs/calypso/
• [10] http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html
• [11] http://thesocietypages.org/socimages/2011/12/26/are-game-show-audiences-trustworthy/
• [12] http://www.youtube.com/watch?v=ZThVfm3JXdM
• [13] http://www.forbes.com/sites/insertcoin/2012/02/03/you-will-never-kill-piracy-and-piracy-will-never-kill-you/
• [14] http://blog.reidreport.com/2011/12/the-real-bottom-line-on-the-paul-newsletters-race-baiting-for-power-and-profit/
• [15] http://www.care2.com/causes/anonymous-hacks-neo-nazis-finds-ron-paul.html
• [16] http://blog.reidreport.com/2011/12/should-glenn-greenwald-have-to-own-the-ron-paul-blue-plate-special/

Related posts:

1. Links February 2011 Australia s Department of Finance has mandated that the MS-Office document...
2. Links January 2012 Cops in Tennessee routinely steal cash from citizens [1]. They...
3. Links February 2009 Michael Anissimov writes about the theft of computers from the...

For the past 5 days, I have worked on replacing a part of Britney (our beloved testing migration tool). The part I am trying replacing is her installability tester , which is probably one of the most used parts of Britney. I know premature optimization is the root of all evil, but I felt compelled to be a bit evil this time. The reason is that a na ve algorithm could easily spend decades on computing the result of our most complete tests (each consists of roughly 2 * 35 000 packages). Within a couple of days, I felt I had a reasonable candidate to use on the large tests. On the large tests, it ran with a noticeable speed regression and it produced one different result. The result difference was about 20-40 source packages and their binaries. You may have noticed I used difference and not regression . In the given test, our current implementation at some point gives up with an AIEEE error. So I decided to turn my attention to the speed regression first. As any good programmer, I decided to use a profiler (cProfile) to determine my bottle-necks. This turned out to be a good idea, as the bottleneck was not quite where I thought it was. I wish I could say I just patched it out the issue, but unfortunately not. I played around with various changes such as using implied for loops rather than explicit ones etc. Some times I would manage to cut the runtime in half only to be unable to reproduce it later. Yesterday I finally realized what caused this. I was working in 3 different chroots, two 64-bit sid chroots[1] and a 32-bit stable chroot. Turns out that my replacement is only slow in the stable chroot. As soon as I moved to a sid chroot, the runtime was more or less cut in half. I have not bothered to check if it is the 32 vs 64 bit, the stable vs sid part or maybe the python profile in stable is just slow[2]. I simply moved to my sid chroots for now and continued working. With that taken care of, I figured I would remind myself of what I was up against. On the largest test I have available, I profiled my implementation to around 4:30 5:00 minutes. Regardless of my changes, I always got times in that range. I did manage to patch a bug in my code that reduced by only diff to 10 source packages (plus their binaries) at the price of 30 seconds. So I was back to 5 minutes according to the profiler, but I noticed that my test runner disagreed. According to my test runner my implementation had a runtime of approx. 3 minutes and 40 seconds. For comparison my test runner puts the original implementation at 3 minutes and 20 seconds for that test. In short, I have spent most of my day trying to optimize my code to compensate for the +20% overhead the profiler introduced. Admittedly, my replacement still needs some speed improvement on some other tests where it still has a factor 2 runtime regression. I also need to fix the last diff, which now I suspect is in my replacement. [1] In case you were wondering, yes those chroots are on two different machines. [2] Of course, it could be a combination of that as well Anyhow, I am too lazy to research it at the moment.

Assume the following situation: You set problem sheets for a class, and at the end of the semester, you have 13 individual LaTeX files, including the preamble (the stuff from \documentclass to \begin document ). Now you want to provide a file that contains all problems, followed by all problems with solutions. You could just concatenate the resulting PDFs, but that would waste quite a bit of paper, as most problem sheets do not fill the whole page. You could just copy n paste the LaTeX code, but that is not neat either. So you want to include the 13 individual documents in one LaTeX document. LaTeX provides \include and \input commands, but these would require that the content of the individual files, i.e. the stuff between \begin document and \end document , is put in a file of its own. You cannot just \input the full problem sheet document, as there must be only one \documentclass command, and a few preamble commands are invalid within the document. There are various solutions suggested, e.g. on Wikibooks and StackOverflow, and there are packages helping with that functionality, such as combine or subfiles. I chose a rather rough, hands-on solution that worked great in my case: The idea is to simply re-define all commands that you have in your preamble to do nothing, including the document environment, before including the other files and constraint to a group. In my case, the preambles were relatively small, as the common definitions of the problem sheets were in a file of their own. I am also using the \foreach command provided by the TikZ package for a convenient loop over all problem sheets:

\foreach \x in  0,...,13   
   
    \excludecomment solution 
    
    \DeclareDocumentCommand \documentclass om 
    \DeclareDocumentCommand \usepackage om 
    \newcommand \Blattnummer [1] 
    \DeclareDocumentCommand \FirstDueDate mmm 
    \DeclareDocumentCommand \ThisDueDate mmm 
    \newcommand \SkippedWeeks [1] 
    \renewenvironment document 
    \section* Problem Sheet \x 
    \setcounter problem 0 
    \input GraphTheoryProblems\x 
   
 
\clearpage
\label sols 
\foreach \x in  0,...,13   
   
    \newenvironment solution 
      \par\addvspace 1em 
     
     
    \DeclareDocumentCommand \documentclass om 
    \DeclareDocumentCommand \usepackage om 
    \newcommand \Blattnummer [1] 
    \DeclareDocumentCommand \FirstDueDate mmm 
    \DeclareDocumentCommand \ThisDueDate mmm 
    \newcommand \SkippedWeeks [1] 
    \renewenvironment document 
    \section* Solution Sheet \x 
    \setcounter problem 0 
    \input GraphTheoryProblems\x 
   
 

To redefine commands taking optional arguments, using the xparse package is convenient. You can see how I include every file twice, first with the solution environment turned into a comment and then again with the solution environment actually showing its content. I create a section header based on the counter of the foreach loop, but I could have easily redefined \title and use that for a header, if my problem sheets had used that command (They don t, as they calculate the problem sheet number and the due date based on the number in the filename very convenient, and maybe worth a blog post of its own). The problem counter is reset for each file, so that the problems are numbered individually. I could have left this out, then the problems would be numbered consecutively across all included problem sheets. The individual and resulting files can be found on the Graph Theory course website.
Flattr this post

If you met Ana, you ll easily remember her. She has a great and pronounced Spanish accent I m glad that the existence of the Debian Women project helped her to join Debian because she has been doing a great job. From KDE packaging to publicity/marketing work, her interests shifted over the years but this allowed her to stay very involved. As she explains it very well, Debian is big enough so that you can stop doing something which is no longer fun for you, and still find something new to do in another part of Debian! Read on to learn more about Ana, the KDE team, Debian s participation to the Google Summer of Code, and more. Raphael: Who are you? Ana: I m Ana Guerrero L pez and I m in my early 30s. I was born and raised in the wonderful city of Sevilla, Spain and I live in Lyon, France. I share my life with another Debian Developer and my paid work is doing Debian support and integration, so you won t be surprised to read that Debian is a big part of my life. Raphael: How did you start contributing to Debian? Ana: Although I knew about the existence of Linux since 1997 or so, I didn t really start using Linux until the summer 2001 when I finally got a computer on my own and an Internet link at home. In the beginning, I was using Mandrake in a dual boot with Windows and later around 2003, I happily moved to only using Debian and ditching the Windows partition. Once settled as a Debian user, I knew anybody could help improve the distribution but I hesitated to join mostly due to two reasons, my perception of Debian was the one of a very elitist and aggressive club and who wants to join this kind of cult^wproject? And even if I wanted to join, I did not know how to get started. By the summer of 2004, the Debian Women project started, it made me seeing Debian as a more welcoming project, and I started maintaining my first packages. The following summer 2005, I attended akademy 2005 (the annual KDE conference) where I had the pleasure to meet there some of the people from the KDE team and this really made a difference for me. Christopher Martin and Adeodato Sim , with the help of other people, have started the maintenance of KDE as a team a few months before and by that time most of the KDE modules where under the maintenance umbrella of the team. This was a very good move since it allowed easily to share the KDE maintenance in a more coordinated way and also eased having non-DDs, like me at that time, to join in and help.

The Debian Women project started, it made me seeing Debian as a more welcoming project.

Raphael: You re part of the Debian KDE team. What s your role in the team and what are your plans for Wheezy? Ana: Nowadays, I am not as active in the KDE team as I used to be in the past. The KDE 3 to KDE 4 transition was quite tiring and changes on the KDE side like the successive marketing renames, the shorter 6 months schedule (it used to be at least 9) or the uncoordinated KDE releases mostly burnt me out. Currently, I am mostly working in helping others to get started within the team, some small fixes here and there, and helping with the uploads: an upload of the full KDE suite to the archive requires some building power and upload bandwidth not everybody have. For Wheezy, with the tentative freeze date in June, the plan is to try to ship the latest possible point release of the KDE 4.8 series. The first release of the series, 4.8.0 was released a couple of weeks ago and while writing these lines, the packaging work for 4.8 hasn t started yet. The next move for the team is getting 4.7.4 in unstable, currently sitting in experimental.

For Wheezy, [ ] the plan is to try to ship the latest possible point release of the KDE 4.8 series.

Besides the KDE packages, there is some software which users perceive as KDE, such as amarok, digikam, etc., which are not part of KDE but fall under its umbrella. These other programs have their own maintainers and their updates depend greatly in the availability of them. For the KDE office suite, we have right now KOffice in the archive. KOffice got a fork some time ago named Calligra and we should replace KOffice by Calligra in the archive before the release of Wheezy. Sadly there isn t yet a final release of Calligra to use. My personal goal for Wheezy was to finish the removal of all the remaining packages depending on KDE 3 and Qt 3 that Squeeze still contained. The removal of the KDE 3 libraries and all the packages using them was quickly achieved after the release of Squeeze. The removal of Qt 3 soon showed that it was task harder than expected since some popular packages (sometimes not in the Debian archive, e.g. third-party scientific software) depend on it, and also Qt 3 is a requirement for LSB compatibility. Right now, Qt 3 has been orphaned for 9 months and nobody has shown any interest in adopting it. Raphael: KDE, much like GNOME, has been forked by people who were unhappy by the direction that the project has taken since version 4 (cf Trinity). What s your personal opinion on KDE 4.x and what s the position of the Debian KDE team concerning this fork? Ana: I use KDE 4 on my laptop and I think it is a solid desktop environment and platform. However I am finding it less and less attractive for me. On one side, my usage of the computer has been slightly changing and on the other side, I do not like how the new developments in KDE are evolving, things like plasmoids or activities are not attractive for me. I have switched my other 2 systems to awesome although I continue to use mainly a bunch of KDE applications: dolphin, konsole, kate, juk, kmix, etc. So you might say my desktop environment is an awesome KDE. Regarding the Trinity project, a lot of users complained very loudly when KDE 3 got replaced by KDE 4 in testing/unstable, so I find quite laudable the decision of some users to act instead and try to continue with a forked development of KDE 3. However the Trinity team seems to be about 3 persons (funny for a project named Trinity ) while KDE 3 is big. In perspective, it does not look that big because KDE 4 is even larger, but it is still too much for such small team. In addition those developers need to maintain Qt3 that has been end-of-lifed years ago by Nokia/Trolltech . So my guess is that sooner or later the project will fade away. Nobody from the KDE team is interested in Trinity and in case someone wants to package it for Debian, they would have to make a new team. For the reasons mentioned above: Qt3 maintenance and reduced upstream group, this would be a bad idea. My advice if you do not like KDE 4 and you miss KDE 3, would be taking a look at razor-qt based on Qt4 and quite similar to KDE 3. I read they have plans to port it to Qt4, but frankly that could take some years same it took to the KDE project for KDE 4.0.0 Raphael: You used to maintain news.debian.net, a WordPress blog dedicated to Debian, but you stopped a while ago. A few months later you started to maintain a Debian page on Google+. Why did you stop the blog and what s your goal with the Google+ page? Ana: I blogged about the reasons I started news.debian.net. In short, I thought Debian needed a better system to publish news, something like a blog. I first tried to suggest the idea to the press/publicity team but they weren t interested, so I started the project alone. IMHO the blog worked quite well and I was feeling like it should be made official. I talked about this with some people but at the time I wasn t pushing it because I had other priorities and I knew pushing it to become official would need some extra time and energy. Stefano decided to start the discussion about making news.debian.net official (that s moving it to a debian.org domain) in its own initiative. After the public discussion and some private exchange of emails with DSA, the situation became frustrating and I decided to close news.debian.net after the release of Squeeze. Later, during DebConf, an officer from the press team announced they were launching a blog and I asked Stefano if he could try to have a discussion about this to see if it could still somehow fit my ideas, and maybe contributing myself, but nobody from the press team answered Stefano s email and the blog hasn t started yet either. Irony that communication didn t work when wanting to improve communication. About the Google+ page, everyday I follow what is going in Debian and quite often I find things I want to share. I do not want to clutter my own profiles with Debian stuff or have people following me because of that, so I decided to create the Debian page when Google+ made them available. I like the fact that people can follow that without having an account in Google+ although they can not comment anonymously. I am not happy about the fact that Google+ is a closed platform but hopefully the data will become easier to export in the near future. Right now, there are some services that provides RSS feeds of Google+ pages if you want to follow the page and you are not in Google+ (or I could setup one if several people ask me). Raphael: Last year you helped to manage Debian s participation to the Google Summer of Code. How did it went? Is there something that you can improve for this year? Ana: I think last year we managed to have people in Debian more aware about what the students were doing. That also helped students to get more feedback and therefore get to know more people in the project and get more integrated. Students were sending periodic public reports available to everybody interested in the status of the projects and some of them also held their own sessions in DebConf. We still failed to start looking for mentors early enough and to give them information about how the GSoC worked and how they could have a successful project. Having good projects in Debian is harder than in other projects because the GSoC mostly promotes having students started in Open Source *coding* for a project, while Debian is more a project about integrating software and we overall do not have so many parts that has to be coded. My personal goal for this year is to try getting the projects earlier to attract good students from the very beginning, even if that means we have less projects than in other years. Raphael: What motivates you to continue to contribute year after year? Ana: Three things. I like improving the OS I use, I like the friends I have made while working in Debian through the years and because I have fun. Also Debian is quite a big project, so if you become tired or burn out working in some area, you always can easily find interesting things to do somewhere else. Raphael: Is there someone in Debian that you admire for their contributions? Ana: Adeodato Sim , he is now in a long leave from the project, but it is one of those persons who made a difference in the project in his job in the release team some years ago. Aur lien Jarno because of his tireless work in (e)glibc and porting of several architectures. I also have special admiration for all those people who have been very active in the project for more than 7-8 years because I know it is not always easy to combine it with real life.

---

Thank you to Ana for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

7 comments Liked this article? Click here. My blog is Flattr-enabled.

Matthew Aslett wrote about how the proportion of projects released under GPL-like licenses appears to be declining, at least as far as various sets of figures go. But what does that actually mean? In absolute terms, GPL use has increased - any change isn't down to GPL projects transitioning over to liberal licenses. But an increasing number of new projects are being released under liberal licenses. Why is that?

The figures from Black Duck aren't a great help here, because they tell us very little about the software they're looking at. FLOSSmole is rather more interesting. I pulled the license figures from a few sites and found the following proportion of GPLed projects:

RubyForge: ~30%
Google Code: ~50%
Launchpad: ~70%

I've left the numbers rough because there's various uncertainties - should proprietary licenses be included in the numbers, is CC Sharealike enough like the GPL to count it there, that kind of thing. But what's clear is that these three sites have massively different levels of GPL use, and it's not hard to imagine why. They all attract different types of developer. The RubyForge figures are obviously going to be heavily influenced by Ruby developers, and that (handwavily) implies more of a bias towards web developers than the general developer population. Launchpad, on the other hand, is going to have a closer association with people with an Ubuntu background - it's probably more representative of Linux developers. Google Code? The 50% figure is the closest to the 56.8% figure that Black Duck give, so it's probably representative of the more general development community.

The impression gained from this is that the probability of you using one of the GPL licenses is influenced by the community that you're part of. And it's not a huge leap to believe that an increasing number of developers are targeting the web, and the web development community has never been especially attached to the GPL. It's not hard to see why - the benefits of the GPL vanish pretty much entirely when you're never actually obliged to distribute the code, and while Affero attempts to compensate from that it also constrains your UI and deployment model. No matter how strong a believer in Copyleft you are, the web makes it difficult for users to take any advantage of the freedoms you'd want to offer. It's as easy not to bother.
So it's pretty unsurprising that an increase in web development would be associated with a decrease in the proportion of projects licensed under the GPL.

This obviously isn't a rigorous analysis. I have very little hard evidence to back up my assumptions. But nor does anyone who claims that the change is because the FSF alienated the community during GPLv3 development. I'd be fascinated to see someone spend some time comparing project type with license use and trying to come up with a more convincing argument.

(Raw data from FLOSSmole: Howison, J., Conklin, M., & Crowston, K. (2006). FLOSSmole: A collaborative repository for FLOSS research data and analyses. International Journal of Information Technology and Web Engineering, 1(3), 17 26.)

comments

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the S in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones? tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

• using extend directive;
• using pass_persist directive;
• using AgentX protocol.

SNMP in a nutshell The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card. Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:

GET

retrieve a variable.

GETNEXT

retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.

SET

request the modification of a variable.

Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578. A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner¹:

$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable
 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0

Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules. Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID.

$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP. To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output:

$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641

I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use:

# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values

The result can be retrieved through NET-SNMP-EXTEND-MIB:: nsExtendOutput2Table:

$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0

It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command:

$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets

This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user. Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction:

   PING
   PONG
   get
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   counter64
   223941
   getnext
   .1.3.6.1.4.1.39178.100.1.1.1.2.2
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
   counter64
   0

The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP:: ExtensionSNMP:: PassPersist extension from S bastien Aperghis-Tramoni. You can find the complete example on Github.

use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;
sub update_tree  
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces)  
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null  ") or next;
        while (<ETHTOOL>)  
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
         
        close(ETHTOOL);
     
 

With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub. pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:

• Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
• While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:

1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
4. 64bit types are fully supported. Traps are also supported.

Perl and Python Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use:

• An implementation of ETHTOOL-MIB I have done in the same way as the pass_persist version. My Perl skills are a bit rusty and the code has many rough edges.
• An implementation of a sub-agent for Redis servers.
• An implementation of BRIDGE-MIB shipped with Net-SNMP.

There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with new API To write a sub-agent in C, you have to choose between two API:

1. the API inherited from UCD-SNMP also known as the traditional API;
2. the new API developed for Net-SNMP 5.x.

With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API. The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables. On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook). I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version² of what has been modified:

/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);
/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next)  
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    ifIndex = if_nametoindex(ifa->ifa_name);
    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++)  
        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);
        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];
        CONTAINER_INSERT(container, rowreq_ctx);
     
 

See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way. Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:

• Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
• Writing a standalone sub-agent for some MIB. As a standalone project, you won t run into coding style problems either.

For other uses, let s have a look at the traditional API.

C with traditional API The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub):

static oid ethtool_oid[] =  1, 3, 6, 1, 4, 1, 39178, 100, 1 ;
static struct variable3 ethtool_vars[] =  
   1, ASN_COUNTER64, RONLY, ethtool_stat, 3,  1, 1, 2 
 ;
int main()
 
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
 

The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:

• magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
• type is the type of the variable that will be returned.
• acl tells if the object is read-only (RONLY) or read-write (RWRITE).
• findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
• The two last fields are the relative OID this entry is responsible for along with its length.

The prototype of ethtool_stat() handler function is the following:

static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);

This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:

• struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
• oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
• int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
• size_t *var_len is used to tell the size of the returned object.
• WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation³.

To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation. Here are three projects using this API to provide SNMP support:

• Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
• lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
• Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

Other solutions The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section Extending agent functionality :

• exec and sh are deprecated. Use extend instead.
• pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
• proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
• smux has been superseeded by AgentX.
• perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: write programs that do one thing and do it well . However, Net-SNMP FAQ says:

Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.

---

1. Additional arguments for snmp* are usually required to specify a version and a community (for SNMPv1 and SNMPv2c) or a user (for SNMPv3). In all examples, I do not specify anything. I assume that ~/.snmp/snmp.conf contains the appropriate bits. Moreover, some outputs are truncated.
2. No allocations, no variable declarations and no error checkings. Do not do this at home!
3. If you want to look at an example of how to handle a SET request, look at the
   SNMP support for Keepalived. A SET operation is more complex because SNMP ensures that the
   SET operation is atomic across all provided variables. It uses a three-staged commit. The traditional API is very inefficient for this kind of operation because the object has to be located again for each step.

As you might have noticed from my previous posts, we've moved phpMyAdmin code to github. Now I'm going to describe some things which might be useful for you if you are about do to similar switch. While using git, moving to another hosting provider should be pretty straightforward. Just add another remote, push code there and it should be done. On the other side you probably have dozens of things in your infrastructure which you need to adjust. So the first thing to do is write down what all services are connected to your current git repositories. Let me name some which might be worth checking:

• continuous integration server
• snapshots generating
• demo server (in case you're providing something like we do)
• commit notifications
• various statistics services such as cia.vc
• website generating
• references in wiki, website and documetation

Once you think you have remembered all important things (the less important will probably show up anyway, but majority of things should work), you're ready to make the move. I've decided to make the move in few steps. First push all data to new location, what can take some time. I'll get in more details to that later. In the same time I asked all contributors to give me their login information, so that I can allow them access to new repositories. Once all recently active developers were migrated, it was time to push all remaining commits to new git repositories and make the switch for real. Pushing git repo to another location, should be pretty easy. On the other side if you have many branches, it get's slightly more complex, I've ended up with following shell snippet (pushing all branches present in origin to github remote):

git branch -r   grep origin/   grep -v HEAD   sed 's@.*/@@'   while read b ; do git checkout $b ; git push github $b:$b ; done

Please ensure that you check output of this, because you may hit network problems somewhere in the middle and you end up with few branches than you expect. As the code is pretty much idempotent, you can safely run it several times until there is nothing to push. You should also push all tags to new location:

git push --tags github

Okay, we've all data on right place, so let's switch all our users to new location:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git # read/write
git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git # read only

Of course everybody has to do this manually. Next good thing is to let people know when they are using wrong repo (which will stay there for some time). Unfortunately there is AFAIK no way to warn them on pull, so let's warn at least on push:

$ cat > hooks/update
#!/bin/sh
echo "phpMyAdmin git repositories have moved to https://github.com/phpmyadmin"
exit 1

I think this is pretty much all. You can find some more bits in our Git migration wiki page. PS: Thanks to github for offering us hosting and sorry for breaking their branch displaying page by too many divergent branches.

Filed under: Coding English Phpmyadmin 4 comments Flattr this!

If you ever get this cryptic error when loading an Android native app:

FATAL EXCEPTION: main
java.lang.RuntimeException: Unable to start activity ComponentInfo org.wikibooks.OpenGL/android.app.NativeActivity : java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1768)
       at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1784)
       at android.app.ActivityThread.access$1500(ActivityThread.java:123)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:939)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loop(Looper.java:130)
       at android.app.ActivityThread.main(ActivityThread.java:3835)
       at java.lang.reflect.Method.invokeNative(Native Method)
       at java.lang.reflect.Method.invoke(Method.java:507)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:847)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:605)
       at dalvik.system.NativeStart.main(Native Method)
Caused by: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so
       at android.app.NativeActivity.onCreate(NativeActivity.java:199)
       at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1047)
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1722)
       ... 11 more

This may mean that Java couldn't find the ANativeActivity_onCreate function in your code, because it was stripped by the compiler. If you use the native_app_glue NDK module, you may have noticed this strange code:

    // Make sure glue isn't stripped.
    app_dummy();

Let's experiment what happens with and without this line: Calling app_dummy:

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so    grep ANativeActivity_onCreate
000067fc g    DF .text  000000f8 ANativeActivity_onCreate

Not calling app_dummy :

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so   grep ANativeActivity_onCreate
$   # nothing

native_app_glue mainly defines Android callbacks. Since none of them are called directly by your code, the compiler strips the android_native_app_glue.o module entirely. If you use app_dummy however, it embeds it. Fortunately the compiler cannot strip the module on a per-function basis That's why you need to call app_dummy when using the native_app_glue NDK module. This looks like a ugly work-around though - isn't there a cleaner way?

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this. Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don t want to know about what s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path. So I created a little tool called fatrace ( file access trace , not fat race ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203
[...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its not possible to determine a path (usually because it s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace. If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p pidof gnome-terminal ). So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes. My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now. The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video coverage of FOSDEM for years, and suggested that this year, maybe they should consider using dvswitch for video coverage. While they seemed agreeable to this idea, they simply had not had the time to look at it in detail, and were therefore not using it. Since I've used dvswitch in the past, both as part of the DebConf video team and for my own concert recordings, I reasoned this a problem that could be solved by joining the video team. However, when it came to be time to start preparing for FOSDEM 2012, we found that many of the NamurLUG people were not going to have as much time to prepare for video coverage as they had during past years, and therefore responsibility of FOSDEM video coverage fell entirely to me. This was fairly unexpected, though not too daunting. For those of you who don't know dvswitch very well: a typical dvswitch installation for coverage of a talk requires:

• Two cameras; one to capture the speaker, one to capture audience members asking questions.
• A twinpact 100, used to convert a VGA output into a DV stream that we can load into dvswitch
• Three laptops: one for the audience camera, one for the twinpact, and one for the speaker camera and the dvswitch installation
• A GigE network between the three laptops, with an uplink to do streaming
• A large amount of diskspace to store the recordings on. Since dvswitch records in raw DV, you require approximately 13GB per hour of recorded video.
• Some powerful systems to do transcoding from DV to a streaming format such as Ogg Theora or WebM
• A fairly extensive audio setup: one wireless headset microphone for the speaker, one or two wireless handheld microphones for people in the audience, and a number of microphones at select positions in the room to pick up ambient noise (which for obvious reasons you don't want to overdo on, but which if done well makes the audio on your recording sound much more natural). This also requires a mixing console.
• And last but not least a (network of) streaming server(s) to do the streaming to people watching from home.

This is a rather large amount of work, and therefore it is not too unexpected that for DebConf, the DebConf video team takes a week to properly set up two rooms so that they can be used for recording. However, for FOSDEM we only have less than a day on-site to set up things, and we were going to cover five rooms. So, I tried to cut corners as much as possible:

• We were going to work with one camera per room, rather than two. Not only would this reduce the number of required volunteers (of which I wasn't sure yet at that point how I'd find them), it would also reduce the amount of preparation needing to be done, and it would reduce the number of things that could go wrong.
• We were going to have the network team do our network, rather than try to do it ourselves. The result was that we were not going to have to lay out cables, cut them, crimp connectors on them, test them, etc, and would only need to worry about using the network, instead of building it.
• Since I'd read Thomas' announcement about an ''offer I cannot refuse'', I contacted him to see about that offer and get it all set up for FOSDEM. This way, I wouldn't have to worry about streaming servers etc, instead just about getting a stream on the network, into a flumotion server running somewhere; and after that, it would no longer be my problem.
• I decided to simplify the audio setup somewhat, and not use ambient mics. These are nice, but not essential, so we could do without them, and that would be (at least) two microphones per room that the team wouldn't have to rent, place, wire up, and tune.

So in the end, I would need to take care of ten firewire-capable laptops, five cameras, five mixing consoles, several microphones, a number of transcoding machines, and shitloads of cabling (power, firewire, and audio). And about a week or two before the conference, as the massiveness of what we were about to do started to sink in, I was starting to have bad dreams of what would happen, wondering what I'd gotten myself into this time. Now, one day after FOSDEM, I have to say it all turned out okay, but there are some things where there's still room for improvement. So that I don't forget, I thought I'd make a list of things that went well, and one of things that didn't go well. And since that might be interesting for other people, I thought I'd do it here, rather than in a private file. First, things that did not go so well:

1. Set-up for some rooms took more time than we had, and as such some rooms did not get properly streamed or recorded for their first few talks. This was due to the fact that there was some confused communication between some members of the team, which meant that we lost a day that we'd planned for preparation, and as such we couldn't test as much as we'd hoped. We need to improve on that next year. Ideally, we should run the full set-up somewhere, with all cameras and all laptops running, and making sure that everything is ready to be set up and that we have every cable we need, before dividing everything over a number of boxes (one per room) and bringing it to the conference.
2. While some of our cameras were semi-professional Panasonic cameras that had a balanced XLR audio input, others were much lower-level camcorder-style cameras that did not have such a thing. When a camera has a good audio input, it's fairly easy to set up a link from the mixing console to the camera and get the audio into the stream that way. When a camera does not have such a thing, the set-up gets much more complex. Since we had so much to worry about, we did not notice that in one room, on saturday, something went wrong with audio. To combat this, we should improve on our audio set-up for next year, and also verify the streams every once in a while (I did the latter on sunday, so everything should be fine there).
3. Inherent to an ad-hoc network, there were some network-related issues. For instance, on sunday evening, during the last session in that room, Ben informed me that the link to room H.1301 had gone down. As we were investigating, we easily found the source of the problem:
   
   The cable had been put under a door, and had been 'protected' with loads and loads of duct tape, but apparently that wasn't enough. This wasn't something we wanted to fix anymore, at that point; instead, we just let the cable be and focused on other things.
4. Storage. None of the systems used for the recordings were my own; instead, some were owned by IRILL and some were rented. If I want to do anything with my recordings, that means I have to copy the data off. The system I chose was to bring a USB3 hard disk and copy everything over; but for saturday, this took an hour and thirty minutes. On sunday, my USB3 disk was fairly full, so I had to revert to a USB2 one, which increased the time by much. In the end, we had to abort copying files, and we'll now have to be a bit creative to get them.
5. Volunteer management. I originally set up a wiki page to allow people to sign up for talks that they wanted to help out with. Unfortunately, that didn't work so well as it did for DebConf; there were several talks that did not have enough people, resulting in the volunteers from the previous talk remaining on post and finishing their talk. Also, there were some people that signed up without contacting me, or without telling me what name they used on the wiki page, which made it somewhat harder to know who to talk to. I lost count of who had signed up to help out. On Sunday, I decided to set up an IRC channel instead, through which volunteers could communicate more directly and just ask for people to take over, instead of registering for a talk first and then hoping nobody would forget. Also, since I'm terribly bad with faces and video volunteers did not have anything that made them stand out from the rest, I couldn't just walk around until I would see someone involved with video work and ask them to take over in a room. Having a separate 'video team' t-shirt could help there.

After the bad news, here's the list of things that did go well:

1. Streaming. Even though choosing flumotion rather than icecast (as we'd done for streaming in the past) involved extra work for me before the conference (I had to write some code so that I could stream from dvswitch into flumotion, which wasn't possible out of the box), the guys at Flumotion decided to send over one of their support engineers. As such, I simply did not have to worry about streaming servers crashing or failing to do what they were supposed to do. While I have no reason to think any of the Flumotion servers were having issues, even if they were I wouldn't have known about it, since Francesc took care of it all and never ever required my involvement about something on the flumotion side. This was extremely valuable.
2. Training right before the welcome talk. Explaining to people in a few words how dvswitch works, and then immediately following that up with a live demonstration, isn't a bad way for them to understand.
3. Despite all the problems we did have, once everything had been set up and all the gear was more or less manned, actually recording and streaming did go fairly well. The feedback I received so far was mostly positive, which makes me very happy.

In the end, while there certainly is still room for improvement and things did not go as smoothly as I'd hoped, they have gone much better than my worst fears. There's still some work to do, which I'll be doing with the NamurLUG people this week, but all in all, it's gone pretty good. As to the usage of the streams, I asked Flumotion for some statistics. Since I was fairly late in asking, they could only provide me with statistics about sunday:

• Number of clients connecting:
  
• Traffic:
  

Fresh from the oven, monthly report of what I've been working on as DPL during January 2012.

---

Dear Developers,
here is another monthly report of what happened in DPL-land, this time for January 2012. There's quite a bit to report about --- including an insane amount of legal-ish stuff --- so please bear with me. Or not. Legal stuff

• Webmaster heroes have decided to tackle the long standing issues of copyright and licensing of the Debian website. I've accepted to help them out in reaching consensus with license choice and I'm happy to report that we've managed to pick a DFSG-free license (BSD-ish) for future contributions. Webmasters will soon contact contributors to re-license old contributions (or get rid of them), so hopefully will have a DFSG-free website RSN. Many thanks go to David Pr vot for successfully tackling such a can of worms.
• I've sought a second legal advice on the constraints that trademarks (might) impose on the work-flow of a distro like Debian. Luckily, it is coherent with one I've sought in the past so I'm now in condition to wrap up the "trademark vs DFSG" thread on -project with the missing legal information. Hopefully, I'll find time to do that sometime next week.
• I've restarted discussions with the Debian France association so that they can become a Debian Trusted Organization (as per Constitution 9.3). Members of the board of the association seem to be interested and I'm positive it could happen fairly soon. The importance of this is that we could use a back-up association in Europe to hold Debian assets, to complement the services that FFIS are already offering us.
• Thanks to the contributions of Benjamin Mako Hill and SPI lawyers, I've now what I consider a final draft of a trademark policy for Debian trademarks. Before proposing it to you, I'm waiting for some feedback from another umbrella organization for Free Software projects, that is working on a trademark policy for all their associated projects. As many Free Software projects are seeking trademark protection these days, I see benefits in having uniform (and sane!) policies. I hope to be able to gather the feedback I still miss this week-end at FOSDEM, and let you know shortly after that. Once this is done, we'll also be able to (finally!) relicense all kinds of Debian logos under a DFSG-free license. On this front, I've also updated http://www.debian.org/trademark with the information needed to contact us about trademark usage; hopefully it'll reduce the burden of answering to such inquiries.
• With the help of Kenshi Muto, Fumitoshi Ukai, Ishikawa Mutsumi, Shuzo Hatta, and Yasuhiro Araki we've started the process to move the Debian trademark in Japan from individuals (who are present or past members of the Debian JP association) to SPI. That would help dealing with these matters, as well as ensure that important Debian assets are held by Debian Trusted Organizations.
• I remind you that we've an ongoing complaint with the current registrant of debian.eu, domain that we believe Debian should legitimately own. Lawyers at SPI has now formally contacted the current owner and hopefully we'll be able to solve the issue amicably in the next months.
• Some of the past legal advice I sought for PPA came handy in a discussion on the legal risks of running a service like mentors.debian.net, hopefully addressing part of the issues in turning that into mentors.debian.org
• Patent policy for the Debian archive is now ready as well and I also have a patch for the website ready to be merged. I'm just waiting for the final blessing from SPI (lawyers) to go ahead and publish it.

Most of the above wouldn't have been possible without the precious help of folks at SFLC working for SPI and Debian. Be sure to thank SFLC for what they're doing for us and many other Free Software projects. Coordination Nobody stepped up to coordinate the artwork collection for Wheezy I've mentioned last month, so I've tried to do a little bit of that myself. The -publicity team is now preparing the call for artwork and hopefully we'll send it out RSN. In case you want to help, there is still a lot of room for that; just show up on the debian-desktop mailing list. Sprints A Debian Med sprint has happened in January, and Andreas Tille has provided a nice and detailed report about it. Some more sprints are forthcoming this spring, how about yours? Money

• We got from SPI a prepaid and rechargeable credit card that we can use for expenses or other kind of guarantees. Many thanks to Michael Schulteiss, SPI treasurer, for his help with that. Using it, we've redeemed 10k$ of credits offered to us by Amazon, that (thanks to ongoing work by Lucas Nussbaum) we're going to use to make our QA rebuilds independent from the underlying computing infrastructure.
• Thanks to the help of Luca Capello, we advanced quite a bit on forming the Debian Event Box kit that should make it easier to set up Debian booth at FOSS events. We bought the machine for it (for about ~755 CHF) and the box to contain it will soon be on its way as well. If you're at FOSDEM, tend to the Debian booth to check it out (and possibly help out with the technical setup).
• We've got quite a bit of donations during the December holidays. I've took the chance to thank donors, discuss what we do with donations and the status of publishing periodic Debian budgets.
• Pinged by Yves-Alexis Perez, I've now properly documented the fact that DDs are welcome to apply for hardware sponsoring, in case the hardware can be used to help/improve their Debian work. As suggested by Yves-Alexis, you can also advocate other DDs for hw sponsoring.
• Given hardware invariably age and that we can afford it, I've prodded DSA to prepare a general hardware replacement plan for our machines. Planning will go on this week-end and FOSDEM (thanks to Martin Zobel-Helas and Faidon Liambotis for their presence here) and I hope to have an approved machine replacement plan well before the end of the current DPL term (although I'm usually optimist...).

Important stuff going on Other important stuff has been going on in various area of the project in January. I'd like to point your attention to a couple of things:

• People active on debian-mentors have proposed an improved work-flow to deal with sponsoring/mentoring requests, based on the usage of a new pseudo package "sponsorship-requests". Thanks to Ansgar Burchardt, Jakub Wilk, Arno T ll, and Gregor Herrmann for working on this.
• Raphael Hertzog has kickstarted work on DEP-2, as a way to rationalize the flow of package-related information that (co-)maintainers get. Discussion about the idea are ongoing on the debian-qa mailing list.

Miscellanea

• Work has further progressed in reaching out to companies with an interest in giving support for, and contributing to Debian. Thanks to Alexander Wirt the technical work is now done and some sort of governance policy has been decided. Further step for me is to announce it properly hoping to reach out to as many interested companies as possible. I hope to finalize that in the next month. (If you're working for such a company and you happen to read this, feel free to reach out to me already.)
• I've completed an old todo item setting up and documenting titanpad.debian.net, service that has been requested for collaborative work during various kinds of online events. Help is welcome to help administering the service (see doc).
• SPI has clarified the role of project representatives and, as a consequence of that, I (as DPL) no longer receive SPI board discussions addressed to board@spi. That is good not only for the sanity of my inbox, but also because it puts all projects affiliated to SPI at the same level of communication within SPI. Thanks to Robert Brockway for his work on this.

In the unlikely case you've read thus far, thanks for your attention! Happy Debian hacking.

---

PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

Wow, a lot happens in 3 months. I'm sure I've missed something... I think the biggest achievement would be toilet training. Zoe's been using the toilet for some time now, but still wearing diapers, and just after Christmas (when she was sick and vomited on the carpet) we figured we had nothing to lose, so we ditched the diapers. I can count on the fingers of one hand the number of accidents she's had since then. I'm really happy to have this out of the way so early. Using a toilet away from home is still more of a challenge, but she seems to have very good bladder control, and fairly predictable bowel movements. Speaking of sick, she was the sickest she's ever been around Christmas time. It coincided with her top canines finally coming through. She had a cold, which progressed into an ear infection, and she had this dreadful persistent dry cough, which then progressed into a dreadful productive wet cough. You knew she was really sick because she was extremely cuddly and lethargic. Since around that time, her sleep has gone completely to hell. We've had some really bad nights where she'll wake up after a couple of hours of going down for bed, or for a while, she was reliably waking up at 4am and taking a good couple of hours to settle back down. Lately she's been sleeping through again, but waking up at 6am instead of the usual 7-7:30am. We bought a clock that changes colour at programmable times, so we're trying to train her that if she wakes up and the clock isn't green yet, she should try and go back to sleep. Not a huge amount of success there yet. Christmas itself was good, but fairly quiet because Zoe was sick. Sarah managed to get a fabulous photo with Santa, which just fills me joy every time I look at it. We had some friends around for Christmas dinner, and Zoe vomited all over her new Cabbage Patch Kid doll and I think she threw up a couple more times between that day and the day after, so that wasn't much fun. She's becoming more and more independent. "Zoe do it" is the most frequent thing out of her mouth, but fortunately she still accepts help as well. We've also been getting plenty of sneak previews of the Terrible Twos. Fortunately they're pretty short-lived, but it's definitely going to be a challenge to my patience. We'd been intending to keep Zoe rear-facing in the car until she was 2, which is the current recommendation over here, but I caved in recently when she was getting in the car and refusing the sit down (she'd stand up facing forwards and it was next to impossible to rectify the situation). After she did this to me a few times when I took her out in the car, I had enough and turned the car seat around. It's improved things a bit, but she still has her moments. We also changed her day care, from the one day a week at a home day care, to two days a week at a commercial place. We were becoming more and more unhappy with the existing day care, and so decided to move her. It's a lot more money, but after just a week, we were seeing a noticeable change (for the better) in Zoe's response to going to day care, and she seemed to have picked up a bunch of new skills as well (sitting crossing her legs, holding a crayon like a pencil). The place she's in now is pretty new and very shiny. There's a couple of kids in her class that she knows from the local park, so overall it's looking like money well spent and definitely a change for the better. Zoe had her first hair cut last weekend, and it was fairly uneventful. We went to this cute place in Palo Alto called Snipits, which was ludicrously expensive and I'd only ever take Zoe there for her first haircut. There was another kid there who was screaming bloody murder, and I think that unsettled Zoe more than anything else, so she was a bit uncomfortable to start with, but once the hairdresser put an Elmo DVD on, it was all good. Sarah's Mum is coming over to visit for a couple of weeks in a couple of weeks time, so Zoe will be very happy to see her Nana. They're going to go on a cruise out of New Orleans down to Mexico and part of the Caribbean. Unfortunately I'm too busy at work at the moment to burn a week, so I'll just take a four-day weekend and check out New Orleans with them when they get back from the cruise.

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well. We scrambled to setup alternative caching servers to handle the bandwidth. When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support. I immediately contacted support and then received the message:

Hello
We have received the following complaint associated with your server/service.
IP: 76.73.121.164
To prevent any further abuse we have suspended this service. In order to
resume, we request you to cooperate with our investigation as promptly as
possible. Please respond to us with the following details:
(1) What has caused the complaint
(2) What is the server used for. Purpose?
(3) How can you resolve the complaint and make sure it will not be repeated.
Depending on the nature of the complaint and your response, we will put back
the server online.  Please note, this has violated our Terms Of Service. We
expect your response within 24 hours, otherwise your account will be
terminated permanently.  Thank you

What?? What complaint?? I followed up but had to wait til the next day to get the response.

Here is the full log,
An email advertizing the Domain Name: saharareporters.com
has been sent to the blacklist.woody.ch spamtrap.
This Domain does resolve to IP addresses one of which your are responsible:
76.73.121.164
Please investigate why this Domain has been advertized.
Attached you find the headers and reports in ARF for automatic processing.
Feedback is appreciated.
Actual listing periods:
Bounce: 1 Hour in DNS.
Whitelisted IP: Not lised in DNS.
Spam: 24 hours in DNS.
Every Hit: 14 days in evidence DB.
For any questions or Feedback, contact abuse@woody.ch
From: is intentionally set to a bit-bucket.
Kind regards
-Benoit Panizzon-

There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody's World Blacklist Page. I plugged in our IP address into their checker and I got:

Output from the Check, if empty the IP is not listed.
164.121.73.76.[name of the blacklist] being tested.
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
164.121.73.76.blacklist.spambag.org has address 208.91.197.182
164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"

So, Woody's World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The "RBL List" link takes me to a page advertising "5 foods you must not eat." Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is "poor". Why? According to Barricuda, the reasons could be:

• Your email server contains a virus and has been sending out spam.
• Your email server may be improperly configured.
• Your PC may be infected with a virus or botnet software program.
• Someone in your organization may have an infected PC with a virus or botnet program.
• You may be using a dynamic IP address which was previously used by a known spammer.
• Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
• You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
• In some rare cases, your recipient's Barracuda Spam Firewall may be improperly configured.

Keep in mind, this IP address is not used for sending email. It's just the web site. I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms "woody spamtrap blacklist" and I found a pattern in URLs that suggested I plug in the following: http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be "JAMES IBORI ex-governor of DELTA STATE oil city." The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori. And that, my friends, is enough to have one of the most prominent independent African news organization taken offline. But, Sahara Reporters shouldn't necessarily feel singled out. Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don't know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change. Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch. I'm currently following up with both Woody's World and Server4All. However, once this particular issue is resolved, we're left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem. All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (213.68 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg The biggest change I made is a small patch that brings to an end years and years of recurring discussions about the build-arch and build-indep targets of debian/rules (see #229357). Last year the technical committee took this issue in its hands (see #629385) but it failed to take any resolution. Fortunately thanks to this we got some concrete numbers on the colateral damages inflicted on the archive for each possible approach. In the end, Guillem and I managed to agree on the way forward. The remaining of what I did as dpkg maintainer has not much to do with coding. I reviewed the work of Gianluca Ciccarelli on dpkg-maintscript-helper who is trying to provide helper functions to handle migration between directories and symlinks. I also reviewed a 2000-lines patch from Patrick Schoenfeld who s trying to provide a perl API to parse dpkg log files and extract meaningful data out of them. I updated the dpkg-architecture manual page to document the Makefile snippet /usr/share/dpkg/architecture.mk and to drop information that s no longer releveant nowadays. I reviewed a huge patch prepared by Russ Alberry to update the Debian policy and document the usage of symbols files for libraries. As the author of dpkg-gensymbols, I was keen to see it properly documented at the policy level. I brought up for discussion a detail that was annoying me for quite some time: some copyright notices were embedded in translatable strings and updating them resulted in useless work for translators. In the end we decided to drop those notices and to keep them only at the source level. I updated my multiarch branch on top of Guillem s branch several times, all the fixes that were in my branch have been integrated (often in a modified form). Unfortunately even if the code works quite well, Guillem doesn t want to release anything to Debian until he has finished to review everything and many people are annoyed by the unreasonable delay that it imposes. Cyril Brulebois tried to release a snapshot of the current multiarch branch to experimental but Guillem has been prompt to revert this upload. I m somewhat at a loss in this situation. I offered my help to Guillem multiple times but he keeps doing his work in private, he doesn t share many details of his review except some comments in commit logs or when it affects the public interface. I complained once more of this sad situation. Debian Package Maintenance Hub That s the codename I use for a new infrastructure that I would like to develop to replace the Package Tracking System and the DDPO and several other services. I started to draft a Debian Enhancement Proposal (DEP), see DEP-2, and requested some comments within the QA team. For now, it looks like that nobody had major objections on the driving idea behind this project. Those who commented were rather enthusiastic. I will continue to improve this DEP within the QA team and at some point I will bring the discussion to a larger audience like debian-devel@lists.debian.org. Package Tracking System Even if I started to design its replacement, the PTS will still be used for quite some time so I implemented two new features that I deemed important: displaying a TODO notice when there is (at least) one open bug related to a release goal, displaying a notice when the package is involved in an ongoing or upcoming transition. Misc packaging tasks I created and uploaded the dh-linktree package which is a debhelper addon to create symlink trees (useful to replace embedded copies of PHP/JavaScript libraries by symlinks to packaged copies of those files). I packaged quilt 0.50. I helped the upstream authors to merge a Debian patch that had been forwarded by Martin Quinson (a quilt s co-maintainer). I packaged a security release of WordPress (3.3.1) and a new upstream release of feed2omb and gnome-shell-timer. I prepared a new Debian release of python-django with a patch cherry-picked from the upstream SVN repository to fix the RC bug #655666. Book update We re again making decent progress in the translation of the Debian Administrator s Handbook, about 12 chapters are already translated. The liberation campaign is also (slowly) going forward. We re at 72% now (thanks to 63 new supporters!) while we were only at 67% at the start of January. Thanks See you next month for a new summary of my activities.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

Hello Thanks to the effort of Felix Geyer (debfx) and Manuel Montecelo (mafm), most of SDL packages in Debian unstable are now up-to-date. These guys did a really great job: most of the time, I could upload the packages they prepared without modification. These guys should become DD, they have the skills. Ok, debfx has started the process (congrats) and mafm is still thinking about it Anyway, here the messages for all those who gave up packaging new SDL games for Debian because of stale SDL packages: SDL team is alive, the core SDL packages are now up-to-date, the wait is over. This is true also for Perl SDL games: I ve uploaded yesterday the latest Perl bindings to SDL. Unfortunately, this upload broke some games like frozen-bubble, pangzero and dizzy. But new versions compatible with new SDL perl are available upstream so theses games won t be broken for long. All the best

(This page was edited a few times, details are available.) Grab it while it s hot! Thanks to the hard work of dpkg developers and many (generations of) developers, multiarch is becoming a reality. If you want to give it a try, install dpkg from experimental, add a foreign architecture, and start trying to install packages. Example on amd64:

# dpkg --add-architecture i386
# dpkg --print-foreign-architectures
i386
# apt-get update
[ lots of amd64 and i386 Packages files get downloaded ]
# apt-get install mksh:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ed:i386
The following NEW packages will be installed:
  mksh:i386
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 414 kB of archives.
After this operation, 707 kB of additional disk space will be used.
Get:1 http://ftp.fr.debian.org/debian/ sid/main mksh i386 40.4-2 [414 kB]
Fetched 414 kB in 0s (664 kB/s)
Selecting previously unselected package mksh.
(Reading database ... 171933 files and directories currently installed.)
Unpacking mksh (from .../archives/mksh_40.4-2_i386.deb) ...
Processing triggers for menu ...
Processing triggers for man-db ...
Setting up mksh (40.4-2) ...
update-alternatives: using /bin/mksh to provide /bin/ksh (ksh) in auto mode.
Processing triggers for menu ...
# mksh
$ ldd $(which mksh)
linux-gate.so.1 =>  (0xf779c000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf75d6000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf75b9000)
/lib/ld-linux.so.2 (0xf779d000)

Of course, installing an i386 mksh package isn t exactly what multiarch is about. Dozens of packages have been patched already to add Multi-Arch fields, but until their (recursive) dependencies have been multiarch-ified, foreign packages can be uninstallable, as can be seen below, with the usual why is it uninstallable? hunt (shortened output):

# apt-get install psutils:i386
The following packages have unmet dependencies:
 psutils:i386 : Depends: libpaper1:i386 but it is not going to be installed
# apt-get install libpaper1:i386
The following packages have unmet dependencies:
 libpaper1:i386 : Depends: ucf:i386 (>= 0.28) but it is not installable
                  Recommends: libpaper-utils:i386 but it is not going to be installed
# apt-get install ucf:i386
E: Package 'ucf:i386' has no installation candidate

Another example, successful handling of the installation of a foreign package, while it s already installed with the host architecture:

# apt-get install xz-utils:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  liblzma5:i386
Suggested packages:
  xz-lzma:i386
The following packages will be REMOVED:
  xz-utils
The following NEW packages will be installed:
  liblzma5:i386 xz-utils:i386
0 upgraded, 2 newly installed, 1 to remove and 9 not upgraded.
Need to get 440 kB of archives.
After this operation, 410 kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://ftp.fr.debian.org/debian/ sid/main liblzma5 i386 5.1.1alpha+20110809-3 [205 kB]
Get:2 http://ftp.fr.debian.org/debian/ sid/main xz-utils i386 5.1.1alpha+20110809-3 [235 kB]
Fetched 440 kB in 0s (478 kB/s)
dpkg: xz-utils: dependency problems, but removing anyway as you requested:
 dpkg depends on xz-utils.
 xz-lzma depends on xz-utils.
 dpkg-dev depends on xz-utils.
(Reading database ... 171952 files and directories currently installed.)
Removing xz-utils ...
Processing triggers for man-db ...
Selecting previously unselected package liblzma5:i386.
(Reading database ... 171908 files and directories currently installed.)
Unpacking liblzma5:i386 (from .../liblzma5_5.1.1alpha+20110809-3_i386.deb) ...
Selecting previously unselected package xz-utils.
Unpacking xz-utils (from .../xz-utils_5.1.1alpha+20110809-3_i386.deb) ...
Processing triggers for man-db ...
Setting up liblzma5:i386 (5.1.1alpha+20110809-3) ...
Setting up xz-utils (5.1.1alpha+20110809-3) ...
# dpkg -l xz-utils xz-utils:i386 'liblzma*:*'
[   ]
un  xz-utils            <none>
ii  xz-utils            5.1.1alpha+20110809-3
ii  liblzma5:amd64      5.1.1alpha+20110809-3
ii  liblzma5:i386       5.1.1alpha+20110809-3
# ldd $(which xz)
    linux-gate.so.1 =>  (0xf776b000)
    liblzma.so.5 => /usr/lib/i386-linux-gnu/liblzma.so.5 (0xf7724000)
    librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xf771b000)
    libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xf7701000)
    libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf75a4000)
    /lib/ld-linux.so.2 (0xf776c000)
# zgrep -a '_("Activities")' gnome-shell-3.2.2.1.tar.xz
        this._label = new St.Label(  text: _("Activities")  );

zgrep is a shell script, but it calls xz, which is from the i386 package, and everything runs just fine. Running it through strace -f -e '', I discovered those messages, which I had never seen before:

[ Process PID=8900 runs in 32 bit mode. ]
[ Process PID=8899 runs in 64 bit mode. ]
[ Process PID=8900 runs in 32 bit mode. ]
[ Process PID=8899 runs in 64 bit mode. ]
 

What s next? We re late! It s time to check what happens with that dpkg package, report bugs, and convert more libraries! Please think of the kittens, and coordinate with the release team to make sure you don t delay a transition when uploading a shiny, multiarch-ified package. In a nutshell, if you received a patch from Steve Langasek or Riku Voipio, it s a good indication your package will be helpful quickly when it s multiarchified. Since zlib is directly depended on by 2000+ packages, #569697 was pinged right after the dpkg upload; but many other packages will need patches and heavy testing. Hurry up, the freeze is coming, we need to shake it up as soon as possible!

Doh.. I thought it s end of problems.
Seems ACPI stuff only reduced unexpected shutdowns but didn t eliminate them. Crap :/

GPL enforcement is a surprisingly difficult task. It's not just a matter of identifying an infringement - you need to make sure you have a copyright holder on your side, spend some money sending letters asking people to come into compliance, spend more money initiating a suit, spend even more money encouraging people to settle, spend yet more money actually taking them to court and then maybe, at the end, you have some source code. One of the (tiny) number of groups involved in doing this is the Software Freedom Conservancy, a non-profit organisation that offers various services to free software projects. One of their notable activities is enforcing the license of Busybox, a GPLed multi-purpose application that's used in many embedded Linux environments. And this is where things get interesting

GPLv2 (the license covering the relevant code) contains the following as part of section 4:

Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License.

There's some argument over what this means, precisely, but GPLv3 adds the following paragraph:

However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation

which tends to support the assertion that, under V2, once the license is terminated you've lost it forever. That gives the SFC a lever. If a vendor is shipping products using Busybox, and is found to be in violation, this interpretation of GPLv2 means that they have no license to ship Busybox again until the copyright holders (or their agents) grant them another. This is a bit of a problem if your entire stock consists of devices running Busybox. The SFC will grant a new license, but on one condition - not only must you provide the source code to Busybox, you must provide the source code to all other works on the device that require source distribution.

The outcome of this is that we've gained access to large bodies of source code that would otherwise have been kept by companies. The SFC have successfully used Busybox to force the source release of many vendor kernels, ensuring that users have the freedoms that the copyright holders granted to them. Everybody wins, with the exception of the violators. And it seems that they're unenthusiastic about that.

A couple of weeks ago, this page appeared on the elinux.org wiki. It's written by an engineer at Sony, and it's calling for contributions to rewriting Busybox. This would be entirely reasonable if it were for technical reasons, but it's not - it's explicitly stated that companies are afraid that Busybox copyright holders may force them to comply with the licenses of software they ship. If you ship this Busybox replacement instead of the original Busybox you'll be safe from the SFC. You'll be able to violate licenses with impunity.

What can we do? The real problem here is that the SFC's reliance on Busybox means that they're only able to target infringers who use that Busybox code. No significant kernel copyright holders have so far offered to allow the SFC to enforce their copyrights, with the result that enforcement action will grind to a halt as vendors move over to this Busybox replacement. So, if you hold copyright over any part of the Linux kernel, I'd urge you to get in touch with them. The alternative is a strangely ironic world where Sony are simultaneously funding lobbying for copyright enforcement against individuals and tools to help large corporations infringe at will. I'm not enthusiastic about that.

comments

When you read this, GNOME Shell 3.2 will (hopefully!) have finally transitioned to Debian s testing suite. Planet GNOME readers might think Debian now has outdated versions of software even in their development versions, or the distribution s development marches at glacial pace. Wheezy GNOME users will finally have a Shell that matches the rest of their GNOME components, something that works with the Shell extensions website and much less problems and limitations compared to 3.0.2. The reality is that GNOME 3.2 s packaging was quite ready back when it was released in late September, but a number of not-so-desirable situations held GNOME Shell from transitioning to testing until today, four months later. So, what happened? TL;DR: transitioning from GNOME 2 GNOME 3 is not so easy if you want to keep testing in a sane state, and when you need to deal with dozens of indirectly related packages, for more than 10 architectures but it shouldn t take nearly a full year, either Let s go back to the last months of 2010. Debian squeeze is in very deep freeze, and the release team and many Debian developers are focusing on squashing as many release critical bugs as they can, in order to make Debian 6.0 the great release it ended up being. The GNOME project has recently delayed the big launch of GNOME 3.0 again, until March 2011; Debian has already settled on GNOME 2.28 for its release, although it will end up cherry-picking many updates from the 2.30 release modules. With most of the stabilization work being done, many Debian GNOME team members were at that time working on packaging very early versions of what would end up being GNOME 3.0 technology: GTK+3.0, GNOME Shell, Mutter and some brave users even tried to use it via the experimental archive. On February 6th, Debian 6.0 was released, and soon after, on April 6th, GNOME made a huge step forward with the much anticipated release of GNOME 3.0. At that time, Debian developers were busy breaking unstable as much as they could, as it s tradition on the weeks following a major release, and the Debian GNOME team was able to start moving some GNOME 3.0 libraries (those which were parallel-installable with their GTK+2.0 versions) to unstable. However, moving the bulk of GNOME 3.0 to unstable wasn t so easy. When you start doing that, you need to be sure you re ready to have all affected packages in a transitionable state as soon as possible, to minimise the chances of blocking transitions of unrelated packages via the dependencies they pick up with rebuilds. All the packages involved in a transition need to be ready to go in the same testing run , for all supported architectures. When you re dealing with dozens of GNOME source packages at the same time, many of which introduce new libraries, or worse, introduce incompatible APIs that affect many more unrelated packages, things get hairy, and you need a plan. So, Joss outlined what a sane approach to this monster transition could look like. The amount of work to do was what we call fun on #debian-gnome. In a nutshell, we had to deal with quite a few transitions, starting with having a newer version of libnotify in unstable, and a pre-requisite for that was making sure all the packages using libnotify1 were ready to use the source-incompatible libnotify4, and this meant preparing patches and NMUs for many of our packages, as well as many others not under our control. Before starting a controlled transition like this one, we had to get an ACK from the release team, who was busy enough handling other huge transitions like Perl 5.12, so by the time we got our own slot, we were well into Summer. With libnotify done in August, it was time to get our hands dirty with more exciting stuff, like getting Nautilus in testing. This meant bumping a soname and requiring all packages providing Nautilus extensions to migrate to GTK+3.0, or drop the extension entirely, as you can t mix GTK+2.0 and GTK+3.0 symbols in the same process. However, in GNOME 3.0, automounting code had moved from Nautilus to gnome-settings-daemon, so in order to not break filesystem automounting in testing for an unreasonable amount of time, both Nautilus and g-s-d needed to go in at the same time. The fun thing is that g-s-d dragged glib2.0, gvfs, gnome-control-center, gdm3, gnome-media, gnome-session and gnome-panel into the equation, so this transition needed extra planning and a lot more work than initially expected: migrating all nautilus extensions, plus ensuring all Panel applets had migrated to GTK+3.0 and the new libpanel-applet-4 interface. In short, this was the monster transition we were trying to avoid. By the time all this mess was sorted out, GNOME 3.2 had been released, and for what users said, it was a lot better than 3.0. We still had no more than a few bits and pieces of 3.0 in testing, and we were working hard to get 3.0 in wheezy. With all the excitement around 3.2, at times it was difficult to explain outsiders why we were beating a dead 3.0 horse Going back to our huge transition, it was just a matter of time before all the packages would be built and be ready to enter, on the same run, in testing. A few weeks later, in early November and after several rounds of mass-bug-filings, fixing unrelated FTBFS, many NMUs, package removal requests and dealing with any possible problem that could block our transition, everything seemed to be set, and our release team magicians had everything in place for the big magic to happen. However, our first clash with the rest of Debian happened a few hours before our victory, in the form of an unannounced ruby-gnome2 upload which resetted the count for everyone. It was fun to see the release team trying all sorts of black magic in an attempt to mitigate the damage. Fortunately, after a few tries they managed to fool britney (the script that handles package transitions from unstable to testing) somehow, and the hardest part of the job was done with just one day of delay. At last, the core of GNOME 3 was in testing, and testing users found soon after. The rest of the week saw a cascade of hate posts against GNOME 3 in Planet Debian, and personally I didn t find that especially motivating to keep on working on the rest of GNOME bits. With experimental clear of GNOME 3.0 stuff, we finally were able to focus on packaging whatever GNOME 3.2 components were not already done, and preparing for what should be a plain simple transition of GNOME 3.0 to 3.2. After our share of wait for a transition slot, as Perl 5.14, ICU and OpenSSL were in the line before us, and after dealing with a minor tracker 0.12 transition, we were ready for our next episode: evolution-data-server. At first sight, we thought this would be a lot easier, but it still got a bit hairy due to evo-data-server massive soname bumps. We were given our slot just before Christmas, after a few weeks of wait for others to finish their migration rounds, and most of the pack entered wheezy a few days before the new year. No rejoicing, though, as GNOME Shell 3.2 didn t make it. First, we discovered it was FTBFS on kFreeBSD architectures, as NetworkManager had been promoted from optional to required, for apparently no good reason, leaving the BSD world in the cold, including our exotic GNU/kFreeBSD architectures. Now, let s clarify that I m a supporter of the Debian kFreeBSD architectures and was really happy to see it accepted as a technology preview in squeeze. However, as you know, GNOME Shell currently requires hardware acceleration to run, a requirement hardly met in kFreeBSD, unless you re using a DRI1 X driver. We seriously doubted anyone had ever ran a GNOME 3 session on kfreebsd-*. However, if it didn t build, it was a blocker bug for GNOME Shell. We considered creating different meta-packages for kFreeBSD architectures, to conclude it d be a mess, so our awesome Michael Biebl ended up cooking up a patch that restored the ability to build the Shell without NetworkManager support. With this out of our way, we just needed to upload Michael s fix and watch the buildds do their part of the job. Or maybe not? Enter Iceweasel 9. In parallel, and with incredible bad timing, Iceweasel 9.0 was uploaded to Debian the very same day it was released by Mozilla. Again, it greeted us with a nasty surprise: yet another mozjs API change, which made gjs FTBFS, which meant our kFreeBSD fixes would be unusable until someone who knew Gjs internals well enough bit the bullet and worked around the new API changes. Again, Michael Biebl tried to be our saviour, but unfortunately wasn t able to fix all the problems, so we tried to focus on plan B. Mozilla had released a fork of the mozjs that is included in Firefox, so that embedders would have a bit less of a hard time with these recurrent API changes. This was based on Firefox 4, and was already being packaged by Ubuntu. Gjs would build using this older version just fine, so we just needed to get it in Debian as soon as possible. We just needed to find a sucke^Wvolunteer that would be inclined to maintain the beast. Only after a few weeks we managed to get Chris Coulson, the Ubuntu packager, to maintain the package directly through the Debian archive via package syncs. However, his package had only been auto-compiled in the three Ubuntu architectures, that is amd64, armel and i386. It s late January 2012, and we ve been fighting this war for 10 months. After getting some help from Michael to get the new package in shape for Debian standards, we were excited to sponsor it for Chris. Duh, after a few days in the NEW fridge, it was rejected by the ftp-masters. The license statement was missing quite a few details, so I went ahead and sacrificed a few hours of my copious free time to get this sorted out. A few days later, mozjs was accepted, but the result was horrible. It was very red. mozjs didn t build on half of our targets. Mike Hommey was quick to file a bug and point us to the most obvious fuckups. As he had dealt with this in the past as the Iceweasel maintainer, all of these issues were fixed and patches were ready to be applied verbatim or with minimal changes to our sources. With mozjs finally built successfully (although with severe problems on ia64), we were finally able to rebuild Gjs against it, upload GNOME Shell with our kFreeBSD fixes and wait until today for this mess to be over. Whew. I can t say I ve enjoyed all the stages of this ride. Some bumps on the road were clearly there to test our patience, but it has helped me get back in touch with non-leaf GNOME packaging, which was all I was doing for a while due to being super-busy lately with studies. It also reminds me of the privilege of working side by side with some awesome people, not only Joss, Michael, Sjoerd, Laurent or Gustavo, to name just a few Debian GNOME team members, but also the receptive release team members like Julien or Cyril, and NEW-processing record-breaking ftp-master Luca. Without them, we might be trying to figure out the Nautilus transition since last Summer. We really hope GNOME 3.4 will be a piece of cake compared to this. ;)