About 10 months ago, we enabled an auto-decrufter in dak. Then after 3 months it had become the top 11th remover . Today, there are only 3 humans left that have removed more packages than the auto-decrufter impressively enough, one of them is not even an active FTP-master (anymore). The current score board:

 5371 Luca Falavigna
 5121 Alexander Reichle-Schmehl
 4401 Ansgar Burchardt
 3928 DAK's auto-decrufter
 3257 Scott Kitterman
 2225 Joerg Jaspert
 1983 James Troup
 1793 Torsten Werner
 1025 Jeroen van Wolffelaar
  763 Ryan Murray

For comparison, here is the number removals by year for the past 6 years:

 5103 2011
 2765 2012
 3342 2013
 3394 2014
 3766 2015  (1842 removed by auto-decrufter)
 2845 2016  (2086 removed by auto-decrufter)

Which tells us that in 2015, the FTP masters and the decrufter performed on average over 10 removals a day. And by the looks of it, 2016 will surpass that. Of course, the auto-decrufter has a tendency to increase the number of removed items since it is an advocate of remove early, remove often! . Data is from https://ftp-master.debian.org/removals-full.txt. Scoreboard computed as:

  grep ftpmaster: removals-full.txt   \
   perl -pe 's/.*ftpmaster:\s+//; s/\]$//;'   \
   sort   uniq -c   sort --numeric --reverse   head -n10

Removals by year computed as:

 grep ftpmaster: removals-full.txt   \
   perl -pe 's/.* (\d 4 ) \d 2 :\d 2 :\d 2 .*/$1/'   uniq -c   tail -n6

(yes, both could be done with fewer commands)
Filed under: Debian

You might not know who Peter is because he s not very visible on Debian mailing lists. He s very active however and in particular on IRC. He was an admin of the OFTC IRC network at the time Debian switched from Freenode to OFTC. Nowadays he s a member of the Debian System Administration team who runs all the debian.org servers. If you went to a Debconf you probably met him since he s always looking for new signatures of his GPG key. He owns the best connected key in the PGP web of trust. He also wrote caff a popular GPG key signing tool. Raphael: Who are you? Peter: I m Peter Palfrader, also known as weasel. I m in my early 30s, born and raised in Innsbruck, Austria and am now living and working in Salzburg, Austria. In my copious free time, other than help running Debian s servers I also help maintaining the Tor project s infrastructure. Away from the computer I enjoy reading fiction (mostly English language Science Fiction and Fantasy), playing board games and going to the movies. Weather permitting, I also occasionally do some cycling. Raphael: How did you start contributing to Debian? Peter: I installed my first Debian the week slink came out. That was Debian 2.1 for the youngsters, in early 1999. The one thing I immediately liked about slink was that Debian s pppd supported RAS authentication which my university s dial-up system required. No way I d go back to SuSE 5.3 when I had working Internet with my Debian box. During that year I started getting involved in the German language Debian channel on IRCnet which got me in contact with some DDs. Christian Kurz (<shorty>) was working on Debian QA at the time and he asked my help in writing a couple of scripts. Some of that work, debcheck, still produces parts of the qa.d.o website, tho the relevance of that nowadays is probably negligible. While trying to learn more Perl earlier, I had written a program to produce syntax highlighted HTML for code snippets in various languages. I didn t really know what I was doing but it kinda worked, and probably still does since I still get mail from users every now and then. I figured that it would be really nice if people could just get my software together with Debian. According to code2html s Debian changelog the initial release of the package was done on a weekday at 2:30 in the morning early in 2000, and if my memory serves me correctly, shorty uploaded it shortly afterwards. I started packaging a couple of other piece of software and in the same year I sent my mail to the debian account managers to register my intent to become a DD. No new developers where being accepted at that time since the DAMs wanted to overhaul the entire process so I wasn t surprised to not get any immediate reply. Of course what the silence also meant was that the mail had been lost, but I only learned of that later when I took all my courage to ask DAM about the status of application a couple months later. Once that was sorted out I was assigned an AM, did the usual dance, and got my account late in November 2000. Raphael: Four years ago, the Debian System Administration team was a real bottleneck for the project and personal conflicts made it almost impossible to find solutions. You were eager to help and at some point you got dropped as a new member in that team. Can you share your story and how you managed the transition in the difficult climate at that time? Peter: Ah, that was quite the surprise for an awful lot of people, me included. Branden Robinson, who was our DPL for the 2005-2006 term, tried to get some new blood added to DSA who were at the time quite divided. He briefly talked to me on IRC some time in summer 2005, telling me I had come recommended for a role on the sysadmin team . In the course of these 15 minutes he outlined some of the issues he thought a new member of DSA would face and asked me if I thought I could help. My reply was cautiously positive, saying that I didn t want to step on anybody s toes but maybe I could be of some assistance. And that was the first and last of it, until some fine November day two years later I got an email from Phil Hands saying I ve just added you to the adm group, and added you to the debian-admin@d.o alias. and welcome on board . *blink* What!? My teammates at the time were James Troup (elmo), Phil Hands (fil), Martin Joey Schulze and Ryan Murray (neuro). The old team, while apparently not on good terms with one another, was however still around to do heavy lifting when required. I still remember when on my first or second day on the team two disks failed in the raid5 of ftp-master.debian.org aka ries. Neuro did the reinstall once new disks had arrived at Brown University. I m sure I d have been way out of my league had this job fallen to me. Fortunately my teammates were all willing and able to help me find whatever pieces of information existed that might help me learn how debian.org does its stuff. Unfortunately a lot of it only existed in various heads, or when lucky, in one of the huge mbox archives of the debian-admin alias or list. Anyway, soon I was able to get my hands dirty with upgrading from sarge to etch, which had been released about half a year earlier. Raphael: I know the DSA team has accomplished a lot over the last few years. Can you share some interesting figures? Peter: Indeed we have accomplished a lot. In my opinion the most important of these accomplishment is that we re actually once again a team nowadays. A team where people talk to one another and where nobody should be a SPoF. Since this year s debconf we are six people in the admin team: Tollef Fog Heen (Mithrandir) and Faidon Liambotis (paravoid) joined the existing members: Luca Filipozzi, Stephen Gran, Martin Zobel-Helas, and myself. Growing a core team, especially one where membership comes with uid0 on all machines, is not easy and that s why I m very glad we managed to actually do this step. I also think the infrastructure and our workflows have matured well over the last four years. We now have essential monitoring as a matter of course: Nagios not only checks whether all daemons that should be running are in fact running, but it also monitors hardware health of disks, fans, etc. where possible. We are alerted of outstanding security updates that need to be installed and of changes made to our systems that weren t then explicitly acked by one of us. We have set up a centralized configuration system, puppet, for some of our configuration that is the same, or at least similar, on all our machines. Most, if not all, pieces of software, scripts and helpers that we use on debian.org infrastructure is in publicly accessible git repositories. We have good communication with other teams in Debian that need our support, like the ftp folks or the buildd people. As for figures, I don t think there s anything spectacular. As of the time of our BoF at this year s DebConf, we take care of approximately 135 systems, about 100 of them being real iron, the other virtual machines (KVM). They are hosted at over 30 different locations, tho we are trying to cut down on that number, but that s a long and difficult process. We don t really collect a lot of other figures like web hits on www.debian.org or downloads from the ftp archive. The web team might do the former and the latter is pretty much impossible due to the distributed nature of our mirrors, as you well know. Raphael: The DSA team has a policy of eating its own dog food, i.e. you re trying to rely only on what s available in Debian. How does that work out and what are the remaining gaps? Peter: Mostly Debian, the OS, just meets our needs. Sure, the update frequency is a bit high, we probably wouldn t mind a longer release cycle. But on the other hand most software is recent enough. And when it s not, that s easy to fix with backports. If they aren t on backports.debian.org already, we ll just put them there (or ask somebody else to prepare a backport for us) and so everybody else benefits from that work too. Some things we need just don t, and probably won t, exist in Debian. These are mainly proprietary hardware health checks like HP s tools for their servers, or various vendors programs to query their raid controller. HP actually makes packages for their stuff which is very nice, but other things we just put into /usr/local, or if we really need it on a number of machines, package ourselves. The push to cripple our installers and kernels by removing firmware was quite annoying, since it made installing from the official media next to impossible in some cases. Support for working around these limitations has improved with squeeze so that s probably ok now. One of the other problems is that especially on embedded platforms most of the buildd work happens on some variation of development boards, usually due to increased memory and hard disk requirements than the intended market audience. This often implies that the kernel shipped with Debian won t be usable on our own debian.org machines. This makes keeping up with security and other kernel fixes way more error prone and time intensive. We keep annoying the right people in Debian to add kernel flavors that actually boot on our machines, and things are getting better, so maybe in the future this will no longer be a problem. Raphael: If you could spend all your time on Debian, what would you work on? Peter: One of the things that I think is a bit annoying for admins that maintain machines all over the globe is mirror selection. I shouldn t have to care where my packages come from, apt-get should just fetch them from a mirror, any mirror, that is close by, fast and recent. I don t need to know which one it was. We have deployed geodns for security.debian.org a while ago, and it seems to work quite well for the coarse granularity we desired for that setup, but geodns is an ugly hack (I think it is a layer violation), it might not scale to hundreds or thousands of mirrors, and it doesn t play well with DNSSEC. What I d really like to see is Debian support apt s mirror method that I think (and I apologize if I m wronging somebody) Michael Vogt implemented recently. The basic idea is that you simply add deb mirror://mirror.debian.org/ or something like that to your sources.list, and apt goes and asks that server for a list of mirrors it should use right now. The client code exists, but I don t know how well tested it is. What is missing is the server part. One that gives clients a mirror, or list of mirrors, that are close to them, current, and carry their architecture. It s probably not a huge amount of work, but at the same time it s also not entirely trivial. If I had more time on my hands this is something that I d try to do. Hopefully somebody will pick it up. Raphael: What motivates you to continue to contribute year after year? Peter: It s fun, mostly. Sure, there are things that need to be done regularly that are boring or become so after a while, but as a sysadmin you tend to do things once or twice and then seek to automate it. DSA s users, i.e. DDs, constantly want to play with new services or approaches to make Debian better and often they need our support or help in their endeavors. So that s a constant flow of interesting challenges. Another reason is that Debian is simply where some of my friends are. Working on Debian with them is interacting with friends. I not only use Debian at debian.org. I use it at work, I use it on my own machines, on the servers of the Tor project. When I was with OFTC Debian is what we put on our machines. Being a part of Debian is one way to ensure what Debian releases is actually usable to me, professionally and with other projects. Raphael: Is there someone in Debian that you admire for their contributions? Peter: That s a hard one. There are certainly people who I respect greatly for their technical or other contributions to Debian, but I don t want to single anybody out in particular. I think we all, everyone who ever contributed to Debian with code, support or a bug report, can be very proud of what we are producing one of the best operating systems out there.

---

Thank you to Peter for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Seriously. Now that we have been using the request tracker for quite some time, it’s even more obvious that the DSA team is not up to its task. Use login “guest” and password “readonly” if you want to check the RT tickets linked in this article. The facts

• 65 public tickets open (and 5 private tickets)
  • Want to help and seek input from DSA? No response.
  • Want to help and you provide patches and/or solutions? Many are still waiting.
  • Security updates? Who cares…
  • Backup of ftp-master (ries)? No reply (though ftp-master stuff is mirrored on merkel)…
  • New alpha machine? No movement despite the fact that we still have no DD-accessible porter machine.
• 68 tickets closed. Here are some unscientific and manual stats (I counted one each time that someone was involved for the work or for closing the ticket):
  • 27 for James Troup (elmo)
  • 26 for Phil Hands (fil)
  • 13 for me (buxy)
  • 3 for Martin ‘Joey’ Schulze (those I manually forwarded him)
  • 3 for Ryan Murray (neuro)
  • 3 for Matt Taggart (taggart)
  • 2 for Josip Rodin (he handles tickets concerning the mirrors until they have a dedicated queue in the RT)

Note that myself and Matt do not have the needed rights to fix most of the tickets, so we provided help on a best-effort basis. Otherwise we would have done more. The communication problem It’s a multi-level problem. Each of the members has some problems with one or more other members. Joey’s behavior has been part of the recurring problems mentioned: he doesn’t use the RT, doesn’t read the DSA email alias and doesn’t follow the DSA IRC channel but he still does stuff very regularly without reporting anything and obviously problems happen. Ryan and James tried to impose him a rule to document what he does, without success apparently. On the other side, as far as I know, Ryan and James also don’t impose themselves to document everything in a central changelog. Joey has refused to provide me an explanation for his behavior. He just reminded me that he holds grudges against James and Ryan because as ftpmasters they didn’t cooperate well with him while he was stable release manager. In general, outside of all personal griefs that they might have, the DSA members do not communicate very much (at least not on their own official channels). Some examples have already been given concerning the request tracker, but it’s not much more effective on IRC. Most of the traffic on the channel is made up by local admins fixing the problems themselves without any intervention by any DSA. I also use the channel to regularly ping some DSA about simple issues and/or stuff that they usually handle. It used to work somewhat but lately fil has been busy (with the kernel summit and other conferences) and I simply got no answer at all… for example I pinged elmo, neuro and fil several times in the last weeks in the hope that they handle the tickets of the security team (#150, #157, #164) without results. There’s room for improvement. The leadership problem The team has no designated leader and every time that there’s a decision to take, they are blocked. Joey wouldn’t communicate and give his opinion, Ryan is extremely requiring and perfectionist, there’s not much room for compromise… A long time ago in a galaxy far, far away, Joey and elmo were friends. It’s even Joey who gave root rights to elmo. Nowadays, it’s rather James that is sort-of leading the team but he’s fed up of the situation and hasn’t managed to get out of this mess. He refuses to take drastic measures by himself because he’s not clearly the leader and doesn’t solicit a decision of the Debian leader (or the project) because he believes that the DSA team is not under the scope of the constitution! This can’t last any further. We’ll have to do something about it. Stay tuned.

I'm currently sitting in Denver International awaiting a flight to Houston where I'll meet up with Taggart for the flight into Mexico. There we'll meet up with some other folks (Ryan Murray, Troy Heber & Junichi Uekawa at least) and take a cab down to Oaxtepec. We've had *tremendous* trouble booking our hotel room (and by we, I mean Troy). In fact, last we heard they were cancelling our reservation. I strongly suspect this is an attempt at bribe soliciation, or maybe they actually just don't want to do business with us? Not quite sure - my employer is sponsoring my travel here, and I haven't noticed a "bribe" category on the expense report.. maybe I can list it under tip? :) Oh well, I suspect the worst case is that we can find asylum for a night on someone's floor until our accomodations get worked out. I'm very much looking forward to arriving at DebConf & seeing everyone, not to mention trying the local cuisine :)

I wish I could say that the subject says it all, but that's not quite true.

First off: it's true that X.Org has been uploaded to unstable. Steve Langasek talked with the gcc maintainers and they decided that X.Org wouldn't block gcc4, so I got the approval to upload. Steve, as ever, is awesome.

Then Daniel Stone noticed that the packages are missing updated MANIFEST files for many of the ports. These files were put in place in the X build system long ago so as to manage to complexity of new upstream revisions. Because there's just so much stuff in the X source tree, there had to be an automated way to deal with making sure everything was present and accounted for. The end result of this is that if a MANIFEST that we have doesn't match up with the one generated while building the package, the build will fail. So what's going to happen is that with this upload, some architectures will get the X.Org packages, and others will not due to build failures. I uploaded i386 packages, so that will definitely go through right away, and ppc, sparc, amd64, and I believe mips should all go through as well.

The reason we went with this plan is that when the builds fail, the build logs will give us a diff for the MANIFEST for each arch. So what I can do is use those to fix the MANIFEST files tomorrow, rebuild as fast as my laptop can manage, and upload the -2 revision which has correct MANIFEST files. Steve, Daniel, and Ryan Murray all directed me in deciding on this plan, and I owe them a ton for all their help.

Already we've hit a build error in s390 that we're working through, but hopefully we can get all the ports in line ASAP so that this can transition to etch. But here it is. After a long wait and lots of work by lots of people, X.Org is finally entering Debian.

I think I need a glass of scotch.

My good buddy Ryan Murray turned me on to the Quod Libet audio player. So far I like it. I never have liked library managers before but I don't find myself fighting with this one. Maybe using iTunes has made it more comfortable. The best thing i like is that I can use Audioscrobbler on my m4a's. XMMS had problems with that and now Last.fm shows more accurate information for what I listened to in the past week.

Over the last few weeks, I've implemented some new features in DDPO. The system, originally written by Igor Genibel, is a mix of Python, Perl, and PHP generated from WML, so it's quite interesting to see how these languages interact. The main new features are the ability to add arbitrary packages to the list displayed, and an automatic listing of all NMUs and sponsored uploads in the new "uploads" section. (Thanks to Ryan Murray, Joerg Jaspert, and Joey Schulze for helping fix the projectb for that!) I won't repeat the details here, read the d-d-a posting for that. Another nice thing is the link to Ian Lynagh's popcon graphs which I had mostly ignored so far. Have a look at my DDPO page to see the new features - feedback welcome!