podlators is the Perl distribution providing Pod::Man and Pod::Text, along with related modules and supporting scripts. The primary change in this release is the addition of configurable guesswork for Pod::Text, paralleling Pod::Man. I had forgotten that Pod::Text also had complex heuristics for whether to quote C<> text that have the same Perl-specific properties as Pod::Man. This is now configurable via a guesswork option, the same as in Pod::Man, although the only type of guesswork supported is quoting. I also updated the default regexes, which include some fixes from Pod::Man. Thanks to discussion with G. Branden Robinson, I now understand quoting in roff considerably better, which let me fix a few obscure bugs with strange page titles or configured quote characters. Pod::Man now avoids quoting macro arguments when the quoting is unnecessary, which should hopefully produce slightly more readable output. Finally, I had started using a Pod::Simple feature introduced in 3.26 in Pod::Text but forgot to update the dependency, resulting in test failures on some old versions of Perl. (The same tests didn't fail in GitHub CI, which is probably related to how I install dependencies.) That's been fixed in this release. You can get the latest version from CPAN or from the podlators distribution page.

Russell Coker wrote about Sysadmin Skills and University Degrees. I couldn t agree more that a major deficiency in Computer Science degrees is the lack of sysadmin training. It seems like most sysadmins learned most of what they know from experience. It s very hard to recruit young engineers (freshly out of university) for sysadmin jobs, and the job interviews are often a bit depressing. Sysadmins jobs are also not very popular with this public, probably because university curriculums fail to emphasize what s exciting about those jobs. However, I think I disagree rather deeply with Russell s detailed analysis. First, Version Control. Well, I think that it s pretty well covered in university curriculums nowadays. From my point of view, teaching CS in Universit de Lorraine (France), mostly in Licence Professionnelle Administration de Syst mes, R seaux et Applications base de Logiciels Libres (warning: french), a BSc degree focusing on Linux systems administration, it s not usual to see student projects with a mandatory use of Git. And it doesn t seem to be a major problem for students (which always surprises me). However, I wouldn t rate Version Control as the most important thing that is required for a sysadmin. Similarly Dependencies and Backups are things that should be covered, but probably not as first class citizens. I think that there are several pillars in the typical sysadmin knowledge. First and foremost, sysadmins need a good understanding of the inner workings of an operating system. I sometimes feel that many Operating Systems Design courses are a bit too much focused on the Design side of things. Yes, it s useful to understand the low-level mechanisms, and be able to (mentally) recreate an OS from scratch. But it s also interesting to know how real systems are actually built, and what are the trade-off involved. I very much enjoyed reading Branden Gregg s Systems Performance: Enterprise and the Cloud because each chapter starts with a great overview of how things are in the real world, with a very good level of detail. Also, addressing OS design from the point of view of performance could be a way to turn those courses into something more attractive for students: many people like to measure, benchmark, optimize things, and it s quite easy to demonstrate how different designs, or different configurations, make a big difference in terms of performance in the context of OS design. It s possible to be a sysadmin and ignore, say, the existence of the VFS, but there s a large class of problems that you will never be able to solve. It can be a good trade-off for a curriculum (e.g. at the BSc level) to decide to ignore most of the low-level stuff, but it s important to be aware of it. Students also need to learn how to design a proper infrastructure (that meets requirements in terms of scalability, availability, security, and maybe elasticity). Yes, backups are important. But monitoring is, too. As well as high availability. In order to scale, it s important to be able to automatize stuff. Russell writes that Sysadmins need some programming skills, but that s mostly scripting and basic debugging. Well, when you design an infrastructure, or when you use configuration management tools such as Puppet, in some sense, you are programming, and in terms of needs to abstract things, it s actually similar to doing object-oriented programming, with similar choices (should I use that off-the-shelf puppet module, or re-develop my own? How should everything fit together?). Also, when debugging, it s often useful to be able to dig into code, understand what the developer was trying to do, and if the expected behavior actually matches what you are seeing. It often results in spending a lot of time to create a one-line fix, and it requires very advanced programming skills. Again, it s possible to be a sysadmin with only limited software development knowledge, but there s a large class of things that you are unlikely to address properly. I think that what makes sysadmins jobs both very interesting and very challenging is that they require a very wide range of knowledge. There s often the ability to learn about new stuff (much more than in software development jobs). Of course, the difficult question is where to draw the line. What is the sysadmin knowledge that every CS graduate should have, even in curriculums not targeting sysadmin jobs? What is the sysadmin knowledge for a sysadmin BSc degree? for a sysadmin MSc degree?

I've spent this weekend with my girlfriend in Berlin, which is one of the cities staying quite long on list of places to visit. Now I've been there, though there are still lot of reasons to come back in future. We stayed with great hosts found through CouchSurfing and have seen hopefully all important places in the city. As usual, you can find some pictures in my gallery:

Filed under: English Life Travelling 0 comments Flattr this!

...but I didn't break it myself. Still, Patrick Makau broke today Haile Gebresselassie's world record by 21 seconds, lowering it down to 2h03'38". This is an impressive performance, considering the quite "high" temporature today in Berlain (about 23 C). My own race was quite good, though not perfect. My goal was targeting 3h36', to break my own record (3h40'24"). However, my preparation was complicated by an hurted ankle in early August, then a very shaky September month at work with a lot of stress and work days up to 12 hours. Not the best way to prepare a marathon. Last two weeks were fairly good wrt running, with an half-marathon run in 1h43' two weeks ago. Still, we complicated this by speding a few days to do sightseeing in Berlin (you can't really be in such a nice city and not try to see it!). So, that means a lot of walking here or there, with museum visits and quite long days. Again, not the best way to prepare a marathon..:-) Still, my watch was set on a 5'07"/km pace. My goal was starting like this, work to avoid going faster over the first 20K ad see what happens. At the beginning, it worked fairly well. Contrary to Paris marathon, Berlin's start in Strasse des 17. Juni is very cool for the first 3 kilometers with an incredibly wide avenue, whilst Paris marathon shrinks in Rue de Rivoli after km 1.5, making it fairly complicated to pass. Also, runners in Germany are much more respectful of running principles and very few register for starting blocks that are nont their real performance (like registering for 3h30 while your best time if 4h). So less passing and a much more costant pace. Up to km15 in Kreuzbeg, everything went well, though the heat was growing, which already made me suspect that I would not make it (0-5K in 25'31", 5-10K in 25'54", 10-15K in 25'55") . From this point, I had to "fight" a bit more to keep the pace...and a few streets without shade had their effect. Moreover, I already had to refill my belt bottle after drinking 0,8l for the first 17KM. The pace thus went down to about 5'10-5'15" and I reached half-marathon in 1h50'28", already not really on time for less than 3h40). 15-20K in 26'41", including a bottle refill. Then 20-25 in 25'44", so small drop after all. However, the wall hit me on km28. For the first time in my marathon life, I think I considered abandoning because I was feeling without any energy and still 1/3 of a marathon to run. Well, that was only the 30K wall which they apparently moved 2K earlier..:-) After calming myself down, I decided to do what has to be done in such case: refill with energy. So long for the gels I had and particularly the hyper-glucidic "Coup de Fouet". Finally, around KM30, I found more motivation to complete the race. Anyway, we were then very very far away, at the end of Hohenzollendamm, so the only option was to come back anyway! Despite all this, I kept a very decent pace, after all: 26'45" for 25-30K. Each kilometer, my watch was beeping to remind me that I was not crawling on the street but more keeping about 5'20". Still, the last kilometers on these giants avenues (oh, endless Leipziger Strasse after Postdamer Platz!) needed a lot of motivation to be completed. It's indeed only at Gendarmenmarkt that I finally could be reliefed and know that I would not only finish, but also do it in a very decent time. 30-35K in 27'43", 35-40K in 27'38". Then we turned in Unter den Linden and, even better, at KM41, my beloved Elizabeth was there yelling. This, plus Brandenburder Tor not so far away....I could then "sprint" at 5'08" for the last kilometer. Finally, this marathon (my 6th official) was completed in 3h44'50", which is my second best. I'm really happy with this result, finally. OK, I didn't break my record as I was intending to (I think I can go down to 3h35 with a normal preparation and good conditions), but the way I managed the difficulty was finally good. So, now, let's prepare the next goal, a 67km night race in November, the longest distance I have ever run. Yet another different thing to prepare. And what about Berlin in all this? Well, in short, that city is *great*. What could I tell more?

It s been a while since I last blogged, and some things has happened. Jobs
I m out looking for a job. Preferably KDE/Qt or Debian related. Or at least opensource related. Feel free to contact me if you know of anything. sune AT vuorela DOT dk Desktop Summit I ve been at Desktop Summit, awesome as usual to see all these great people, except monday morning. I think intel is partly to blame here :-).

• I got to a set of exciting talks, including one about a toaster.
• I got to meet some nice people, both old and new
• I hacked a bit on libprison
• I walked along the Berlin Wall, sat and enjoyed Brandenburger Tor and other parts of Berlin
• Learned to feel home in Berlin public transportation

Prison and such
The upcoming release of Prison will at least have a couple of new features, beside a couple of bugfixes:

• It is now possible to set the colors to be used for the barcodes. That s going to be nice in some cases.
• I have a patch in my mailbox adding EAN13-support from a brand new contributor. It just needs a bit of polishing and then ready to go

I ve also started to enjoy -Werror=unused-but-set-variable in GCC and killed some of those in KDE land. end There is more to come later, hopefully things about the ExoPC I gotborrowed at Desktop Summit, which is now running Plasma Active & Contour. Things about a new librison release, and maybe more places to use libprison. Oh. And yes, I m out looking for a job.

You might not know who Peter is because he s not very visible on Debian mailing lists. He s very active however and in particular on IRC. He was an admin of the OFTC IRC network at the time Debian switched from Freenode to OFTC. Nowadays he s a member of the Debian System Administration team who runs all the debian.org servers. If you went to a Debconf you probably met him since he s always looking for new signatures of his GPG key. He owns the best connected key in the PGP web of trust. He also wrote caff a popular GPG key signing tool. Raphael: Who are you? Peter: I m Peter Palfrader, also known as weasel. I m in my early 30s, born and raised in Innsbruck, Austria and am now living and working in Salzburg, Austria. In my copious free time, other than help running Debian s servers I also help maintaining the Tor project s infrastructure. Away from the computer I enjoy reading fiction (mostly English language Science Fiction and Fantasy), playing board games and going to the movies. Weather permitting, I also occasionally do some cycling. Raphael: How did you start contributing to Debian? Peter: I installed my first Debian the week slink came out. That was Debian 2.1 for the youngsters, in early 1999. The one thing I immediately liked about slink was that Debian s pppd supported RAS authentication which my university s dial-up system required. No way I d go back to SuSE 5.3 when I had working Internet with my Debian box. During that year I started getting involved in the German language Debian channel on IRCnet which got me in contact with some DDs. Christian Kurz (<shorty>) was working on Debian QA at the time and he asked my help in writing a couple of scripts. Some of that work, debcheck, still produces parts of the qa.d.o website, tho the relevance of that nowadays is probably negligible. While trying to learn more Perl earlier, I had written a program to produce syntax highlighted HTML for code snippets in various languages. I didn t really know what I was doing but it kinda worked, and probably still does since I still get mail from users every now and then. I figured that it would be really nice if people could just get my software together with Debian. According to code2html s Debian changelog the initial release of the package was done on a weekday at 2:30 in the morning early in 2000, and if my memory serves me correctly, shorty uploaded it shortly afterwards. I started packaging a couple of other piece of software and in the same year I sent my mail to the debian account managers to register my intent to become a DD. No new developers where being accepted at that time since the DAMs wanted to overhaul the entire process so I wasn t surprised to not get any immediate reply. Of course what the silence also meant was that the mail had been lost, but I only learned of that later when I took all my courage to ask DAM about the status of application a couple months later. Once that was sorted out I was assigned an AM, did the usual dance, and got my account late in November 2000. Raphael: Four years ago, the Debian System Administration team was a real bottleneck for the project and personal conflicts made it almost impossible to find solutions. You were eager to help and at some point you got dropped as a new member in that team. Can you share your story and how you managed the transition in the difficult climate at that time? Peter: Ah, that was quite the surprise for an awful lot of people, me included. Branden Robinson, who was our DPL for the 2005-2006 term, tried to get some new blood added to DSA who were at the time quite divided. He briefly talked to me on IRC some time in summer 2005, telling me I had come recommended for a role on the sysadmin team . In the course of these 15 minutes he outlined some of the issues he thought a new member of DSA would face and asked me if I thought I could help. My reply was cautiously positive, saying that I didn t want to step on anybody s toes but maybe I could be of some assistance. And that was the first and last of it, until some fine November day two years later I got an email from Phil Hands saying I ve just added you to the adm group, and added you to the debian-admin@d.o alias. and welcome on board . *blink* What!? My teammates at the time were James Troup (elmo), Phil Hands (fil), Martin Joey Schulze and Ryan Murray (neuro). The old team, while apparently not on good terms with one another, was however still around to do heavy lifting when required. I still remember when on my first or second day on the team two disks failed in the raid5 of ftp-master.debian.org aka ries. Neuro did the reinstall once new disks had arrived at Brown University. I m sure I d have been way out of my league had this job fallen to me. Fortunately my teammates were all willing and able to help me find whatever pieces of information existed that might help me learn how debian.org does its stuff. Unfortunately a lot of it only existed in various heads, or when lucky, in one of the huge mbox archives of the debian-admin alias or list. Anyway, soon I was able to get my hands dirty with upgrading from sarge to etch, which had been released about half a year earlier. Raphael: I know the DSA team has accomplished a lot over the last few years. Can you share some interesting figures? Peter: Indeed we have accomplished a lot. In my opinion the most important of these accomplishment is that we re actually once again a team nowadays. A team where people talk to one another and where nobody should be a SPoF. Since this year s debconf we are six people in the admin team: Tollef Fog Heen (Mithrandir) and Faidon Liambotis (paravoid) joined the existing members: Luca Filipozzi, Stephen Gran, Martin Zobel-Helas, and myself. Growing a core team, especially one where membership comes with uid0 on all machines, is not easy and that s why I m very glad we managed to actually do this step. I also think the infrastructure and our workflows have matured well over the last four years. We now have essential monitoring as a matter of course: Nagios not only checks whether all daemons that should be running are in fact running, but it also monitors hardware health of disks, fans, etc. where possible. We are alerted of outstanding security updates that need to be installed and of changes made to our systems that weren t then explicitly acked by one of us. We have set up a centralized configuration system, puppet, for some of our configuration that is the same, or at least similar, on all our machines. Most, if not all, pieces of software, scripts and helpers that we use on debian.org infrastructure is in publicly accessible git repositories. We have good communication with other teams in Debian that need our support, like the ftp folks or the buildd people. As for figures, I don t think there s anything spectacular. As of the time of our BoF at this year s DebConf, we take care of approximately 135 systems, about 100 of them being real iron, the other virtual machines (KVM). They are hosted at over 30 different locations, tho we are trying to cut down on that number, but that s a long and difficult process. We don t really collect a lot of other figures like web hits on www.debian.org or downloads from the ftp archive. The web team might do the former and the latter is pretty much impossible due to the distributed nature of our mirrors, as you well know. Raphael: The DSA team has a policy of eating its own dog food, i.e. you re trying to rely only on what s available in Debian. How does that work out and what are the remaining gaps? Peter: Mostly Debian, the OS, just meets our needs. Sure, the update frequency is a bit high, we probably wouldn t mind a longer release cycle. But on the other hand most software is recent enough. And when it s not, that s easy to fix with backports. If they aren t on backports.debian.org already, we ll just put them there (or ask somebody else to prepare a backport for us) and so everybody else benefits from that work too. Some things we need just don t, and probably won t, exist in Debian. These are mainly proprietary hardware health checks like HP s tools for their servers, or various vendors programs to query their raid controller. HP actually makes packages for their stuff which is very nice, but other things we just put into /usr/local, or if we really need it on a number of machines, package ourselves. The push to cripple our installers and kernels by removing firmware was quite annoying, since it made installing from the official media next to impossible in some cases. Support for working around these limitations has improved with squeeze so that s probably ok now. One of the other problems is that especially on embedded platforms most of the buildd work happens on some variation of development boards, usually due to increased memory and hard disk requirements than the intended market audience. This often implies that the kernel shipped with Debian won t be usable on our own debian.org machines. This makes keeping up with security and other kernel fixes way more error prone and time intensive. We keep annoying the right people in Debian to add kernel flavors that actually boot on our machines, and things are getting better, so maybe in the future this will no longer be a problem. Raphael: If you could spend all your time on Debian, what would you work on? Peter: One of the things that I think is a bit annoying for admins that maintain machines all over the globe is mirror selection. I shouldn t have to care where my packages come from, apt-get should just fetch them from a mirror, any mirror, that is close by, fast and recent. I don t need to know which one it was. We have deployed geodns for security.debian.org a while ago, and it seems to work quite well for the coarse granularity we desired for that setup, but geodns is an ugly hack (I think it is a layer violation), it might not scale to hundreds or thousands of mirrors, and it doesn t play well with DNSSEC. What I d really like to see is Debian support apt s mirror method that I think (and I apologize if I m wronging somebody) Michael Vogt implemented recently. The basic idea is that you simply add deb mirror://mirror.debian.org/ or something like that to your sources.list, and apt goes and asks that server for a list of mirrors it should use right now. The client code exists, but I don t know how well tested it is. What is missing is the server part. One that gives clients a mirror, or list of mirrors, that are close to them, current, and carry their architecture. It s probably not a huge amount of work, but at the same time it s also not entirely trivial. If I had more time on my hands this is something that I d try to do. Hopefully somebody will pick it up. Raphael: What motivates you to continue to contribute year after year? Peter: It s fun, mostly. Sure, there are things that need to be done regularly that are boring or become so after a while, but as a sysadmin you tend to do things once or twice and then seek to automate it. DSA s users, i.e. DDs, constantly want to play with new services or approaches to make Debian better and often they need our support or help in their endeavors. So that s a constant flow of interesting challenges. Another reason is that Debian is simply where some of my friends are. Working on Debian with them is interacting with friends. I not only use Debian at debian.org. I use it at work, I use it on my own machines, on the servers of the Tor project. When I was with OFTC Debian is what we put on our machines. Being a part of Debian is one way to ensure what Debian releases is actually usable to me, professionally and with other projects. Raphael: Is there someone in Debian that you admire for their contributions? Peter: That s a hard one. There are certainly people who I respect greatly for their technical or other contributions to Debian, but I don t want to single anybody out in particular. I think we all, everyone who ever contributed to Debian with code, support or a bug report, can be very proud of what we are producing one of the best operating systems out there.

---

Thank you to Peter for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Out of curiosity, and because it is Sunday morning and I have a cold and can't get my brain to do anything tricky, I counted the number of candidates in each year's DPL elections.

Year	Count	Names
1999	4	Joseph Carter, Ben Collins, Wichert Akkerman, Richard Braakman
2000	4	Ben Collins, Wichert Akkerman, Joel Klecker, Matthew Vernon
2001	4	Branden Robinson, Anand Kumria, Ben Collins, Bdale Garbee
2002	3	Branden Robinson, Rapha l Hertzog, Bdale Garbee
2003	4	Moshe Zadka, Bdale Garbee, Branden Robinson, Martin Michlmayr
2004	3	Martin Michlmayr, Gergely Nagy, Branden Robinson
2005	6	Matthew Garrett, Andreas Schuldei, Angus Lees, Anthony Towns, Jonathan Walther, Branden Robinson
2006	7	Jeroen van Wolffelaar, Ari Pollak, Steve McIntyre, Anthony Towns, Andreas Schuldei, Jonathan (Ted) Walther, Bill Allombert
2007	8	Wouter Verhelst, Aigars Mahinovs, Gustavo Franco, Sam Hocevar, Steve McIntyre, Rapha l Hertzog, Anthony Towns, Simon Richter
2008	3	Marc Brockschmidt, Rapha l Hertzog, Steve McIntyre
2009	2	Stefano Zacchiroli, Steve McIntyre
2010	4	Stefano Zacchiroli, Wouter Verhelst, Charles Plessy, Margarita Manterola
2011	1	Stefano Zacchiroli (no vote yet)

Winner indicate by boldface. I expect Zack to win over "None Of The Above", so I went ahead and boldfaced him already, even if there has not been a vote for this year. Median number of candidates is 4.

Note: this post was written on March 16 and posted after our return home. Also, I took no photos in Berlin, reasoning that I could leave my camera at the hotel so as to not worry about it, since I m sure there are enough photos of the Brandenburg Gate in the world already. Photos on this story only are from others. We ve had a good time in Berlin it s been adventurous to be out on our own in an unfamiliar city with an unfamiliar language, but has gone well. I feel that I m finally getting used to it a bit, and now tomorrow we move on to Leipzig. Stepping off the train at the Berlin Hauptbahnhof, and seeing all those people connecting to all those trains, I had a feeling of excitement: here is a grand train station that is actively used by so many people. I got a sense of what the beautiful and enormous Kansas City Union Station once felt like, I think; that station at its peak served almost the same number of people in a day at Berlin Hbf does.
(photo by eliotc) Stepping outside in the cold and snow to wait for our bus, I got the first sense that Berlin felt a bit more like an American city than did L beck: a beggar with a suspicious story was working the crowd with Speak English? (We saw many dressed similarly using the exact same tactic during our stay in Berlin.) Our bus ride to the hotel showed us some graffiti a fact of life in many cities in the USA too. Our hotel, the Circus, most definitely did not feel American. The staff was very friendly (I think I ve only seen that level of friendliness and helpfulness at one hotel in the USA: the Portland Doubletree). The room was small (which we expected) but very nice. There was a fresh flower waiting, and a whole printout of information with my name on it waiting in the room: info about the hotel, restaurant, and a multi-page history of this part of the city. They are very energy-conscious there. The hall lights automatically turn off, but you can touch your (apparently RFID-enabled) room card to any switch to turn them on for a few minutes. When in your room, you put your key card in a little holder that keeps it safe and enables the use of the lights. They don t put shampoo in the rooms, but have a selection free for the taking at the front desk: the rationale being that it generates waste when they have to replace it for people that don t need it replaced. They are very environmentally conscious with everything except the showerhead, which appears to use so much water that it would be illegal in the USA. Here s the obligatory surprising to an American comment about Germany: the complete lack of water drinking fountains. In the US, you can get a drink of water at any building of any size airports, train stations, museums, shopping centers, and also in many public places such as parks. I may have seen exactly one water fountain in the Hamburg airport, but that was it. It is odd given the general sense of environmentalism here that so much energy is being wasted on bottling water, not to mention the expense of having to pay for it all over the place. The bus ride to the hotel was interesting. I needed to buy two tickets from the bus driver. I didn t know the German word for ticket, so I just guessed and went with Zwei Ticket, bitte. Wohin? Rosenthaler Str. Then the price came up. I believe that was my first completely successful German-only conversation. (Most Germans hear a couple of words of mine and quickly switch at least partially to English, which probably gets things done a lot faster, and saves me embarrassment, but doesn t give me much chance to practice my German.) We ate dinner Monday night at the Hackesche H fe at Weihenstephaner, a Bavarian restaurant. It was a fun meal, and the tables were long and seated multiple parties. I tried out their special Bavarian beer, and of course had some Wienerschnitzel. I had noticed apple strudel on their website but not on their menu, so when it was time for dessert, I asked our waitress if they had apple strudel today. They did, and it was delicious. Tuesday began with a walk to the Pergamon Museum. From our hotel, this was a walk of about a mile. We had been introduced to excellent German bakeries during our time in L beck. So, since we hadn t had breakfast, when we spotted a bakery along the road, we went in. Terah got a croissant with chocolate on top, which turned out to also have chocolate inside. I found a couple of smaller rolls with various seeds and flavorings on top. All were excellent, and I believe we spent less than EUR 2 all together. We ate our breakfast of rolls as we walked towards the museum. It was a cold and somewhat windy morning, but it was also fun and exciting to be there. The Pergamon was quite the experience. The Pergamon Altar was the first large artifact we saw, and was particularly interesting given that I have recently read The Iliad and The Odyssey.
(photo by *hoodrat*) The Market Gate of Miletus was also impressive, but the true highlight has to be the Ishtar Gate and processional way. Wow. It was built in 575BC by King Nebuchadnezzar II, and importantly was built out of glazed tiles, so the original artwork, color and all, still survives. You can walk through the processional way and get the feeling of kings and armies proceeding there. Truly spectacular.
(photo by Rictor Norton & David Allen) After the Pergamon, we crossed a bridge to the east taking us off Museumsinsel (Museum Island). We walked down a quiet and somewhat forgotten back street and found a small restaurant for lunch. I had the Berlin Wurst , some excellent sausage with excellent and not-so-sour sauerkraut. Terah had some breakfast-type items with fruit. After that, we walked over to the famous Unter den Linden. It wasn t yet spring, so the scenery wasn t all that spectacular yet, but that and the light snow didn t stop the tourists, or the tacky tourist shops, which appear to be a universal global feature. ( Berlin flip-flops and t-shirts made in China, anyone?) The destination of all this walking was the Brandenburger Tor, the famous Berlin icon. The Brandenburg Gate was immense, and the sense of history of standing there was impressive.
(photo by Andrew Mason) We of course walked through the Brandenburg Gate on our way to the Bundestag, which we intended to tour. We didn t, though, due to the cold and very long line snaking out of the building. We then got on the train to Checkpoint Charlie. We saw the outdoor checkpoint still in the middle of the street. We intended to visit the museum Haus am Checkpoint Charlie, but was so packed that it was difficult to even get in the door. We stood in line to buy tickets for a few minutes, but made absolutely no progress; meanwhile, it appeared that an entire school group came in after us somehow and was also waiting to pay. So we decided to go see Schloss Charlottenburg instead.
(photo by Poom!) The Schloss (palace) was a baroque design, intended to impress visitors with the power and wealth of the owners. Only the old wing was open Tuesday, but we had about an hour and a half available to tour it: enough to see the whole thing, but not enough to linger and read all of the information. It was impressive and interesting. Particularly interesting was the chapel, which was this odd state/religion combination decorated both with sacred symbols as well as symbols of the king. After Charlottenburg, we had dinner at the Prussian Restaurant Marjellchen near the Savignyplatz S-bahn. We started with a smoked herring appetizer. Then I had the K nigsberg meatballs, and Terah had creamed ham. Both of us really enjoyed our meals, and Terah s reminded her of Verenike. Wednesday started with an early rise, then we went to Mauerpark to see a remnant of the Berlin Wall. I am glad to have seen it, and was a bit surprised with how small it was at that point compared to my expectations.
(photo by Eichental) Then we had a few minutes back in our hotel to munch on some rolls for breakfast; after that, it was off to the Hauptbahnhof for our train to Leipzig.

This talk at Manchester Free Software s meeting is covering the question:

From the literally hundreds of GNU/Linux distributions in existence, what makes Debian special?

For our co-op, it s that Debian GNU/Linux, like all GNU/Linux distributions, is the product of a massive cooperative effort (in the words of former Debian project leader Branden Robinson) and that debian is the best balance of a large project and a not-for-profit implementation of those values. It s not perfect, but it s pretty good. Manchester Free Software has put recordings of some past talks online and I hope this one will appear there. I m also looking forward to Richard Smedley s talk on 20th April 2010. If you re interested in the background to debian, go along to Manchester and hear some other views! If you re already using or developing debian, why does it keep you coming back?

On the train to my parents' house for Christmas I finished up a wild run through the book Coders at Work. The book isn't even mine, but I'd been borrowing it very often, sometimes to the chagrin of its owner, because I could barely put it down. The book is a collection of interviews with 15 great programmers of our time, starting with Jamie Zawinsky and ending with Donald Knuth. It's written in an interview style each interview starts with a brief introduction to the person being interviewed, summarizing what the person is known for and what he or she has accomplished and a few of the highlights of the interview, and then a transcript of the interview follows, with the author/editor, Peter Siebel, will saying something or asking a question, and the interviewee responding. I was skeptical about this format at first because I feel like it can be an easy way out of good editing and make the reader have to do the work of the editor, but on finishing I think that Siebel uses the format to his advantage in this case. For one, the speech format allows the reader to really form a picture of how the person being interviewed speaks and would act in a conversation. Jamie is somewhat bitter and pretty informal. Brad Fitzpatrick is flippant and energetic, his speech littered with profanity and colloquialisms. Others seem more stately and verbose Joe Armstrong's responses can go on for a page or more. In this way, not only do readers learn something about what these greats have learned about programming, but we also feel a bit more like we've met or know them, and can connect to them more as people. I always have this problem where I want to read computer books, but often computer books seem inextricably tied to the computer, so there's this dynamic of reading a bit and then wanting to get on a machine and try something out, write some code, play around especially with books focused on a specific language. Coders at Work retains some of this computer-book dynamic in that I constantly encountered things that I want to investigate or play around with more: Erlang, OCaml, various papers and essays, Knuth's literate programs, and books such as Higher Order Perl and others. Siebel makes a point to ask each person what her short-list of books and papers programmers should read are, so this book is a great source of pointers to other reading material. Unlike a more specific book, however, keeping a list in a notebook was enough to settle the mind to read away-from-a-computer for chapters at a time. It's obvious that despite the interview format, Siebel has done some serious editing. None of the prose is boring to read, and I can't imagine that the text is a straight transcript of how the interviews went. He also has arranged the interviews in an order such that different interviews play off each other. In Branden Eich's interview, for example, he disparages the book Design Patterns:

I never bought the Gamma book. Some people at Netscape did, some of Jamie Zawinski's and my nemeses from another acquisition, they waved it around like the Bible and they were kind of insufferable because they weren't the best programmers.

In the next chapter, Joshua Bloch names it as a book he thinks programmers should read:

Another one, which I have slightly mixed feelings about but I still think everyone should read, is Design Patterns. It gives us a common vocabulary. There are a lot of good ideas in there.

Similar plays, such as Ken Thompson and Fran Allen disagreeing on the badness of C, happen in later chapters, tieing together the different chapters and illustrating how even really good programmers disagree on the Right Thing all the time. Clearly the craft of programming is no settled thing. Besides the general structure of the book being well thought-out, the material is generally thought-provoking and interesting. One thing that stood out to me was Joshua Bloch describing what he called the "empathy gene", which is what a programmer has to have if he's going to be able to design good APIs and programming languages he has to be able to put himself in the shoes of the person who will be using the language or API. This is one thing that differentiates how different programmers can be good at different things. Another thing that stood out to me is that many of those interviewed stated that they don't use much in the way of modern tools and IDEs Joshua Bloch and Simon Peyton-Jones both touch on this, just to name a couple examples, even though some say that they think using these tools would make them more productive, especially when it comes to refactoring. This is a testament to the power of inertia sometimes there is just no chance to be unproductive now in order to be more productive later. Or perhaps just a sign that a programmable text-editor can stand on the same level as a heavier tool in terms of productivity in the right hands. I could go on with examples, but the conclusion here is that I thoroughly enjoyed Coders at Work, and I think it is a book that is well-worth the time spent reading the entire thing.

The next week I m going to Berlin (and some days to Frankfurt). I ll be in the 25th Chaos Communication Congress (25C3). I ll be out of home from 24th Dec until 7th Jan. It ll be a non-traditional holidays, be far away from family. But, in the other hand, it gonna be awesome be and talk in a CCC and spend a Christmas Eve in a plane and a New Year s Eve in Brandenburg Gate with friends. See you there?

The monthly IRC board meeting of Software in the Public Interest will take place later today, as announced by SPI’s secretary last week. While the announcement is back on time (yay!), the agenda isn’t (aww!). I’d be quite interested to learn how SPI is going to try to reduce the risk to its reserves, given the current slow decline of its primary bank which is not one of the first US banks getting bailed out. I think the best way for not-for-profits to avoid risking donations at the moment is to avoid having them in their bank accounts, in line with the Better Business Bureau standard that

“the charity’s unrestricted net assets available for use should not be more than three times the size of the past year’s expenses or three times the size of the current year’s budget, whichever is higher.”

Back in June 2005, SPI’s board of the time (Ian Jackson, John Goerzen, Jimmy Kaplowitz, David Graham, Bruce Perens, Benj. Mako Hill, Branden Robinson) decided to “remain noncompliant” with that standard and I fear that chicken could be coming home to roost now. I hope we don’t lose anything, but AIUI we’ve got nearly $150,000 in play. Update: Unlike its UK analogue, the Federal Deposit Insurance Corporation covers corporation accounts up to $250,000, so SPI is only risking temporary unavailability, not yet a risk of loss. Thanks to bd_ for pointing me to that.

I just wrote about the system administration issues related to the recent Debian SSL/SSH security flaw [1]. The next thing we need to consider is how we can change things to reduce the incidence of such problems. The problem we just had was due to the most important part of the entropy supply for the random number generator not being used due to a mistake in commenting out some code. The only entropy that was used was the PID number of the process which uses the SSL library code which gives us 15 bits of entropy. It seems to me that if we had zero bits of entropy the problem would have been discovered a lot sooner (almost certainly before the code was released in a stable version). Therefore it seems that using a second-rate source of entropy (which was never required) masked the problem that the primary source of entropy was not working. Would it make sense to have a practice of not using such second-rate sources of entropy to reduce the risk of such problems being undetected for any length of time? Is this a general issue or just a corner case? Joss makes some suggestions for process improvements [2]. He suggests that having a single .diff.gz file (the traditional method for maintaining Debian packages) that directly contains all patches can obscure some patches. The other extreme is when you have a patch management system with several dozen small patches and the reader has to try and discover what each of them does. For an example of this see the 43 patches which are included in the Debian PAM package for Etch, also note that the PAM system is comprised of many separate shared objects (modules), this means that the patching system lends itself to having one patch per module and thus 43 patches for PAM isn’t as difficult to manage as 43 patches for a complex package which is not comprised of multiple separate components might be. That said I think that there is some potential for separating out patches. Having a distinction between different types of patches might help. For example we could have a patch for Makefiles etc (including autoconf etc), a patch for adding features, and a patch for fixing bugs. Then people reviewing the source for potential bugs could pay a lot of attention to bug fixes, a moderate amount of attention to new features, and casually skim the Makefile stuff. The problem began with this mailing list discussion [3]. Kurt’s first message starts with “When debbuging applications” and ends with “What do you people think about removing those 2 lines of code?“. The reply he received from Ulf (a member of the OpenSSL development team) is “If it helps with debugging, I’m in favor of removing them“. It seems to me that there might have been a miscommunication there, Ulf may have believed that the discussion only concerned a debugging built and not a build that would eventually end up on millions of machines. It seems possible that the reaction would have been different if Kurt had mentioned that he wanted to have a single source tree for both debugging and for regular use. It also seems likely that his proposed change may have received more inspection if he had clearly stated that he was doing to include it in Debian where it would be used by millions of people. When I am doing Debian development I generally don’t mention all the time “this code will be used by millions of people so it’s important that we get it right“, although I do sometimes make such statements if I feel that my questions are not getting the amount of consideration from upstream that a binary package destined for use by millions of people deserves. Maybe it would be a good practice to clarify such things in the case of important packages. For a package that is relevant to the security of the entire distribution (and possibly to other machines around the net - as demonstrated in this case) it doesn’t seem unreasonable to include a post-script mentioning the scope of the code use (it could be done with an alternate SIG if using a MUA that allows selecting from multiple SIGs in a convenient manner). In the response from the OpenSSL upstream [4] it is claimed that the mailing list used was not the correct one. Branden points out that the openssl-team mailing list address seems not to be documented anywhere [5]. One thing to be learned from this is that distribution developers need to be proactive in making contact with upstream developers. You might think that building packages for a major distribution and asking questions about it on the mailing list would result in someone from the team noticing and mentioning any other things that you might need to do. But maybe it would make sense to send private mail to one of the core developers, introduce yourself, and ask for advice on the best way to manage communication to avoid this type of confusion. I think that it is ideal for distribution developers to have the phone numbers of some of the upstream developers. If the upstream work is sponsored by the employer of one of the upstream developers then it seems reasonable to ask for their office phone number. Sometimes it’s easier to sort things out by phone than by email. Gunnar Wolf describes how the way this bug was discovered and handled shows that the Debian processes work [6]. A similar bug in proprietary software would probably not be discovered nearly as quickly and would almost certainly not be fixed in such a responsible manner. Update: According to the OpenSSL project about page [7], Ulf is actually not a “core” member, just a team member. I had used the term “core” in a slang manner based on the fact that Ulf has an official @openssl.org email address.

• [1] http://etbe.coker.com.au/2008/05/18/debian-ssh-problems/


• [2] http://np237.livejournal.com/17981.html


• [3] http://marc.info/?t=114651088900003&r=1&w=2


• [4] http://www.links.org/?p=328


• [5] http://advogato.org/person/branden/diary/5.html


• [6] http://gwolf.org/node/1743


• [7] http://www.openssl.org/about/

Share This

Ok, so it seems that the whirlwind on OpenSSL has settled down a bit. Posts about it are coming from everywhere, ranging from rants on package maintenance to blame-pointing on both upstream and packager sides. And, of course, Slashdot. Where does all this leave the end user with? Well, probably not much except to regenerate weak SSH keys with the new openssh-server (now enhanced with openssh-blacklist, see the new advisory) and hope to $DEITY all gets well soon. And maybe, just maybe, a minor suspicion that other Debian-packaged software may be "tainted" with a similar blemish (that being having patches that are supposed to fix something, applied with upstream's blessing, and yet not really audited enough to ensure functionality AND security of the system is maintained.) Obviously, there's going to be some adjustments to be made on the Debian side. But I do hope to $DEITY that major revamps ought to happen on the OpenSSL side as well, in particular on clarifying their public channels to reaching upstream developers (read: publish openssl-team@openssl.org in a legitimate way, being the legitimate upstream contact endpoint it is,) and keeping a closer eye on the vendors who package their software (yeah, it may not be an obligation at all for OpenSSL, but heck, their vendors are users, too!) Upstream may be free not to partake on a social contract like Debian's, but it shouldn't escape from them the fact that vendors nevertheless aggregate continuing and potential users (aside from being users themselves) for their benefit. More importantly though, is that delivering FOSS is a community effort. Sure, its easy to put blame now, but in the end, the blame isn't as important as the real cause and effects of the problem/bug/issue are. Better to move on and work together towards a real fix, rather than the bickering that currently passes as FOSS entertainment.

Of course, this post is about the results for the recent Debian Project Leader elections. Although I've written quite a bit about politics in my country, I cannot sadly hold any hopes for decent electoral (or post-electoral, or political in general) results here in Banana-land. All hail our de-facto president, BTW.
Anyway, on to happier realms. The DPL elections are over. Voter turnout was low, as Manoj pointed out, but it's not as low as I feared. And, /methinks, not only because of the shortened period - but because this was generally an easy-going election, with three quite good candidates. As CMOT pointed out in my previous posting on this topic, many people would have voted 111- (which means, for the casual reader, any of them is quite OK with me). Debian elections are voted with a very interesting system, the Condorcet method. Developers don't vote just for their favorite option, but rank all of the available options (including none of the above, or NOTA) according to their personal preference.
The final numbers are very worth noting - Excuse me for hot-linking the image, but I know many non-Debian people read this:

What is so unique this time? First, as I anticipated on my previous posting, all of the candidates are above NOTA. Not only that, they are all well over NOTA, with the smallest distance being 237 votes (out of a 401 total votes received).
Second, something very positive as well, the distance between the three candidates is quite large this time. In 2006, the distance between first and second place was 6 votes (reason for which, together with the proximity in their platforms of course, Steve was appointed Second in Command or 2IC by AJ). In 2007, Steve was (again) second place, eight votes under Sam. Looking a bit into the history, in 2003, the closest election we have had, Martin beat Bdale by four and Branden by 11 votes... No, the distance is not really that important in the end, as we are quite far from having political quarrels over vote results - But still, having nice, clear numbers feels much better. And as I said in a previous message as well, it somehow speaks of Debian being a more mature, stable project.
Anyway - Congratulations, Steve! Best wishes for a stressful year, quite probably full of travelling, presentations and work in general!

A few years back I asked Branden Robinson for access to the X Strike Force SVN repository in order to improve the X server's autodetection, with the goal of stealing what I could from knoppix. At the time, users were constantly wandering in to #debian saying that they had used knoppix to create an XF86Config-4 and then they used that file on their Debian installations. They were also constantly whining about how Debian wasn't doing as good a job. So I decided to do something about it, and took a look at what knoppix was actually doing so much better than us, and I was surprised to find that they pretty much just wrote a skeleton configuration file and let the X server fill in the details. I had no idea the server was capable of such things. So Branden graciously gave me SVN access and I began comparing the knoppix method to the script called by "dpkg-reconfigure xserver-xfree86" and realized that we probably couldn't adopt the same method because we had tons of checks for portability. So I put the problem on the backburner for few weeks and worked on something else for a while.

Well, somewhere in between then and now I got sucked in to transitioning Debian to X.org (using the aforementioned SVN access) and then working on all the things that went along with maintaining X in Debian, some well and others less so. Between transitioning to Xorg, and then transitioning to a modular Xorg, even with the ability to steal Ubuntu's packages it still took about two full years with me being a rookie and all that. There were tons of people who worked on this with me, but it was just a damn big job. Eventually though, etch released with modular X packages, and we were running at a pretty good pace with upstream, so it was time to revisit the problem of configuration again after the two year detour.

The problem of configuration had been reframed for me by having looked at what knoppix was doing and discussions with upstream. Upstream was starting to come to the conclusion that we shouldn't have a config file at all, and that the server should be smart enough to do everything by itself. It was already partially there, it just needed a push in the right direction. This was a major shift in how I thought of it. Coming from a Debian background, where the answer is to always just regenerate or edit your config file, having the server work things out for you was a totally alien idea. But I have my deepest roots in the Macintosh world, so immediately fell in love with it. The problem was that the server had a whole body of code to use if you had no config file at all, and another, with far less automagic goodness, if you had a config file, even if it was a 0 byte file. The goal became to have the server work really well with a minimal config file, so you could override what you don't like in the defaults and let the server figure things out for itself at boot. The way forward was to translate as much of the logic in our configure script in to the X server itself as possible.

Ubuntu had put in a lot of work in to the configuration setup that we automatically benefitted from, so given that most users were happy with things as they were, I was able to carve away at the problem without disrupting anything. And there was a lot to carve out, as the script that runs the config is an absolute mess that was slated by Branden for a rewrite all those years ago. Early on I picked off the low hanging fruits like the font path and modules. Similarly, Redhat's Adam Jackson had also been working on this problem, and he killed off the ServerLayout section as well as putting in lots of critical fixes all over the place elsewhere. More recently, I've gone and cut explicit modesetting out of the configure script. Hardcoding this information is generally a bad idea in the randr 1.2 world, and most of the drivers will do as good or better job of figuring out the modes than our config script would. This let us jettison using the xresprobe program to ask the monitor for the settings to use. This cut out a lot of the code that we had to deal with, simplifying the configure script and letting us all benefit from upstream's work. This leaves a big gaping hole in user configuration, which is something I'm looking to address in a few weeks, but for now there are workarounds like editing your xorg.conf manually to make things work.

Finally, yesterday I was able to upload a version of the script that no longer uses the discover program to figure out what driver to load. I've patched the server to do this at runtime. If there's no driver listed it simply scans the PCI bus, picks out your primary video card, and loads the first driver that claims to support that PCI ID. This let us jettison the last external dependency that the configure script had, so now we have a relatively small chunk of shell script with no external C code and a simplified setup. At this point we're now shipping a more skeletal config file than knoppix ever did simply because it wasn't possible to ship something like this before and have the server work at all. This is a huge milestone for me because finally, after over three years, the problem I originally set out to work on with X is done. There's still bugs to uncover and fix with all of this, but I'm convinced that what we have now is superior to our old method. Eventually, with any luck, xorg.conf will just fade away for most people. There's a lot to work on before that happens though, but I'm happy to have finally gotten here.

Today while looking for something else, I stumbled across a DVD with the "last archive" of my old personal website. On it were a number of photos from the 2000 Annual Linux Conference in Atlanta, and the Debian developers that were there. These were posted in public for several years.

I've now posted all of them on flickr, preserving the original captions.

Here's the obligatory sample:



That's Joey Hess, using what I think was his Vaio. Most acrobatic keyboardist ever. Probably the only person that could write Perl with one hand comfortably.

What else can you see? The best of show award that Debian won that is now in my basement due to a complicated series of events, the Debian machines that were being shown off at the show, Sean Perry and Manoj, the photo with long-term corrupted caption, and of course, numerous shots of Branden.

I know the size stinks. It was scanned at a web resolution for 2000. I do still have the negatives somewhere and will post the rest of them, in higher res, when I find them.

Click here to view the full set.

For various reasons, Software in the Public Interest (SPI) owned the domains opensource.org and opensource.net, which have been managed by the Open Source Initiative (OSI) for most of their life. This supported various SPI goals and also gave a possibility of community control of opensource.org/net just in case it was ever needed. OSI does not offer much opportunity for community control, as it is a self-appointing board. (Regular readers may remember that the principles of democratic member control, autonomy and concern for community are important to me.) I was watching as the SPI board voted in favour of giving away the opensource domains, proposal 2006-11-18.dbg.mjs.1 on the agenda. The vote happened without any discussion in the meeting. From memory (I guess logs will appear on the SPI site eventually):- David Graham ("SPI must also be involved in the promotion of and education about open source") proposed and voted for this action that stops SPI being involved in this promotion and education about open source; Michael Schultheiss ("I look forward to SPI's future expansion") (amended and?) voted for this action that reduces SPI; Neil McGovern ("achieve a greater degree of involvement [...] from the community in general") voted for this action that lessens SPI's involvement; Jimmy Kaplowitz ("It is important that it continue holding in trust the money and other legal assets belonging to its member projects [...] fulfill some more of its stated corporate purposes, involving education of the general public about free software and about computers in general") voted for this action that drops these assets and reduces involvement in education of the general public; Bdale Garbee ("would like to close this issue one way or another, once and for all") seemed willing to vote for anything which prevented further discussion - of course, only surrendering the domains could close it once and for all, as this is a one-way trapdoor topic - but had apologised for absence from the meeting; Martin "Joey" Schulze had apologised for absence from the meeting; Josh Berkus (nothing obviously relevant in his platform) abstained - consistent but disappointing; I think Branden Robinson was missing. Ian Jackson ("SPI's role is to provide a stable legal entity which can hold assets") voted against this action so that SPI would keep holding these assets. Bravo! Sources: The quotes above are the board's own promises in their election platforms, as listed on the SPI site in 2003 2006 apart from Bdale Garbee's platform which wasn't at the link given in 2004 so that quote is from the minutes of the previous meeting If some board members are willing to give away assets for no good reason, that strikes at one of SPI's core methods: holding assets for the community. What can we trust about this board? Their platforms are full of fine sentiments, but how do their actions reflect their platforms? This may be a mostly-dormant issue until the next election. If you want any of SPI's assets, just ask. If your request is rejected (as SPI rejected the opensource domain give-away before), just keep asking new sets of voters until they agree.

...a little, Jesus. I just sent 3 bug reports in a row, in name of the French translation team and the last one (which happens to be my translation for libroxen-imho debconf templates) won (yeah, I cheated twice). Considering that Michel Grentzinger already got #200000 in the past, we couldn't make less, indeed (yes, Noel). Funnily, looking at these old bug reports, I found that #100000 was fixed by a future DPL (Branden) and #200000 by the then DPL (Martin Michlmayr). So, if you dream to run for DPL, you should consider NMU'ing libroxen-imho with the patch offered in #400000, or you should be Turbo Fredrikson, the package maintainer. Interesting stats:

• I couldn't find the oldest bug report. #2531 is dated March 14th 1996 (before Buzz release)
• Bug #100000 was reported on Jun 7th 2001. More than 5 years
• Bug #200000 was reported on Jul 4th 2003. 23 months
• Bug #300000 was reported on Mar 17th 2005. 20 months
• Bug #400000 is reported on Nov 23rd 2006. 20 months

So, the bug reporting rate is indeed more or less constant during last years.