Some days ago, a hacker group, THC, released a denial of service tool for SSL web servers. As stated in its description, the problem is not really new: a complete SSL handshake implies costly cryptographic computations. There are two different aspects in the presented attack:

• The computation cost of an handshake is more important on the server side than on the client side. The advisory explains that a server will require 15 times the processing power of a client. This means a single average workstation could challenge a multi-core high-end server.
• The use of SSL renegotiation allows to trigger hundreds of handshakes in the same TCP connection. Even a client behind a DSL connection can therefore bomb a server with a lot of renegotiation requests.

Mitigation techniques There is no definitive solution to this attack but there exists some workarounds. Since the DoS tool from THC relies heavily on renegotiation, the most obvious one is to disable this mechanism on the server side but we will explore other possibilities.

Disabling SSL renegotiation Tackling the second problem seems easy: just disable SSL renegotiation. It is hardly needed: a server can trigger a renegotiation to ask a client to present a certificate but a client usually does not have any reason to trigger one. Because of a past vulnerability in SSL renegotiation, recent version of Apache and nginx just forbid it, even when the non-vulnerable version is available. openssl s_client can be used to test if SSL renegotiation is really disabled. Sending R on an empty line trigger renegotiation. Here is an example where renegotiation is disabled (despite being advertised as supported):

$ openssl s_client -connect www.luffy.cx:443 -tls1
[...]
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
[...]
R
RENEGOTIATING
140675659794088:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:

Disabling renegotation is not trivial with OpenSSL. As an example, I have pushed a patch to disable renegotiation in stud, the scalable TLS unwrapping daemon.

Rate limiting SSL handshakes Disabling SSL renegotiation on the client side is not always possible. For example, your web server may be too old to propose such an option. Since those renegotiations should not happen often, a workaround is to limit them. When the flaw was first advertised, F5 Networks provided a way to configure such a limitation with an iRule on their load-balancers. We can do something similar with just Netfilter. We can spot most TCP packets triggering such a renegotiation by looking for encrypted TLS handshake record. They may happen in a regular handshake but in this case, they usually are not at the beginning of the TCP payload. There is no field saying if a TLS record is encrypted or not (TLS is stateful for this purpose). Therefore, we have to use some heuristics. If the handshake type is unknown, we assume that this is an encrypted record. Moreover, renegotiation requests are usually encapsulated in a TCP packet flagged with push .

# Access to TCP payload (if not fragmented)
payload="0 >> 22 & 0x3C @ 12 >> 26 & 0x3C @"
iptables -A LIMIT_RENEGOCIATION \
    -p tcp --dport 443 \
    --tcp-flags SYN,FIN,RST,PSH PSH \
    -m u32 \
    --u32 "$payload 0 >> 8 = 0x160300:0x160303 && $payload 2 & 0xFF = 3:10,17:19,21:255" \
    -m hashlimit \
    --hashlimit-above 5/minute --hashlimit-burst 3 \
    --hashlimit-mode srcip --hashlimit-name ssl-reneg \
    -j DROP

The use of u32 match is a bit difficult to read. The manual page gives some insightful examples. $payload allows to seek for the TCP payload. It only works if there is no fragmentation. Then, we check if we have a handshake (0x16) and if we recognise TLS version (0x0300, 0x0301, 0x0302 or 0x0303). At least, we check if the handshake type is not a known value. There is a risk of false positives but since we use hashlimit, we should be safe. This is not a bullet proof solution: TCP fragmentation would allow an attacker to evade detection. Another equivalent solution would be to use CONNMARK to record the fact the initial handshake has been done and forbid any subsequent handshakes. If you happen to disable SSL renegociation, you can still use some Netfilter rule to limit the number of SSL handshakes by limiting the number of TCP connections from one IP:

iptables -A LIMIT_SSL \
    -p tcp --dport 443 \
    --syn -m state --state NEW \
    -m hashlimit \
    --hashlimit-above 120/minute --hashlimit-burst 20 \
    --hashlimit-mode srcip --hashlimit-name ssl-conn \
    -j DROP

Your servers will still be vulnerable to a large botnet but if there is only a handful of source IP, this rule will work just fine¹. I have made all those solutions available in a single file.

Increasing server-side power processing SSL can easily be scaled up and out. Since SSL performance increases linearly with the number of cores, scaling up can be done by throwing in more CPU or more cores per CPU. Adding expensive SSL accelerators would also do the trick. Scaling out is also relatively easy but you should care about SSL session resume.

Putting more work on the client side In their presentation of the denial of service tool, THC explains:

Establishing a secure SSL connection requires 15 more processing power on the server than on the client.

I don t know where this figure comes from. To check it, I built a small tool to measure CPU time of a client and a server doing 1000 handshakes with various parameters (cipher suites and key sizes). The results are summarized on the following plot: For example, with 2048bit RSA certificates and a cipher suite like AES256-SHA, the server needs 6 times more CPU power than the client. However, if we use DHE-RSA-AES256-SHA instead, the server needs 34% less CPU power. The most efficient cipher suite from the server point of view seems to be something like DHE-DSS-AES256-SHA where the server needs half the power of the client. However, you can t really uses those shiny cipher suites:

1. Some browsers do not support them: they are limited to RSA cipher suites².
2. Using them will increase your regular load a lot. Your servers may collapse with just legitimate traffic.
3. They are expensive for some mobile clients: they need more memory, more processing power and will drain battery faster.

Let s dig a bit more on why the server needs more computational power in the case of RSA. Here is a SSL handshake when using a cipher suite like AES256-SHA: When sending the Client Key Exchange message, the client will encrypt TLS version and 46 random bytes with the public key of the certificate sent by the server in its Certificate message. The server will have to decrypt this message with her private key. Those are the two most expensive operations in the handshake. Encryption and decryption are done with RSA (because of the selected cipher suite). To understand why decryption is more expensive than encryption, let me explain how RSA works. First, the server needs a public and a private key. Here are the main steps to generate them:

1. Pick two random distinct prime numbers p and q , each roughly the same size.
2. Compute n=pq . It is the modulus.
3. Compute \varphi(n)=(p-1)(q-1) .
4. Choose an integer e such that 1<e<\varphi(n) and \gcd(\varphi(n),e) = 1 (i.e. e and \varphi(n) are coprime). It is the public exponent.
5. Compute d=e^ -1 \mod\varphi(n) . It is the private key exponent.

The public key is (n,e) while the private key is (n,d) . A message to be encrypted is first turned into an integer m<n (with some appropriate padding). It is then encrypted to a ciphered message c with the public key and should only be decrypted with the private key:

• c=m^e\mod n (encryption)
• m=c^d\mod n (decryption)

So, why is decryption more expensive? In fact, the key pair is not really generated like I said above. Usually, e is a small fixed prime number with a lot of 0, like 17 (0x11) or 65537 (0x10001) and p and q are choosen such that \varphi(n) is coprime with e . This allows encryption to be fast using exponentiation by squaring. On the other hand, its inverse d is a big number with no special property and therefore, exponentiation is more costly and slow. Instead of computing d from e , it is possible to choose d and compute e . We could choose d to be small and coprime with \varphi(n) and then compute e=d^ -1 \mod\varphi(n) and get blazingly fast decryption. Unfortunately, there are two problems with this:

• Most SSL implementation expects e to be a 32bit integer.
• If d is too small (less than one-quarter as many bits as the modulus n ), it can be computed efficiently from the public key.

Therefore, we cannot use a small private exponent. The best we can do is to choose the public exponent to be e =4294967291 (the biggest prime 32bit number and it contains only one 0). However, there is no change as you can see on our comparative plot. To summarize, no real solution here. You need to allow RSA cipher suites and there is no way to improve the computational ratio between the server and the client with such a cipher suite.

Things get worse Shortly after the release the denial of service tool, Eric Rescorla³ published a good analysis on the impact of such a tool. He asks himself about the efficiency to use renegotiation for such an attack:

What you should be asking at this point is whether a computational DoS attack based on renegotiation is any better for the attacker than a computational DoS attack based on multiple connections. The way we measure this is by the ratio of the work the attacker has to do to the work that the server has to do. I ve never seen any actual measurements here (and the THC guys don t present any), but some back of the envelope calculations suggest that the difference is small. If I want to mount the old, multiple connection attack, I need to incur the following costs:

1. Do the TCP handshake (3 packets)
2. Send the SSL/TLS ClientHello (1 packet). This can be a canned message.
3. Send the SSL/TLS ClientKeyExchange, ChangeCipherSpec, Finished messages (1 packet). These can also be canned.

Note that I don t need to parse any SSL/TLS messages from the server, and I don t need to do any cryptography. I m just going to send the server junk anyway, so I can (for instance) send the same bogus ClientKeyExchange and Finished every time. The server can t find out that they are bogus until it s done the expensive part. So, roughly speaking, this attack consists of sending a bunch of canned packets in order to force the server to do one RSA decryption.

I have written a quick proof of concept of such a tool. To avoid any abuse, it will only work if the server supports NULL-MD5 cipher suite. No sane server in the wild will support such a cipher. You need to configure your web server to support it before using this tool. While Eric explains that there is no need to parse any SSL/TLS messages, I have found that if the key exchange message is sent before the server send the answer, the connection will be aborted. Therefore, I quickly parse the server s answer to check if I can continue. Eric also says a bogus key exchange message can be sent since the server will have to decrypt it before discovering it is bogus. I have choosen to build a valid key exchange message during the first handshake (using the certificate presented by the server) and replay it on subsequent handshakes because I think the server may dismiss the message before the computation is complete (for example, if the size does not match the size of the certificate). With such a tool and 2048bit RSA certificate, a server is using 100 times more processing power than the client. Unfortunately, this means that most solutions, except rate limiting, exposed on this page may just be ineffective.

---

1. However, since this rule relies on source IP to identify the attacker, the risk of false positive is real. You can slow down legitimate proxies, networks NATed behind a single IP, mobile users sharing an IP address or people behind a CGN.
2. Cipher suites supported by all browsers are RC4-MD5, RC4-SHA and 3DES-SHA. Support for DHE-DSS-AES256-SHA requires TLS 1.2 (not supported by any browser).
3. Eric is one of the author of several RFC related to TLS. He knows his stuff.

As you will know, every file on a unixoid system has some meta-data like owner, group and permission bits. This is of course also true for files part of some Debian package. And it is not very surprising that different files should have different permissions. Or even different owners or groups. Which file has which settings is of course for the package maintainer to decide and Debian would not be Debian if there were not ways for the user to give their own preferences and have them preserved. This post is thus about how those settings end up in the package and what is to be observed when doing it. As you will also have heard, a .deb file is simply a tar archive stored as part of an ar archive, as you can verify by unpacking a package manually:

$ ar t reprepro_4.8.1-1_amd64.deb
debian-binary
control.tar.gz
data.tar.gz
$ ar p reprepro_4.8.1-1_amd64.deb data.tar.gz   gunzip   tar -tvvf -
drwxr-xr-x root/root         0 2011-10-10 12:05 ./
drwxr-xr-x root/root         0 2011-10-10 12:05 ./etc/
drwxr-xr-x root/root         0 2011-10-10 12:05 ./etc/bash_completion.d/
-rw-r--r-- root/root     19823 2011-10-10 12:05 ./etc/bash_completion.d/reprepro
drwxr-xr-x root/root         0 2011-10-10 12:05 ./usr/
drwxr-xr-x root/root         0 2011-10-10 12:05 ./usr/share/
--More--

(For unpacking stuff from scripts, you should of course use dpkg-deb --fsys-tarfile instead of ar gunzip. Above example is about the format, not a recipe to unpack files). This already explains how the information is usually encoded in the package: A tar file contains that information for each contained file and dpkg is simply using that information. (As tar stores numeric owner and group information, that limits group and owner information to users and groups with fixed numbers, i.e. 0-99. Other cases will be covered later.) The question for the maintainer is now: Where is the information which file has which owner/group/permissions in the .tar inside the .deb, and the answer is simple: It's taken from the files to be put into the .deb. This means that package tools could simply be implemented first by simply calling tar and there is no imminent need to write you own tar generator. It also means that the maintainer has full control and does not have to learn new descriptive languages or tools to change permissions, but can simply put the usually shell commands into debian/rules. There are some disadvantages, though: A normal user cannot change ownership of files and one has to make sure all files have proper permissions and owners. This means that dpkg-deb -b (or the usually used wrapper dh_builddeb) must be run in some context where you could change the file ownership to root first. This means you either need to be root, or at least to fake being root by using fakeroot. (While this could be considered some ugly workaround, it also means upstream's make install is run believing to be root, which also avoids some -- for a packager -- quite annoying automatisms in upstream build scripts assuming a package is not installed system wide if not installed as root). Another problem are random build host characteristics changing how files are created in the directory later given to dpkg-deb -b. For example an umask which might make all files non-world-readable by default. The usual workaround is to first fix up all those permissions. Most packages use dh_fixperms for this, which also sets executable bits according to some simple rules and has some more special cases so that the overall majority of packages does not need to look at permissions at all. So using some debhelper setup, every special permissions and all owner/group information for owner groups with fixed numbers only needs to be set using the normal command line tools between dh_fixperms and dh_builddeb. Everything else happens automatically. Note that games is a group with fixed gid. So it is not necessary (and usually a bug) to change group-ownership of files withing the package to group games in maintainer scripts (postinst,...). If a user wants to change permissions or ownership of a file, dpkg allows this using the dpkg-statoverride command. This command essentially manages a list of files to get special treatment and ownership and permission information they should get. This way a user can specify that files should have different permissions and this setting is applied if a new version of this file is installed by dpkg. Being a user setting especially means, that packages (that means their maintainer scripts) should not usually use dpkg-statoverride. There are two exceptions, though: Different permissions based on interaction with the user (e.g. asking question with debconf) and dynamically allocated users/groups with dynamic id. In both cases one should note that settings to dpkg-statoverride are settings of the user, so the same care should be given as to files in /etc, especially one should never override something the user has set in there. (I can think of no example where calling dpkg-statoverride --add without dpkg-statoverride --list in some maintainer script is not a serious bug: Either you override user settings or you are using debconf as a registry. Moral To recap, your package is doing something wrong if:

• Your maintainer scripts unconditionally change permissions of files within the package (Those should either be set within the package or you miss user configuration in dpkg-statoverride).
• Your maintainer scripts call dpkg-statoverride --add without having checked first whether the user already has done so with different values.

Many people around me switched to Chrome or Chromium. I also used it for a bit, but I was a bit disappointed about the extensions available. To show why, here's a list of the extensions I've currently installed:

• Adblock Plus: I guess everyone knows this. It sits quitely in the background and removes quite a bunch of eye-distracting stuff from web pages. You see the web differently and are always confused when viewing pages on your smartphone.
• British English Dictionary: Pretty self-explaining. Firefox has an integrated spell checker and that one needs a dictionary.
• BugMeNot: You can right-click on a login field and let it insert BugMeNot data automatically. I don't use it often enough to remember that it's there.
• Certificate Patrol: SSL's X.509 trust model is weak, to say the least. This extension implements the "save on first visit" trust model and warns you if the certificate or the CA of a URL changes.
• Download Statusbar: If you're used from, say, Chromium to see your download progress above the status bar, this extension will give you that. I don't like the separate window, especially with Awesome as my window manager.
• Firebug: Most people know this, too. Very useful when you do web development. You can see the effects of CSS as you type, for instance.
• Flashblock: YouTube has its HTML5 trial, so you don't need Flash for it. Sadly I had to give in recently (for live streaming sites like the German Parliament, go figure) and have it installed for "emergency" cases. But really nobody needs Flash advertisements or other silly Flash animations, so Flashblock will conveniently refuse to load the Flash plugin unless you tell it to.
• German Dictionary: As above.
• Google Translator for Firefox: Translates marked sections on a web page in place. It lets Google guess the source language, so you really only mark the area, click the button and be done.
• Greasemonkey: Modifies web pages in place to make them more sane, using little scripts. I use two currently:
• Better Outlook Web Access (OWA): For odd reasons I'm forced to use OWA 2007 at a company of Windows and Mac users. This makes it a little bit more bearable, by allowing you to save your password and by adding a message preview pane. You really don't want to use OWA Light 2007 without this.
• Tagesschau.de - video tag: A simple script that lets you watch videos on tagesschau.de without resorting to Flash. They ship Theora videos alongside H.264, which is supported by Firefox out of the box.
• Lazarus: Form Recovery: This already saved me many, many times. Sometimes I hit the stupid Thinkpad page back/page forward keys, or the browser crashes or I'm torn away from a page in another way. This extension keeps your form content mostly save, so that you don't need to start from scratch. A must have.
• Live HTTP headers: curl -D usually works. But sometimes you need authentication and cookies and stuff, hence doing it in the browser makes sort of sense.
• Modify Headers: I don't really use it anymore. There was this legend that YouTube lets you watch videos that are blocked in your country (Hello, GEMA, I'm looking at you!) if you provide a X-Forwarded-For header with an IP from another country. Never worked for me.
• Mozilla Archive Format: If you need to archive a web page or want to collect a bunch of pages for offline reading, this is the way to go. You can conveniently select either single pages or several tabs to be stored.
• Perspectives: I'm paranoid, so here's the second SSL certificate check. Perspectives uses a bunch of network notaries hosted on PlanetLab to check if everyone sees the same certificate. Together with Certificate Patrol that means if that you save a self-signed certificate on first visit if the notaries all agree that it's the currently installed one.
• PwdHash: Password re-use on different web sites is bad. There are sites where I'm not very concerned about the strength of my password, but where I don't want to leak my main ones. PwdHash uses the domain name and the password I give as components to a hash function. So if I'm at a computer without access to my saved passwords, I can easily reconstruct the hash. If need be, I can use the JavaScript on the developer's web site to do that.
• RequestPolicy: It's currently installed but I don't think it will stick around much longer. Too many sites are using embedded cross-site requests, which this extension will allow you to review. As an example every Blogger site that's on a custom domain (like my blog) will trigger it. Instead half of your web will have red flags everywhere full of blocked requests. Not that helpful. Interestingly enough it also polices Flash's outgoing web requests. Which breaks even more often than normal web pages.
• Secure Login: This gives you a button (or a keyboard shortcut) that allows you to choose from several logins and then goes on and posts those credentials securely to the page in question, bypassing any JavaScript that might want to see them. You also get auto-login bookmarks for those sites which don't allow you to remain logged-in across browser sessions. (Like OWA.)

If Firefox on Android were quicker to start and faster overall, I might even use it there. But as-is it's not very useful. Sadly this also means that I can't use Firefox Sync on my phone and as I don't use Chrome on my desktop I also can't use Chrome to Phone. So I usually go and build a QR code on my laptop and read that with Android's Barcode Scanner.

Of course I'm actually using Iceweasel and I'm very grateful for Mike Hommey's efforts to track the release channel on mozilla.debian.net.

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (130.30 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg work The month started with fixing newly reported bugs to prepare the 1.16.1.1 release:

• #644492: there was a flaw in a change I made to the trigger setup code. This resulted in packages being incorrectly marked as configured while they were only unpacked in a new chroot.
• #642656: dpkg-source s refusal to build when it detects unrecorded changes broke the (mostly unused, except by the lintian test suite apparently) 2.0 source format.
• #644412: the Makefile snippet buildflags.mk did not respect the new maintainer specific environment variables (like DEB_CFLAGS_MAINT_APPEND) because make does not forward environment variable through $(shell ). Fixed that by manually exporting the required variables.
• I also disabled dpkg-buildpackage s output of the build flags since it was confusing several maintainers. dpkg-buildpackage invokes debian/rules and it has no (clean) way to discover the build flags changes that maintainer request by setting the dedicated environment variables in debian/rules. Maintainers expect to see the build flags with the modifications they have requested and not just the default values set by the distribution.

With the help of Guillem, we decided on a proper fix for a race condition sometimes triggered by parallel builds when 2 concurrent dpkg-gencontrol try to update debian/files (see #642608). This ended up requiring a new package (libfile-fcntllock-perl) that the Debian perl team kindly packaged for us. With all this sorted, it was a rather easy fix. Multiarch progress I also spent lots of time on multiarch. I fixed an old bug that requested to support the multi-arch paths in case of cross-building (see #595144), the discussion was not really conclusive on which of the two proposed patches was better so I ended up picking my own patch because it was closer to how we currently deal with cross-building. Then I fixed 2 issues that have been reported on Ubuntu s dpkg. The first one (LP #863675) was rather severe since an installed package ended being disappeared in favor of its foreign counterpart that was removed (but that had some config files left). The second one (LP #853679) only affected dselect users (apparently there are still some!) who had a self-conflicting library (Provides: foo, Conflicts: foo) installed for multiple architectures. But the bulk of the time spent on multiarch has been spent discussing with various parties on how to go forward with multiarch. The release team commented on the schedule of the merge to ensure it makes it into Wheezy, and the Debian project leader also commented on the problems encountered so far. While not the best course of action I could have hoped for, it certainly helped since Guillem started pushing some reviewed commits. Out of the 66 commits that were in my pu/multiarch/full branch one week ago, 20 have been merged in the master branch already. Python-django security update and RC bug Since python-django s maintainer did not manage to prepare the required security updates, I stepped in and prepared version 1.2.3-3+squeeze2 for Squeeze and 1.0.2-1+lenny3 for Lenny. Unfortunately this security update is an example of how an inactive maintainer is likely to result in a severe delay for the release of security updates. Furthermore in this specific case, the security team did not want to release the Squeeze security update until the Lenny one had been investigated (which required some time since upstream no longer supports the version in Lenny) but they did not make this very clear. Later another release critical bug had been filed against the package (#646634) but after investigation, it turned out to be a local configuration problem so I downgraded it. I still forwarded the test suite failure to upstream authors since the test could be enhanced. In any case, co-maintainers for python-django are welcome. I really preferred the situation where I can quietly sit down as backup maintainer WordPress packaging WordPress sounds similar to python-django. I m also only a backup maintainer but Giuseppe has been inactive for many months and I had to step in August because I wanted the new upstream version. I discovered a bit late that I was not subscribed to wordpress bugs and thus the release critical bug #639733 (that I introduced with my new upstream version) went unattended for a rather long time. Once aware, though, I quickly fixed it. I also took the opportunity to start a discussion on debian-devel about how to deal with embedded javascript libraries and proposed a mechanism of opportunistic replacement with symlinks . WordPress is my testbed package for this mechanism, you can check out its debian/dh_linktree that implements the replacement logic. The discussion has not been very interesting but at least I learned that Debian now requires that each source package shipping minified javascript files includes the original files too. It s somewhat of a pain since it s not a license requirement in many cases (many of those libraries are not under the GPL), but just a Debian requirement that many upstreams are not complying with. WordPress is affected and Jakub Wilk thus opened #646729 which is going to be a long-standing RC bug. To give good measures, I spent several hours investigating the case of each javascript file in the WordPress source package and I filed a new ticket on the upstream bugtracker. Dropbox packaging work A few months after the introduction of nautilus-dropbox to Debian and Ubuntu, I can say that the decision to only support the download of dropbox in the postinst has been a mistake. Because of this decision I had to make the postinst fail if the download failed. Even if the error message is relatively clear, this lead to many (mostly automated) bug reports on the Ubuntu side. Various other problems cropped up on top of this (trying to start dropbox while the package was not configured would result in an error because the user did not have the required rights to install the software, reinstalling the package while dropbox was running would result in a failure too, etc.). I have fixed all those issues in the version 0.7.0-2 of the package. Now if the user has to install dropbox, it will use PolicyKit to request the root rights. The postinst will no longer fail if the dropbox download fails since it can be run later by the user. And I fixed the download code to remove the replaced file before unpacking a new file (insead of overwriting the existing file). All this work has been forwarded upstream. The Debian Administrator s Handbook Update I m glad to tell you that the translation will happen because we reached the minimal funding goal on October 22th with the help of 380 supporters. Now the fundraising continues, but this time the goal is the liberation of the resulting book. For this to happen, we need to reach 25000 EUR in the liberation fund. So far we re at 37% of this goal with 9400 EUR in the liberation fund (which means that 59% of the money raised has been put in the liberation fund).

Click here if you want to contribute towards the liberation of this book. With (less than) 27 days left, it s going to be a challenge to meet the goal, but we do like challenges, don t we? Misc work

• I filed #644486 against dh-make so that new packages have proper support of dpkg-buildflags from the start.
• I merged lots of patches from Luca Falavigna in the developers-reference.
• I discussed debtags integration in the PTS with Enrico Zini and Paul Wise.
• I updated publican s packaging for the new upstream version 2.8. I had to write a new patch that I forwarded upstream.
• I filed an upstream bug on hamster-applet because just running hamster-time-tracker no longer brings its window forward.

Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Volker Theile is well known for his work on FreeNAS. But at one point, he decided to give up on it and to restart from scratch but this time based on Debian (instead of FreeBSD). OpenMediaVault is the name of this new Debian based product. Here s how he defines the project:

OpenMediaVault is the next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, DAAP media server, RSync, BitTorrent client and many more. Thanks to the modular design of the framework it can be enhanced via plugins. OpenMediaVault is primarily designed to be used in home environments or small home offices, but is not limited to those scenarios. It is a simple and easy to use out-of-the-box solution that will allow everyone to install and administrate a Network Attached Storage without deeper knowledge.

Even though all the work was private, he managed to attract an important following, and I must say that I was looking forward to this project. He regularly blogged on his progress, sharing some good-looking video of the resulting product (example here). The first public release (Version 0.2, codenamed Ix) happened on October 17th. I have yet to try it but I took a look on the website. As a Debian developer, I was keen on seeing the source code and how the project was managed. The GPLv3 license is presented as an important feature and I was expecting a well-managed open source project. The fact that it was a private one-man project up to now did not bother me, we re quite used to the scratch your itch kind of start for free software projects. Enough to say is that I have been very disappointed. First you come across a contributor agreement, it s frowned upon by many free software developers. But why not, maybe he bought the argument of Mark Shuttleworth and wants to give it a try. But then I looked at the subversion repository, it s obvious that it s just a dumping ground of files that are managed somewhere else in another repository. A bit like Android which is not developed in the open but released as a whole from time to time. But the worst was yet to be found on the licensing page:

Beside being freely available for personal end-users, System Builders and System Integrators, in general Installers, require a commercial license for OpenMediaVault.

Besides the fact that I m still not sure what this means, I would like to know how he reconciles this requirement with the terms of the GPLv3. It s a shame that a so promising project ends up being a disaster from a free software perspective. Have you tried OpenMediaVault already? If yes, I d be glad to read your thoughts in the comments.

9 comments Liked this article? Click here. My blog is Flattr-enabled.

Hello With Config::Model, configuration models are created to validate configuration data. But, before that, how can we validate a model ? This has long been a sore spot of Config::Model. During its development, I ve progressively enhanced a test script to tests the various models (and their corner cases) shipped with Config::Model. This script is able to test several models with several subtests. Currently, more that 400 unit tests are done on 5 models and about 50 configuration files. Which is great, but usable only inside Config::Model distribution. Which is not so great So I ve bitten the bullet and created a new Config::Model::Tester class from this test script. This new class is shipped with Config::Model 1.260 (also available on Debian/sid). For instance, let s see how this tester is used to test Xorg model (shipped in Config::Model::Xorg). Xorg features a small test file (t/model_test.t):

use warnings;
use strict;
use Config::Model::Tester ;
use ExtUtils::testlib;
my $arg = shift   '';
my $test_only_model = shift   '';
my $do = shift ;
run_tests($arg, $test_only_model, $do) ;

The xorg files to test are specified by t/model_tests.d/xorg-test-conf.pl:

$conf_file_name = "xorg.conf" ;
$conf_dir = "etc/X11" ;
$model_to_test = "Xorg" ;
@tests = (
      name => 'fglrx',  ,
      name => 'modern',  ,
      name => 'vesa',  ,
      name => 'xorg',  ,
      name => 'xorg-ati',  ,
);
1;

And the xorg.conf files are there:

$ ls t/model_tests.d/xorg-examples/
fglrx  modern  vesa  xorg  xorg-ati

Here s the output of the xorg-ati subtest:

$ perl -Ilib t/model_tests.t x xorg xorg-ati
ok 1 - compiled
# Beginning xorg test (t/model_tests.d/xorg-test-conf.pl)
# xorg uses Xorg model on file xorg.conf
# Beginning subtest xorg xorg-ati
ok 2 - Copied xorg example xorg-ati
ok 3 - Read wr_root/test-xorg-ati/etc/X11/xorg.conf and created instance with init() method with warning check
ok 4 - Ran dump_tree
ok 5 - Dumped xorg config tree in full mode
ok 6 - Dumped xorg config tree in custom mode
ok 7 - xorg write back done
ok 8 - Created instance xorg-test-xorg-ati-w
ok 9 - Dumped xorg 2nd config tree in custom mode
ok 10 - compare original xorg custom data with 2nd instance custom data
ok 11 - check that original xorg file was not clobbered
# End of subtest xorg xorg-ati
# End of xorg test
1..11

Now, it s quite easy to create non regression tests for bug triggered by err, unexpected, configuration files: Just drop the file in the example directory and edit the *-test-conf.pl file. This saves a lot of typing. All the best

Executive summary Debian should consider:

• less fine-grained software packaging for overall simplification of the system and its development
• less flexibility at the lower levels of the software stack, again for overall simplicity
• reducing friction in the development workflow, especially by making it possible to branch and merge at the whole system level

Introduction I've taken a job with Codethink, as part of a team to develop a new embedded Linux system, called Baserock. Baserock is not a distribution as such, but deals with many of the same issues. I thought it might be interesting to write out my new thoughts related to this, since they might be useful for proper distributions to think about too. I'll hasten to add that many of these thoughts are not originally mine. I shamelessly adopt any idea that I think is good. The core ideas for Baserock come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my colleagues and bosses at Codethink. At the recent GNOME Summit in Montreal, I was greatly influenced by Colin Walters. This is also not an advertisment for Baserock, but since my of my current thinking come from that project, I'll discuss things in the context of Baserock. Finally, I'm writing this to express my personal opinion and thoughts. I'm not speaking for anyone else, least of all Codethink. On the package abstraction Baserock abandons the idea of packages for all individual programs. In the early 1990s, when Linux was new, and the number of packages in a distribution could be counted on two hands in binary, packages were a great idea. It was feasible to know at least something about every piece of software, and pick the exact set of software to install on a machine. It is still feasible to do so, but only in quite restricted circumstances. For example, picking the packages to install a DNS server, or an NFS server, or a mail server, by hand, without using meta packages or tasks (in Debian terminology), is still quite possible. On embedded devices, there's also usually only a handful of programs installed, and the people doing the install can be expected to understand all of them and decide which ones to install on each specific device. However, those are the exceptions, and they're getting rarer. For most people, manually picking software to install is much too tedious. In Debian, we've realised this a great many years ago, and developed meta packages (whose only purpose is to depend on other packages) and tasks (solves the same problem, but differently). These make it possible for a user to say "I want to have the GNOME desktop environment", and not have to worry about finding every piece that belongs in GNOME, and install that separately. For much of the past decade, computers have had sufficient hard disk space that it is no longer necessary to be quite so picky about what to install. A new cheap laptop will now typically come with at least 250 gigabytes of disk space. An expensive one, with an SSD drive, will have at least 128 gigabytes. A fairly complete desktop install uses less than ten gigabytes, so there's rarely a need to pick and choose between the various components. From a usability point of view, choosing from a list of a dozen or two options is much easier than from a list of thirty five thousand (the number of Debian packages as I write this). This is one reason why Baserock won't be using the traditional package abstraction. Instead, we'll collect programs into larger collections, called strata, which form some kind of logical or functional whole. So there'll be one stratum for the core userland software: init, shell, libc, perhaps a bit more. There'll be another for a development environment, one for a desktop environment, etc. Another, equally important reason, to move beyond packages is the problems caused by the combinatorial explosion of packages and versions. Colin Walters talks about this very well. When every system has a fairly unique set of packages, and versions of them, it becomes much harder to ensure that software works well for everyone, that upgrades work well, and that any problems get solved. When the number of possible package combinations is small, getting the interactions between various packages right is easier, QA has a much easier time to test all upgrade paths, and manual test coverage improves a lot when everyone is testing the same versions. Even debugging gets easier, when everyone can easily run the same versions. Grouping software into bigger collections does reduce flexibility of what gets installed. In some cases this is important: very constrainted embedded devices, for example, still need to be very picky about what software gets installed. However, even for them, the price of flash storage is low enough that it might not matter too much, anymore. The benefit of a simpler overall system may well outweigh the benefit of fine-grained software choice. Everything in git In Baserock, we'll be building everything from source in git. It will not be possible to build anything, unless the source is committed. This will allow us to track, for each binary blob we produce, the precise sources that were used to bulid it. We will also try to achieve something a bit more ambitious: anything that affects any bit in the final system image can be traced to files committed to git. This means tracking also all configuration settings for the build, and the whole build environment, in git. This is important for us so that we can reproduce an image used in the field. When a customer is deploying a specific image, and needs it to be changed, we want to be able to make the change with the minimal changes compared to the previous version of the image. This requires that we can re-create the original image, from source, bit for bit, so that when we make the actual change, only the changes we need to make affect the image. We will make it easy to branch and merge not just individual projects, but the whole system. This will make it easy to do large changes to the system, such as transitioning to a new version of GNOME, or the toolchain. Currently, in Debian, such large changes need to be serialised, so that they do not affect each other. It is easy, for example, for a GNOME transition to be broken by a toolchain transition. Branching and merging has long been considered the best available solution for concurrent development within a project. With Baserock, we want to have that for the whole system. Our build servers will be able to build images for each branch, without requiring massive hardware investment: any software that is shared between branches only gets built once. Launchpad PPAs and similar solutions provide many of the benefits of branching and merging on the system level. However, they're much more work than "git checkout -b gnome-3.4-transition". I believe that the git based approach will make concurrent development much more efficient. Ask me in a year if I was right. Git, git, and only git There are a lot of version control systems in active use. For the sake of simplicity, we'll use only git. When an upstream project uses something else, we'll import their code into git. Luckily, there are tools for that. The import and updates to it will be fully automatic, of course. Git is not my favorite version control system, but it's clearly the winner. Everything else will eventually fade away into obscurity. Or that's what we think. If it turns out that we're wrong about that, we'll switch to something else. However, we do not intend to have to deal with more than one at a time. Life's too short to use all possible tools at the same time. Tracking upstreams closely We will track upstream version control repositories, and we will have an automatic mechanism of building our binaries directly from git. This will, we hope, make it easy to follow closely upstream development, so that when, say, GNOME developers make commits, we want to be able to generate a new system image which includes those changes the same day, if not within minutes, rather than waiting days or weeks or months. This kind of closeness is greatly enhanced by having everything in version control. When upstream commits changes to their version control system, we'll mirror them automatically, and this then triggers a new system image build. When upstream makes changes that do not work, we can easily create a branch from any earlier commit, and build images off that branch. This will, we hope, also make it simpler to make changes, and give them back to upstream. Whenever we change anything, it'll be done in a branch, and we'll have system images available to test the change. So not only will upstream be able to easily get the change from our git repository, they'll also be easily verify, on a running system, that the change fixes the problem. Automatic testing, maybe even test driven development We will be automatically building system images from git commits for Baserock. This will potentially result in a very large number of images. We can't possibly test all of them manually, so we will implement some kind of automatic testing. The details of that are still under discussion. I hope to be able to start adding some test driven development to Baserock systems. In other words, when we are requested to make changes to the system, I want the specification to be provided as executable tests. This will probably be impossible in real life, but I can hope. I've talked about doing the same thing for Debian, but it's much harder to push through such changes in an established, large project. Solving the install, upgrade, and downgrade problems All mainstream Linux distributions are based on packages, and they all, pretty much, do installations and upgrades by unpacking packages onto a running system, and then maybe running some scripts from the packages. This works well for a completely idle system, but not so well on systems that are in active use. Colin Walters again talks about this. For installation of new software, the problem is that someone or something may invoke it before it is fully configured by the package's maintainer script. For example, a web application might unpack in such a way that the web server notices it, and a web browser may request the web app to run before it is configured to connect to the right database. Or a GUI program may unpack a .desktop file before the executable or its data files are unpacked, and a user may notice the program in their menu and start it, resulting in an error. Upgrades suffer from additonal problems. Software that gets upgraded may be running during the upgrade. Should the package manager replace the software's data files with new versions, which may be in a format that the old program does not understand? Or install new plugins that will cause the old version of the program to segfault? If the package manager does that, users may experience turbulence without having put on their seat belts. If it doesn't do that, it can't install the package, or it needs to wait, perhaps for a very long time, for a safe time to do the upgrade. These problems have usually been either ignored, or solved by using package specific hacks. For example, plugins might be stored in a directory that embeds the program's version number, ensuring that the old version won't see the new plugins. Some people would like to apply installs and upgrades only at shutdown or bootup, but that has other problems. None of the hacks solve the downgrade problem. The package managers can replace a package with an older version, and often this works well. However, in many cases, any package maintainer scripts won't be able to deal with downgrades. For example, they might convert data files to a new format or name or location upon upgrades, but won't try to undo that if the package gets downgraded. Given the combinatorial explosion of package versions, it's perhaps just as well that they don't try. For Baserock, we absolutely need to have downgrades. We need to be able to go back to a previous version of the system if an upgrade fails. Traditionally, this has been done by providing a "factory reset", where the current version of the system gets replaced with whatever version was installed in the factory. We want that, but we also want to be able to choose other versions, not just the factory one. If a device is running version X, and upgrades to X+1, but that version turns out to be a dud, we want to be able to go back to X, rather than all the way back to the factory version. The approach we'll be taking with Baserock relies on btrfs and subvolumes and snapshots. Each version of the system will be installed in a separate subvolume, which gets cloned from the previous version, using copy-on-write to conserve space. We'll make the bootloader be able to choose a version of the system to boot, and (waving hands here) add some logic to be able to automatically revert to the previous version when necessary. We expect this to work better and more reliably than the current package based one. Making choices Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every init implementation, various implementations of /bin/sh, and we even have at least three entirely different kernels. Sometimes this is non-choice is a good thing. Our users may need features that only one of the kernels support, for example. And we certainly need to be able to provide both mysql and postresql, since various software we want to provide to our uses needs one and won't work with the other. At other times, the inability to choose causes trouble. Do we really need to support more than one implemenation of /bin/sh? By supporting both dash and bash for that, we double the load on testing and QA, and introduce yet another variable to deal with into any debugging situation involving shell scripts. Especially for core components of the system, it makes sense to limit the flexibility of users to pick and choose. Combinatorial explosion d j vu. Every binary choice doubles the number of possible combinations that need to be tested and supported and checked during debugging. Flexibility begets complexity, complexity begets problems. This is less of a problem at upper levels of the software stack. At the very top level, it doesn't really matter if there are many choices. If a user can freely choose between vi and Emacs, and this does not add complexity at the system level, since nothing else is affected by that choice. However, if we were to add a choice between glibc, eglibc, and uClibc for the system C library, then everything else in the system needs to be tested three times rather than once. Reducing the friction coefficient for system development Currently, a Debian developer takes upstream code, adds packaging, perhaps adds some patches (using one of several methods), builds a binary package, tests it, uploads it, and waits for the build daemons and the package archive and user-testers to report any problems. That's quite a number of steps to go through for the simple act of adding a new program to Debian, or updating it to a new version. Some of it can be automated, but there's still hoops to jump through. Friction does not prevent you from getting stuff done, but the more friction there is, the more energy you have to spend to get it done. Friction slows down the crucial hack-build-test cycle of software development, and that hurts productivity a lot. Every time a developer has to jump through any hoops, or wait for anything, he slows down. It is, of course, not just a matter of the number of steps. Debian requires a source package to be uploaded with the binary package. Many, if not most, packages in Debian are maintained using version control systems. Having to generate a source package and then wait for it to be uploaded is unnecessary work. The build daemon could get the source from version control directly. With signed commits, this is as safe as uploading a tarball. The above examples are specific to maintaining a single package. The friction that really hurts Debian is the friction of making large-scale changes, or changes that affect many packages. I've already mention the difficulty of making large transitions above. Another case is making policy changes, and then implementing them. An excellent example of that is in Debian is the policy change to use /usr/share/doc for documentation, instead of /usr/doc. This took us many years to do. We are, I think, perhaps a little better at such things now, but even so, it is something that should not take more than a few days to implement, rather than half a decade. On the future of distributions Occasionally, people say things like "distributions are not needed", or that "distributions are an unnecessary buffer between upstream developers and users". Some even claim that there should only be one distribution. I disagree. A common view of a Linux distribution is that it takes some source provided by upstream, compiles that, adds an installer, and gives all of that to the users. This view is too simplistic. The important part of developing a distribution is choosing the upstream projects and their versions wisely, and then integrating them into a whole system that works well. The integration part is particularly important. Many upstreams are not even aware of each other, nor should they need to be, even if their software may need to interact with each other. For example, not every developer of HTTP servers should need to be aware of every web application, or vice versa. (It they had to be, it'd be a combinatorial explosion that'd ruin everything, again.) Instead, someone needs to set a policy of how web apps and web servers interface, what their common interface is, and what files should be put where, for web apps to work out of the box, with minimal fuss for the users. That's part of the integration work that goes into a Linux distribution. For Debian, such decisions are recorded in the Policy Manual and its various sub-policies. Further, distributions provide quality assurance, particularly at the system level. It's not realistic to expect most upstream projects to do that. It's a whole different skillset and approach that is needed to develop a system, rather than just a single component. Distributions also provide user support, security support, longer term support than many upstreams, and port software to a much wider range of architectures and platforms than most upstreams actively care about, have access to, or even know about. In some cases, these are things that can and should be done in collaboration with upstreams; if nothing else, portability fixes should be given back to upstreams. So I do think distributions have a bright future, but the way they're working will need to change.

Maybe you've already heard, that Google is starting to charge a fee for using the Maps API (Google may decide you're an non-profit organization and waive the fees). That is, in my opinion, Google telling you to look for better alternatives. And thanks to the awesome effort of the OpenStreetMaps community, there is such an alternative. Thus today's addition to the mini-tips series will tell you, how you get an OpenStreetMaps-powered map on your website.

1. First you need to determine the coordinates, where you want to center, the map. You can do this with OpenRouteService.org. Just search for the address, right-click on the marker highlighting the result, and set it as e.g. your start point, that'll give you the coordinates in the start field on the left. Alternatively you can just move your pointer over a place on the map and have a look at the lower right corner, where the current coordinates for the pointer position are displayed.
2. Then you need to add a few things to the website, where the card is to be shown:
   • Include the OpenLayers script in the <head> of your webpage:

     <script src="http://www.openlayers.org/api/OpenLayers.js" type="text/javascript" />
     

   • Add a short ECMAScript snippet right after that, which will initialize your map:

     function osmInit()  
       document.getElementById("ol_map").style.display = 'block';
       var lonLat = new OpenLayers.LonLat(13.376096, 52.518598) // Center of the map
       .transform(
         new OpenLayers.Projection("EPSG:4326"),   // transform from WGS 1984
         new OpenLayers.Projection("EPSG:900913")  // to Spherical Mercator Projection
       );
       var map = new OpenLayers.Map("ol_map");
       var osmlayer = new OpenLayers.Layer.OSM();
       map.addLayer(osmlayer);
       map.setCenter(lonLat,
         16 // Zoom level
       );
      
     

     The first line is not needed, but I find it nice to hide the map on default, in case the user has disabled ECMAScript they haven't a large empty area somewhere on the website. The first line changes then the display value from "none" to "block". The rest is, I hope, already explained by the code itself.
   • Now only two more changes are needed, first you need to add onload="osmInit();" to your <body> tag. And afterwards an element (say a <p>) with the ID ol_map somewhere in the website <body>. You should set a width and height for that tag (style is your friend), otherwise the map might be bigger than you like.
3. Done. Enjoy your free (not just as in free beer) map!

You can, of course, customize your map further by adding additional layers and changing the available options (e.g. the zoom level). If you would want to mark a location on your map, you'd need to add something like

var markers = new OpenLayers.Layer.Markers("POI1");
map.addLayer(markers);
markers.addMarker(new OpenLayers.Marker(lonLat));

to your ECMAScript snipped after you added the OpenStreetMaps layer. These three lines would mark the location given by the variable lonLat. For further information, you can check out the OpenStreetMaps wiki and the OpenLayers documentation. In case you wonder, where my coordinates from this example point, you can check. Oh, and don't forget to consider a donation to OpenStreetMaps, so the project can pay the bills for its servers.

Augmented reality is available on all relatively modern smart phones. I ve played with it on my Android phone but it hasn t delivered the benefits that I hoped, there is a game where you can walk through a virtual maze which didn t work for me, and a bunch of programs which show me the position of stars, pizza restaurants, and other things which are cool but not really useful. It has been proven that larger screen size can make a surprising difference in productivity for increasing monitor size. The general concept seems to be that ideally everything you are thinking about at one time should be on the screen at once. I m not aware of any research comparing phones to desktop monitors but it is obvious that some tasks become extremely difficult or nearly impossible when attempted on the tiny screen of a phone. One significant example is coding. One noteworthy thing about coding is that the amount of typing is often quite small when compared to the amount of time spent looking at code, so the lack of good keyboard options on phones isn t always a serious problem. The iPhone 4 has a resolution of 640*960 which seems to be the best available phone resolution (with 480*854 being the highest resolution that is available in many phones). The Dell Streak at 5 inches seemed to have the largest screen in a phone, but they have stopped selling them. It seems that the largest screen available in a phone is about 4.2 inches. Probably the minimum that would be considered usable for development would be a resolution of about 1280*1024 and a screen size of about 14 inches, while opinion will vary a lot about this I think that the vast majority of programmers will agree that the bigger tablet computers and Netbooks (at about 10 inches and something like 1366*768 resolution) are well below the minimum size. It seems to me that a possible solution to this problem involves using augmented reality to provide a virtual desktop that is significantly larger and which has a significantly higher resolution. The advantage of augmented reality over merely scrolling is that it should allow faster and more reliable seeking for the section of virtual desktop that is of interest, and seek speed is probably the bottleneck with small monitors. One problem for this would be turning corners when on public transport, but the camera button could be used to reset the current phone position to be the middle of the viewing area, if the process of resetting the angle is fast enough it wouldn t be a great distraction. I don t think that a mobile phone will ever be a great device for software development and I don t think that the places where a serious computer isn t available are good places to work. But sometimes I get inspiration for tracking down a difficult bug when on the move and it would be really good to be able to read the code immediately. I won t have any time to work on such things myself. I m just publishing the idea in case someone who likes it happens to have a lot of spare time

A few years ago I got up on stage and briefly talked about how the Linux community contained far too many people who were willing to engage in entirely inappropriate behaviour, how this discouraged people from getting involved and how we weren't very good at standing up against that sort of behaviour. Despite doing this in front of several hundred people, and despite the video of me doing so then being uploaded to the internet, this got me a sum total of:

• No death threats
• No discussion about any of my physical attributes or lack thereof
• No stalkers
• No accusations that I was selling out the Linux community
• No accusations that I was a traitor to my gender
• No real negative feedback at all[1]

Which is, really, what you'd expect, right? The internet seems intent on telling me otherwise:

Well, she didn't do herself any favors by talking at conferences about women in tech, or setting up a feminist movement. If you wanted to attract abuse, that's a good way to go about it. It should be expected.
(Source)

MikeeUSA is a troll. He has no means to actually harm anyone, and he does it purely for the lulz.

Thus, MikeeUSA trolled a woman, and she took the bait. I just don't get why this is news, I've been trolled before, I don't get a news story.
(Source)

I was going to start a rant about how this behavior is encouraged by the macho men online, but this was just one guy harassing her. "Due to harassment" reads as due to harassment from the community, but she gave in to one idiot. She let him win.
(Source)

The full comment thread has rather more examples. If you stand up and say anything controversial, you should expect abuse. And if you let that abuse change your behaviour in any way, you've let the trolls win.

These attitudes are problematic.

The immediate assumption underlying such advice is that the degree of abuse is related to what you've said, not who you are. I'm reasonably visible in the geek world. I've said a few controversial things. The worst thing that's happened to me has been Ryan Farmer deciding to subscribe me to several thousand mailing lists. Inconvenient, but not really threatening. I haven't, for instance, been sent death threats. Nobody has threatened to rape me. And even if they had done, I wouldn't need to worry too much - there's a rather stronger track record of violent antifeminism being targeted at women than men.

I don't have to worry about this kind of thing. That means I don't get to tell other people that they should have expected it. Nor do I get to tell them that they should ignore it, or that if they don't call the police then they have no grounds to complain. And nor does anyone else.

The trolls don't win because someone decides that getting out of the tech business is more appealing than continuing to face abuse. The trolls win because we consider their behaviour acceptable and then blame the victim for inviting them in the first place. That needs to change.

[1] It was justifiably pointed out that saying all this while standing on stage next to a mostly naked guy wearing a loincloth with a raccoon tail covering his penis may have weakened my message somewhat.

comments

If you have been following Lintian s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests. The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a live data sample , which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong. The test suite has a somewhat unfair focus on auto-hint [3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations. Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve. On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):

// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    od
    return current
end

This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows. If you are interested in playing around with this, you can get sources from:

• britney2
  • Currently only works in stable (i.e. requires python2.5 and python-apt < 0.8 or so)
    • There some patches for this filed against release.debian.org.
  • See the INSTALL file for instructions
  • Adam s branch
    • use the p-u branch.
• SAT-britney
  • I haven t tested this one and I do not know the requirements here
• britney-tests
  • See the README file for instructions

Footnotes: [1] http://lists.debian.org/debian-release/2011/10/msg00178.html [2] These tests are auto-generated, so it is merely an up-scaled pattern . [3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in. [4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. [5] I assume the test called sat-britney-death (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the approve tpu package test case should fail. [6] A function that maps an archive into another archive erh, I mean, it maps a set of packages into another set of packages [7] http://en.wikipedia.org/wiki/Fixed_point_%28mathematics%29 Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing. As an example:
y depends on x
x in testing has RC bugs If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it. (Dis-)Proving my claim is an exercise left for the reader.

Looking through the Libravatar access logs, I found that most of the traffic we currently serve consists of 302 redirects to Gravatar. Optimising that path is therefore very important.

While Apache allows admins to provide custom error pages for things like 404 or 500, it's not quite that straightforward for 30x return codes.

Standard 301 / 302 responsesBy default, Apache (and most web servers out there) returns a fairly large HTML page along with a 30x redirection. Try it for yourself by disabling automatic redirections in Firefox (Preferences Advanced General Accessibility) or by installing the Request Policy add-on.

The 302 responses sent by Libravatar looked like this:

$ curl -i http://cdn.libravatar.org/avatar/12345678901234567890123456789012
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 01:51:52 GMT
Server: Apache
Cache-Control: max-age=86400
Location: http://www.gravatar.com/avatar/12345678901234567890123456789012.jpg?r=g&s=80&d=http://cdn.libravatar.org/nobody/80.png
Vary: Accept-Encoding
Content-Length: 310
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.gravatar.com/avatar/12345678901234567890123456789012.jpg?r=g&s=80&d=http://cdn.libravatar.org/nobody/80.png">here</a>.</p>
</body></html>

As you can see, the body of the response is just as large as the headers and isn't really necessary.

Body-less 301 responsesAfter reading about the ErrorDocument directive, I created an empty file called 302 in the root of the web server and included this directive in my vhost configuration file:

ErrorDocument 302 /302

which made the responses look like this:

$ curl -i http://example.com/redir
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 03:39:26 GMT
Server: Apache
Last-Modified: Wed, 21 Sep 2011 03:39:17 GMT
ETag: "8024d-0-4ad6b52201036"
Accept-Ranges: bytes
Content-Length: 0
Content-Type: text/plain

This one does have a completely empty body, however, there's an important problem with this solution: the Location header is missing! Not much point in reducing the size of the redirect if it's no longer working.

Custom 302 response pageThe next thing I tried (and ended up settling on) is this:

ErrorDocument 302 " "

which results in a 1-byte response (a single space) in the body:

$ curl -i http://example.com/redir
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 03:37:50 GMT
Server: Apache
Location: http://www.example.com
Vary: Accept-Encoding
Content-Length: 1
Content-Type: text/html; charset=iso-8859-1

There is still a little bit of unnecessary information in this response (character set, Vary and Server headers), but it's a major improvement over the original.

If you know of any other ways to reduce this further, please leave a comment!

A common question about hosting is whether to use a dedicated server or a virtual server. Dedicated Servers If you use a dedicated server then you will face the risk of problems which interrupt the boot process. It seems that all the affordable dedicated server offerings lack any good remote management, so when the server doesn t boot you either have to raise a trouble ticket with the company running the Data-Center (DC) or use some sort of hack. Hetzner is a dedicated server company that I have had good experiences with [1], when a server in their DC fails to boot you can use their web based interface (at no extra charge or delay) to boot a Linux recovery environment which can then be used to fix whatever the problem may be. They also charge extra for hands-on support which could be used if the Linux recovery environment revealed no errors but the system just didn t work. This isn t nearly as good as using something like IPMI which permits remote console access to see error messages and more direct control of rebooting. The up-side of a dedicated server is performance. Some people think that avoiding virtualisation improves performance, but in practice most virtual servers use virtualisation technologies that have little overhead. A bigger performance issue than the virtualisation overhead is the fact that most companies running DCs have a range of hardware in their DC and your system (whether a virtual server or a dedicated server) will be on a random system from their DC. I have observed hosting companies to give different speed CPUs and for dedicated servers different amounts of RAM for the same price. I expect that the disk IO performance also varies a lot but I have no evidence. As long as the hosting company provides everything that they offered before you sign the contract you can t complain. It s worth noting that CPU performance is either poorly specified or absent in most offers and disk IO performance is almost never specified. One advantage of dedicated servers in this regard is that you get to know the details of the hardware and can therefore refuse certain low spec hardware. The real performance benefit of a dedicated server is that disk IO performance won t be hurt by other users of the same system. Disk IO is the real issue as CPU and RAM are easy to share fairly but disk performance is difficult to share and is also a significant bottleneck on many servers. Dedicated servers also have a higher minimum price due to the fact that a real server is being used which involves hardware purchase and rack space. Hetzner s offers which start at E29 per month are about as cheap as it s possible to get. But it appears that the E29 offer is for an old server new hardware starts at E49 per month which is still quite cheap. But no dedicated server compares to the virtual servers which can be rented for prices less than $10 per month. Virtual Servers A virtual server will typically have an effective management interface. You should expect to get web based access to the system console as well as ssh console access. If console access is not sufficient to recover the system then there is an option to boot from a recovery device. This allows you to avoid many situations that could potentially result in down-time and when things go wrong it allows you to recover faster. Linode is an example of a company that provides virtual servers and provides a great management interface [2]. It would take a lot of work with performance monitoring and graphing tools to give the performance overview that comes for free with the Linode interface. Disk IO performance can suck badly on virtual servers and it can happen suddenly and semi-randomly. If someone else who is using a virtual server on the same hardware is the target of a DoS attack then your performance can disappear. Performance for CPU is generally fairly reliable though. So a CPU bound server would be a better fit on the typical virtual server options than a disk IO bound server. Virtual servers are a lot cheaper at the low end so if you don t need the hardware capabilities of a minimal dedicated server (with 1G of RAM for Hetzner and a minimum of 8G of RAM for some other providers) then you can save a lot of money by getting a virtual server. Finally the options for running a virtual machine under a virtual machine aren t good, AFAIK the only options that would work on a commercial VPS offering are QEMU (an x86 CPU instruction emulator), Hercules (a S/370 S/390, and Z series IBM mainframe emulator), and similar CPU emulators. Please let me know if there are any other good options for running a virtual machine on a VPS. Now while these emulators are apparently good for debugging OS development they aren t something that are generally useful for running a virtual machine. I knew someone who ran his important servers under Hercules so that x86 exploits couldn t be used for attacking them, but apart from that CPU emulation isn t generally useful for servers. Summary If you want to have entire control of the hardware or if you want to run your own virtual machines that suit your needs (EG one with lots of RAM and another with lots of disk space) then a dedicated server is required. If you want to have minimal expense or the greatest ease of sysadmin use then a virtual server is a better option. But the cheapest option for virtual hosting is to rent a server from Hetzner, run Xen on it, and then rent out DomUs to other people. Apart from the inevitable pain that you experience if anything goes wrong with the Dom0 this is a great option. As an aside, if anyone knows of a reliable company that offers some benefits over Hetzner then please let me know. What I would Like to See There is no technical reason why a company like Linode couldn t make an offer which was a single DomU on a server taking up all available RAM, CPU, and disk space. Such an offer would be really compelling if it wasn t excessively expensive. That would give Linode ease of management and also a guarantee that no-one else could disrupt your system by doing a lot of disk IO. This would be really easy for Linode (or any virtual server provider) to implement. There is also no technical reason why a company like Linode couldn t allow their customers to rent all the capacity of a physical system and then subdivide it among DomUs as they wish. I have a few clients who would be better suited by Linode DomUs that are configured for their needs rather than stock Linode offerings (which never seem to offer the exact amounts of RAM, disk, or CPU that are required). Also if I had a Linode physical server that only had DomUs for my clients then I could make sure that none of them had excessive disk IO that affected the others. This would require many extra features in the Linode management web pages, so it seems unlikely that they will do it. Please let me know if there is someone doing this, it s obvious enough that someone must be doing it. Update: A Rimuhosting employee pointed out that they offer virtual servers on dedicated hardware which meets this criteria [3]. Rimuhosting allows you to use their VPS management system for running all the resources in a single server (so no-one else can slow your VM down) and also allow custom partitioning of a server into as many VMs as you desire.

• [1] http://www.hetzner.de/
• [2] http://www.linode.com/
• [3] https://rimuhosting.com/order/vps-on-dedicated.jsp

I received recently a mail about my attempt to provide Grsecurity kernels in Debian. The sender found the bug by accident, and asked me why I didn't do some more publicity here. So here we are. I won't go into details on what grsecurity is, it's fairly complex. But it's basically a hardening patch for the Linux kernel, with three main components:

• the PaX patch, which purpose is to harden the memory layout of the Linux kernel and improve existing options: enforcing of non-executable memory pages (userland and in kernel), W^X (no page marked as writable and executable), ASLR, prevention of invalid userland pointers dereference, copies between userland and kernel memory
• RBAC (Role Based Access Control), an implementation of Mandatory Access Control
• various hardening features: /proc restrictions, chroot restrictions, kernel symbols hiding etc.

A lot of this touches low level stuff in the kernel, especially memory management. Ideally this patch would be pushed upstream, but Brad Spengler (grsecurity main developper) already said he wasn't interested in upstreaming it and upstream already said the patch was too huge and invasive to include it like that (especially since the original authors aren't interested in maintaining it upstream). There's an ongoing effort to split the patch and merge things little by little, but in the meanwhile having a mid-term solution would be nice. I know Debian users rebuilding grsecurity-patched kernels themselves, and I know some of them would appreciate having them included in the Debian kernel. Fortunately, the linux-2.6 source package has a nice feature which is called featureset. Basically it's a way to build some (binary) packages using a different set of patches and a different config. For example this was used to provide xen/openvz/vserver patchsets, and is now used to provide rt kernels. So I though it'd be nice to provide a grsec featureset, and starting doing the work. I have a working setup for producing those kernels, so I've opened a wishlist bug against the kernel (#605090) to have this merged. Those packages follow the sid kernel. There's an ongoing work for Squeeze, but it's a bit harder there because both the grsecurity patchset and the Debian kernel ship a whole lot of backports to the Linux kernel, meaning the grsecurity patch doesn't apply directly to the Debian source package. Basically I need to remove some of the hunks (since they are already applied to the source) and port some others (since there are some backported code not present in the vanilla 2.6.32, for example the drm code). Until the patches are merged and the bug is closed, I host some of the built packages at:

deb http://molly.corsac.net/~corsac/debian/kernel-grsec/packages/ sid/ The repository is signed by my key which you can add to your apt setup using apt-key add. If you want to rebuild the packages yourself, here's the method:

mkdir kernel-grsec
cd kernel-grsec
svn checkout svn://svn.debian.org/svn/kernel/dists/sid/linux-2.6
git clone git://anonscm.debian.org/users/corsac/grsec-patches.git
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.bz2
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.bz2.sign
gpg --verify linux-3.0.tar.bz2
cd linux-2.6
apt-get build-dep linux-2.6
export QUILT_PATCHES=../grsec-patches
quilt push -a
python debian/bin/genorig.py ../linux-3.0.tar.bz2
debian/rules orig
fakeroot debian/rules source
fakeroot make -f debian/rules.gen binary-arch_amd64_grsec_amd64 You could also do dpkg-buildpackage, pdebuild or whatever. Kernel handbook is a nice reading too if you want more information on how to rebuild Debian kernels. The quilt push -a may fail if you checkout an svn version more recent than mine. I try to keep patches up to date but I usually have some delay. Note that installing the kernel will require installing linux-grsec-base package. Binary is not yet available on my mirror but you can easily build it. Source can be found on git.debian.org. If you're interested by this, don't hesitate to mail me or the bug.

I ve recently been to the LibreOffice Conference 2011 in Paris. It s very nice to finally meet in person all those names from the malinglist and the bug reports. Although meeting most people for the first time, I left the conference with a strong feeling of a community (which I m proud to be a member of), this is a very motivating feeling. During the conference I had the chance to work on LibreOffice related issues:

• Hebrew translation for version 3.5 (up by 2%).
• Fixes to the Hebrew translation for version 3.4 (noticed a few string changes in 3.3 not included in 3.4).
• Patch to fix a small typo in the code (noticed while trying to find the meaning of the string before translating).
• Patch to fix the appearance of the BrOffice logo in RTL UI instead of LibreOffice (fix to the 3.3 branch, already fixed in 3.4).
• 3 patches (1, 2, 3) to cleanup unused .chaos file types with Michael Meeks s encouragement and help with doing some digging in the code.
  Apparently, even non programmers can write patches with the appropriate help.
• Check about 30 RTL and Hebrew related bug reports for my RTL issues presentation, verifying them with 3.4.3.

The 3 patches began when I noticed a few weird strings to translate in the context of installation media good luck to those who will try to install LibreOffice from 5.25 disk, 3.5 disk or a tape drive. These 3 strings are now removed from the code with other 36 unused strings. So translators please take the strings to translate with a grain of salt. Post conference tasks (always ending up with more stuff to do after a conference):

• Recreate my RTL issues presentation so that the live example of bugs could be presented as slides.
• Keep advancing with the Hebrew translation (3.5 and 3.4 fixes).
• Finish the fdo bugzilla search for RTL bugs. I found out some other people who reports bugs (and try to coordinate with them).
  A list of RTL related bugs in LibreOffice I know of is here: http://wiki.hamakor.org.il/index.php/ _ _ (still work in progress, in Hebrew).
• Create a meta bug for the RTL bugs, and prioritize the issues to focus on what s important to make sure is fixed for version 3.5.
• Start testing version 3.5 for RTL bugs (daily / weekly builds).
• Help Christoph Noack with finding which PaperCuts we have in LibreOffice and try to get them fixed.

Filed under: LibreOffice, Openoffice.org

Rapha l Hertzog recently announced a new dpkg-buildflags interface in dpkg that at long last gives the distribution, the package maintainers, and users the control they want over the build flags used when building packages. The announcement mail gives all the gory details about how to invoke dpkg-buildflags in your build to be compliant; but the nice thing is, if you're using dh(1) with debian/compat=9, debhelper does it for you automatically so long as you're using a build system that it knows how to pass compiler flags to. So for the first time, /usr/share/doc/debhelper/examples/rules.tiny can now be used as-is to provide a policy-compliant package by default (setting -g -O2 or -g -O0 for your build regardless of how debian/rules is invoked). Of course, none of my packages actually work that way; among other things I have a habit of liberally sprinkling DEB_MAINT_CFLAGS_APPEND := -Wall in my rules, and sometimes DEB_LDFLAGS_MAINT_APPEND := -Wl,-z,defs and DEB_CFLAGS_MAINT_APPEND := $(shell getconf LFS_CFLAGS) as well. And my upstreams' build systems rarely work 100% out of the box with dhauto* without one override or another somewhere. So in practice, the shortest debian/rules file in any of my packages seems to be 13 lines currently. But that's 13 lines of almost 100% signal, unlike the bad old days of cut'n'pasted dh_* command lists. The biggest benefit, though, isn't in making it shorter to write a rules file with the old, standard build options. The biggest benefit is that dpkg-buildflags now also outputs build-hardening compiler and linker flags by default on Debian. Specifically, using the new interface lets you pick up all of these hardening flags for free:

-fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wl,-z,relro

It also lets you get -fPIE and -Wl,-z,now by adding this one line to your debian/rules (assuming you're using dh(1) and compat 9):

export DEB_BUILD_MAINT_OPTIONS := hardening=+pie,+bindnow

Converting all my packages to use dh(1) has always been a long-term goal, but some packages are easier to convert than others. This was the tipping point for me, though. Even though debhelper compat level 9 isn't yet frozen, meaning there might still be other behavior changes to it that will make more work for me between now and release, over the past couple of weekends I've been systematically converting all my packages to use it with dh. In particular, pam and samba have been rebuilt to use the default hardening flags, and openldap uses these flags plus PIE support. (Samba already builds with PIE by default courtesy of upstream.) You can't really make samba and openldap out on the graph, but they're there (with their rules files reduced by 50% or more). I cannot overstate the significance of proactive hardening. There have been a number of vulnerabilities over the past few years that have been thwarted on Ubuntu because Ubuntu is using -fstack-protector by default. Debian has a great security team that responds quickly to these issues as soon as they're revealed, but we don't always get to find out about them before they're already being exploited in the wild. In this respect, Debian has lagged behind other distros. With dpkg-buildflags, we now have the tools to correct this. It's just a matter of getting packages to use the new interfaces. If you're a maintainer of a security sensitive package (such as a network-facing daemon or a setuid application), please enable dpkg-buildflags in your package for wheezy! (Preferably with PIE as well.) And if you don't maintain security sensitive packages, you can still help out with the hardening release goal.

A couple of times I have written code which basically does this:

1. accept() an incoming connection
2. do some magic to get another socket
3. start passing bit between the two sockets
4. perform a correct close down sequence

An example use case of this is a proxy implementing the HTTP CONNECT method, but in for some known hostnames it will log a message and mangle the hostname before proceeding. This has been used as a legacy fall back solution while changing a network setup. But my uses are not restricted to HTTP proxies I have done the same for a few legacy protocols where the magic has been of different complexities. The two final steps are quite general and it would be nice to have a module doing just that. Take two sockets and make it easy (or even automatic) to pass bytes between them. The na ve non-blocking solution would use a scalar string buffer for each direction and perform a select loop while maintaining the write vector depending on which buffers contain data. I have written this code multiple times. In development this is usually quite successful, in production less so. While Perl might be quite suited for the magic in step 2, the na ve way of passing bytes have quite an overhead for the buffer management. A less na ve way would use a array of strings for buffers, but I m not quite sure if this would be a win in all cases. You might be able to get away with some string operations on the read side of the buffer, but it might be more expensive on the write side. I have not benchmarked this. Most of the time I don t care about Perl level IO handles. I know that there is a real C level file descriptor beneath. So an even better POSIX compliant solution might be to use XS to have plain C strings and use readv()/writev() and a iovec structure as buffer. Can we do even better? At least on Linux we can. With the Linux splice() specific system call it is possible to us a pipe as buffer and never to have to copy data from and to user space. I have not been able to find any off the shelf solution on CPAN. So I think I need to write it myself, but what would the nice and general API be? I guess the basic interface would be something like:

    my $chain = IO::Splice->new($fh1, $fh2);
    $chain->pump(); # read and write from both handles if possible and needed
    $chain->read($fh1);  # read to buffer from one specific handle
    $chain->write($fh2); # write from buffer to one specific handle
    $chain->can_write(); # returns the handles it needs to write to

but it might be simpler to have two callbacks for setting a file handle in write or no-write state:

    my $readset  = IO::Select->new( $fh1, $fh2);
    my $writeset = IO::Select->new();
    my $chain = IO::Splice->new( $fh1, $fh2,
        writable => sub    $writeset->add( shift )  ,
        unwritable => sub   $writeset->remove( shift )  
    );
    while ( ... select ... )  
        $chain->pump();
     

As said, I think I have plenty of implementations of the na ve way but before releasing some code it would be nice to get some input on the API. But the best feedback would be a module that already have a usable API but might not implement the Linux specific way. That would allow me to steal the interface

No, my blog isn't dead, and neither is me. It's just that way too many things happened since this year's debconf that got me a bit off tracks. I managed to do daily business like keeping my packages in shape and the backports queue low, and that was mostly it. No clue if that will change anytime soon, but I guess I would like to keep you updated with an event where you can meet me next week: There will be the Global 2000 Birthday Party going on in the WUK on Thursday 20th, so if you happen to be in Vienna at that time, drop by and enjoy some great bands. ... which brings me to one of the local bands from Vienna: Heinz aus Wien. They are around for well over 10 years now and are still rocking quite well. Here are some examples of their songs, like always:

• Wir sind immer noch: A sort of love song for this great town, Vienna.
• Lieb im Prinzip: A very pithy song about relationships.
• Ich hab mit Tocotronic Bier getrunken: Quite funny song from their early days, and interestingly, people regularly claim that it was done by a different band ...

Like always, enjoy!

/music permanent link Comments: 0 Flattr this

I have recently been working on some code for NetSurf which builds up an output buffer by repeatedly calling snprintf(); No great shock there, well understood trivial pattern that has been used repeatedly for ages. Gosh I feel silly now, in fact I was so convinced I was right I wrote a program to "prove" it.