Search Results: "nate"

3 November 2011

Andrew Pollock: [life] Zoe at 18 months

Wow, a lot (of travel) has happened in the last three months. Zoe has been to Chicago, Langkawi, Brisbane, and is now back in Mountain View. I'm actually feeling really guilty about our carbon footprint for this year. She spent the last about 3.5 weeks in Brisbane with Sarah after the wedding in Langkawi, until I flew there last weekend for my sister's wedding. Losing and finding Smudge helped pass the time, as did having Liam stay with me while he was visiting from Zurich. I could have done without the former, but really appreciated the timing of the latter. There was quite a noticeable vocabulary difference when I finally saw Zoe again. I can't keep up with all the words she can say now, and she's starting to grasp concepts as well. For example, she opened a small cardboard box, and declared it "empty". Apparently we say "uh oh" a lot, as this is one of her favourite things to blurt out whenever anything happens, ranging from my sister's bridal train dragging on the ground at her wedding, to vomiting all over herself in the back of the car. She's definitely not going to be a girly girl. When the flight from Brisbane landed in Los Angeles she yelled out a big "WHOA!" and then "Again!" while furiously doing the baby sign language for "more". Fortunately we were able to oblige for the flight from LAX to San Jose, and she again enjoyed the landing (she got to sit in her own seat for this one) and did a loud "Wow!" and "Again!" I'm looking forward to taking her on her first roller coaster ride. Her canine teeth still haven't come through yet, but I think they've just started to make an appearance. We're still trying to de-jetlag her at the moment, so it's unclear if the wake-ups throughout the night are because of jetlag or teeth issues (or a combination of both). She handled all of the travel fairly well, although she's now too long for a aeroplane bassinet. She didn't get a lot of sleep on the 12.5 hour flight from Brisbane to Los Angeles (which meant we got none), but she was pretty well behaved for the bulk of it. She's good enough on her legs now that she can just tear up and down the aisles without incident, so we made countless laps of the plane to keep her occupied. The return flight was on Halloween, and the cabin crew had gotten into the spirit with temporary face tattoos, some cabin decorations and some "costumes". The trip back home was actually epic in that it was littered with minor annoyances, but I'll save that for a separate blog post. This Christmas should be a fun one, as she'll be old enough to be a bit more interactive about the whole thing. She may also freak out sitting on Santa's knee, so we may not get quite as nice a photo as we got last year.

Sandro Tosi: Print a NUL-terminated string with awk

I thought it would have been easier to print a NUL-terminated string in awk (mawk as it's the default in Debian), but after some trial-and-fail I was able to come up with this kinda ugly solution:


$ echo -e "123\n456" awk 'BEGIN ORS="" print $0 ; printf("%c", "") ' xargs -n1 -0 echo
123
456

That:

There's really no better way to do that?

2 November 2011

Andrew McMillan: Teaching our children to lie

My son is being encouraged to lie. It's a fairly regular occurrence around here, and I'm sure you've seen it yourself. It's that checkbox on the website you're visiting where you say "Yes, I am over X years of age".

How could you not trust this child?

It's well-known that there are 10 year olds on Facebook, and everyone agrees that they lied to get those accounts, but nobody really cares much. Facebook don't care. Facebook just see another pair of eyeballs, and another node in their social graph, but actually Fraser, Max (and even Heather) aren't nodes on that particular graph. Today's annoyance was that Fraser's Youtube account has been blocked. That just seems silly. It's an open website where anyone can read pretty much anything, except that to have an account on there you have to be older than X years (I haven't bothered to discover how old X is - it's not relevant). So it seems there's this website, worth many billions of dollars and these guys can only control the functionality available to signed up members with an "on/off" switch. Is it bad if under 12 year olds post comments in a public forum? Oh, we should not let them create accounts then. Is it bad if under 8 year olds see soft porn? Oh, if we stop them creating accounts they'll be safe from that! What about the 14 year old with the videos of her cat? Well we definitely, definitely don't want that! That's ludicrous! Can these people not design a website to accept a child? Someone who might want to log into the website just like mummy and daddy do? Then, when the honest child is logged in, perhaps the account can be linked to a parent's account, or perhaps it could just be restricted in different ways. Let's see what they say an account on Youtube offers you:
  1. Subscribe to your favorite channels.
  2. Rent or purchase top Hollywood movies
  3. Save videos to watch later
  4. Get recommendations based on what you've watched
  5. Share videos you like on Facebook, Twitter, and more
  6. Find your Facebook, Yahoo, Hotmail and Gmail contacts on YouTube
  7. Watch private videos from friends and family
OK, so as far as I can see:
  1. Might be useful to any age person, and is entirely innocuous in the context of the
    content of Youtube which is available to anonymous visitors of any age.
  2. Well, Fraser wouldn't be able to pay, since he doesn't have anything to pay
    with. I guess he could steal my credit card, but if he's already at that level
    of skullduggery then some silly checkbox would be "child's play" to his lying
    skillz.
  3. Entirely innocuous.
  4. Seems pretty innocuous too.
  5. Presumably he wouldn't because 10 year old's can't have identities. And this
    functionality could easily be restricted, in any case
  6. And if there are no relationships, there are no friends. Maybe there is family,
    which is fine, I guess. I think the risks here are controllable.
So there's plenty of value to a 10 year old in having an account. There's nothing that most reasonable parents would be concerned about, if there was a clear policy of limited functionality in place that parents could see and have confidence in. The Youtube thing actually seems to be related to (maybe) to Google's recent push towards "identity". Children younger than X are not allowed to have an on-line identity, because they might run amok on the intarwebs. Or see scary stuff. Or something. And yet, by denying them an account, they are removing any ability to actually apply a level of control that reflects the presumed maturity (since, after all, age is no more a direct measure of maturity than height). Truly though, what totally pisses me off about this situation, is that Fraser associated a bunch of channels with his Youtube account, and he receives a weekly digest of the activity. Unfortunately we can't log in to turn it off.

1 November 2011

Vincent Bernat: SSL computational DoS mitigation

Some days ago, a hacker group, THC, released a denial of service tool for SSL web servers. As stated in its description, the problem is not really new: a complete SSL handshake implies costly cryptographic computations. There are two different aspects in the presented attack:

Mitigation techniques There is no definitive solution to this attack but there exists some workarounds. Since the DoS tool from THC relies heavily on renegotiation, the most obvious one is to disable this mechanism on the server side but we will explore other possibilities.

Disabling SSL renegotiation Tackling the second problem seems easy: just disable SSL renegotiation. It is hardly needed: a server can trigger a renegotiation to ask a client to present a certificate but a client usually does not have any reason to trigger one. Because of a past vulnerability in SSL renegotiation, recent version of Apache and nginx just forbid it, even when the non-vulnerable version is available. openssl s_client can be used to test if SSL renegotiation is really disabled. Sending R on an empty line trigger renegotiation. Here is an example where renegotiation is disabled (despite being advertised as supported):
$ openssl s_client -connect www.luffy.cx:443 -tls1
[...]
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
[...]
R
RENEGOTIATING
140675659794088:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:
Disabling renegotation is not trivial with OpenSSL. As an example, I have pushed a patch to disable renegotiation in stud, the scalable TLS unwrapping daemon.

Rate limiting SSL handshakes Disabling SSL renegotiation on the client side is not always possible. For example, your web server may be too old to propose such an option. Since those renegotiations should not happen often, a workaround is to limit them. When the flaw was first advertised, F5 Networks provided a way to configure such a limitation with an iRule on their load-balancers. We can do something similar with just Netfilter. We can spot most TCP packets triggering such a renegotiation by looking for encrypted TLS handshake record. They may happen in a regular handshake but in this case, they usually are not at the beginning of the TCP payload. There is no field saying if a TLS record is encrypted or not (TLS is stateful for this purpose). Therefore, we have to use some heuristics. If the handshake type is unknown, we assume that this is an encrypted record. Moreover, renegotiation requests are usually encapsulated in a TCP packet flagged with push .
# Access to TCP payload (if not fragmented)
payload="0 >> 22 & 0x3C @ 12 >> 26 & 0x3C @"
iptables -A LIMIT_RENEGOCIATION \
    -p tcp --dport 443 \
    --tcp-flags SYN,FIN,RST,PSH PSH \
    -m u32 \
    --u32 "$payload 0 >> 8 = 0x160300:0x160303 && $payload 2 & 0xFF = 3:10,17:19,21:255" \
    -m hashlimit \
    --hashlimit-above 5/minute --hashlimit-burst 3 \
    --hashlimit-mode srcip --hashlimit-name ssl-reneg \
    -j DROP
The use of u32 match is a bit difficult to read. The manual page gives some insightful examples. $payload allows to seek for the TCP payload. It only works if there is no fragmentation. Then, we check if we have a handshake (0x16) and if we recognise TLS version (0x0300, 0x0301, 0x0302 or 0x0303). At least, we check if the handshake type is not a known value. There is a risk of false positives but since we use hashlimit, we should be safe. This is not a bullet proof solution: TCP fragmentation would allow an attacker to evade detection. Another equivalent solution would be to use CONNMARK to record the fact the initial handshake has been done and forbid any subsequent handshakes. If you happen to disable SSL renegociation, you can still use some Netfilter rule to limit the number of SSL handshakes by limiting the number of TCP connections from one IP:
iptables -A LIMIT_SSL \
    -p tcp --dport 443 \
    --syn -m state --state NEW \
    -m hashlimit \
    --hashlimit-above 120/minute --hashlimit-burst 20 \
    --hashlimit-mode srcip --hashlimit-name ssl-conn \
    -j DROP
Your servers will still be vulnerable to a large botnet but if there is only a handful of source IP, this rule will work just fine1. I have made all those solutions available in a single file.

Increasing server-side power processing SSL can easily be scaled up and out. Since SSL performance increases linearly with the number of cores, scaling up can be done by throwing in more CPU or more cores per CPU. Adding expensive SSL accelerators would also do the trick. Scaling out is also relatively easy but you should care about SSL session resume.

Putting more work on the client side In their presentation of the denial of service tool, THC explains:
Establishing a secure SSL connection requires 15 more processing power on the server than on the client.
I don t know where this figure comes from. To check it, I built a small tool to measure CPU time of a client and a server doing 1000 handshakes with various parameters (cipher suites and key sizes). The results are summarized on the following plot: Plot to compare computational power required by servers and clients For example, with 2048bit RSA certificates and a cipher suite like AES256-SHA, the server needs 6 times more CPU power than the client. However, if we use DHE-RSA-AES256-SHA instead, the server needs 34% less CPU power. The most efficient cipher suite from the server point of view seems to be something like DHE-DSS-AES256-SHA where the server needs half the power of the client. However, you can t really uses those shiny cipher suites:
  1. Some browsers do not support them: they are limited to RSA cipher suites2.
  2. Using them will increase your regular load a lot. Your servers may collapse with just legitimate traffic.
  3. They are expensive for some mobile clients: they need more memory, more processing power and will drain battery faster.
Let s dig a bit more on why the server needs more computational power in the case of RSA. Here is a SSL handshake when using a cipher suite like AES256-SHA: SSL full handshake When sending the Client Key Exchange message, the client will encrypt TLS version and 46 random bytes with the public key of the certificate sent by the server in its Certificate message. The server will have to decrypt this message with her private key. Those are the two most expensive operations in the handshake. Encryption and decryption are done with RSA (because of the selected cipher suite). To understand why decryption is more expensive than encryption, let me explain how RSA works. First, the server needs a public and a private key. Here are the main steps to generate them:
  1. Pick two random distinct prime numbers p and q , each roughly the same size.
  2. Compute n=pq . It is the modulus.
  3. Compute \varphi(n)=(p-1)(q-1) .
  4. Choose an integer e such that 1<e<\varphi(n) and \gcd(\varphi(n),e) = 1 (i.e. e and \varphi(n) are coprime). It is the public exponent.
  5. Compute d=e^ -1 \mod\varphi(n) . It is the private key exponent.
The public key is (n,e) while the private key is (n,d) . A message to be encrypted is first turned into an integer m<n (with some appropriate padding). It is then encrypted to a ciphered message c with the public key and should only be decrypted with the private key:
  • c=m^e\mod n (encryption)
  • m=c^d\mod n (decryption)
So, why is decryption more expensive? In fact, the key pair is not really generated like I said above. Usually, e is a small fixed prime number with a lot of 0, like 17 (0x11) or 65537 (0x10001) and p and q are choosen such that \varphi(n) is coprime with e . This allows encryption to be fast using exponentiation by squaring. On the other hand, its inverse d is a big number with no special property and therefore, exponentiation is more costly and slow. Instead of computing d from e , it is possible to choose d and compute e . We could choose d to be small and coprime with \varphi(n) and then compute e=d^ -1 \mod\varphi(n) and get blazingly fast decryption. Unfortunately, there are two problems with this: Therefore, we cannot use a small private exponent. The best we can do is to choose the public exponent to be e =4294967291 (the biggest prime 32bit number and it contains only one 0). However, there is no change as you can see on our comparative plot. To summarize, no real solution here. You need to allow RSA cipher suites and there is no way to improve the computational ratio between the server and the client with such a cipher suite.

Things get worse Shortly after the release the denial of service tool, Eric Rescorla3 published a good analysis on the impact of such a tool. He asks himself about the efficiency to use renegotiation for such an attack:
What you should be asking at this point is whether a computational DoS attack based on renegotiation is any better for the attacker than a computational DoS attack based on multiple connections. The way we measure this is by the ratio of the work the attacker has to do to the work that the server has to do. I ve never seen any actual measurements here (and the THC guys don t present any), but some back of the envelope calculations suggest that the difference is small. If I want to mount the old, multiple connection attack, I need to incur the following costs:
  1. Do the TCP handshake (3 packets)
  2. Send the SSL/TLS ClientHello (1 packet). This can be a canned message.
  3. Send the SSL/TLS ClientKeyExchange, ChangeCipherSpec, Finished messages (1 packet). These can also be canned.
Note that I don t need to parse any SSL/TLS messages from the server, and I don t need to do any cryptography. I m just going to send the server junk anyway, so I can (for instance) send the same bogus ClientKeyExchange and Finished every time. The server can t find out that they are bogus until it s done the expensive part. So, roughly speaking, this attack consists of sending a bunch of canned packets in order to force the server to do one RSA decryption.
I have written a quick proof of concept of such a tool. To avoid any abuse, it will only work if the server supports NULL-MD5 cipher suite. No sane server in the wild will support such a cipher. You need to configure your web server to support it before using this tool. While Eric explains that there is no need to parse any SSL/TLS messages, I have found that if the key exchange message is sent before the server send the answer, the connection will be aborted. Therefore, I quickly parse the server s answer to check if I can continue. Eric also says a bogus key exchange message can be sent since the server will have to decrypt it before discovering it is bogus. I have choosen to build a valid key exchange message during the first handshake (using the certificate presented by the server) and replay it on subsequent handshakes because I think the server may dismiss the message before the computation is complete (for example, if the size does not match the size of the certificate). With such a tool and 2048bit RSA certificate, a server is using 100 times more processing power than the client. Unfortunately, this means that most solutions, except rate limiting, exposed on this page may just be ineffective.

  1. However, since this rule relies on source IP to identify the attacker, the risk of false positive is real. You can slow down legitimate proxies, networks NATed behind a single IP, mobile users sharing an IP address or people behind a CGN.
  2. Cipher suites supported by all browsers are RC4-MD5, RC4-SHA and 3DES-SHA. Support for DHE-DSS-AES256-SHA requires TLS 1.2 (not supported by any browser).
  3. Eric is one of the author of several RFC related to TLS. He knows his stuff.

Michal &#268;iha&#345;: New Pootle on l10n.cihar.com

Translation server for phpMyAdmin and Gammu/Wammu has been updated to run latest Pootle. I had to do this upgrade to be able to run on latest Django, which is used by other websites as well (this blog is one of them). Unfortunately it was not as easy as running apt-get upgrade as Pootle package has been dropped from Debian recently. So I had to make quick and dirty packages myself (no removal of embedded software and so on). Installing newer version lead to major breakage as the database upgrade silently failed and I had to recover manually. In the end I've dropped all pootle_* tables and recreated them again. Unfortunately this lead to loss of some data from user profiles, but I could not see other way around. Anyway the service should be now back running in normal shape, so go ahead and translate :-).

Filed under: Debian English Phpmyadmin 0 comments Flattr this!

Francois Marier: Adding X-Content-Security-Policy headers in a Django application

Content Security Policy is a proposed HTTP extension which allows websites to restrict the external content that can be displayed by visiting web browsers. By expressing a set of rules to be enforced by the browser, a website is able to prevent the injection of outside resources by malicious users.

While adding support for the March 2011 draft in Libravatar, I looked at three different approaches.

Controlling the headers in the applicationThe first approach I considered was to have the Django application output all of the headers, which is what the django-csp module does. Unfortunately, I need to be able to vary the policy between pages (the views in Libravatar have different requirements) and that's one of the things that hasn't been implemented yet in that module.

Producing the same headers by hand is fairly simple:
response = render_to_response('app/view.html')
response['X-Content-Security-Policy'] = "allow 'self'"
return response
but it would mean adding a bit of code to every view and/or writing a custom wrapper for render_to_response().

Setting a default header in ApacheIdeally, I'd like to be able to set a default header in Apache using mod_headers and then override it as needed inside the application.

The first problem with this solution is that it's not possible (as far I can tell) for a Django application to override a header set by Apache:
The second problem is that mod_headers doesn't have an action that adds/sets a header only if it didn't already exist. It does have append and merge actions which could in theory be used to add extra terms to the policy but it unfortunately uses a different separator (the comma) from the CSP spec (which uses semi-colons).

Always set headers in ApacheWhile I would have liked to get the second approach working, in the end, I included all of the CSP directives within the main Apache config file:
Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src 'self' data:"

<Location /account/confirm_email>
Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src *"
</Location>

<Location /tools/check>
Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src *"
</Location>
The first Header call sets a default policy which is later overriden based on the path to the Django view that's being used.

Related technologiesIf you are interested in Content Security Policy, you may also want to look into Application Boundaries Enforcer (part of the NoScript Firefox extension) for more security rules that can be supplied by the server and enforced client-side.

It's also worth mentioning the excellent Request Policy extension which solves the same problem by letting users whitelist the cross-site requests they want to allow.

Rapha&#235;l Hertzog: My Debian activities in October 2011

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (130.30 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg work The month started with fixing newly reported bugs to prepare the 1.16.1.1 release: With the help of Guillem, we decided on a proper fix for a race condition sometimes triggered by parallel builds when 2 concurrent dpkg-gencontrol try to update debian/files (see #642608). This ended up requiring a new package (libfile-fcntllock-perl) that the Debian perl team kindly packaged for us. With all this sorted, it was a rather easy fix. Multiarch progress I also spent lots of time on multiarch. I fixed an old bug that requested to support the multi-arch paths in case of cross-building (see #595144), the discussion was not really conclusive on which of the two proposed patches was better so I ended up picking my own patch because it was closer to how we currently deal with cross-building. Then I fixed 2 issues that have been reported on Ubuntu s dpkg. The first one (LP #863675) was rather severe since an installed package ended being disappeared in favor of its foreign counterpart that was removed (but that had some config files left). The second one (LP #853679) only affected dselect users (apparently there are still some!) who had a self-conflicting library (Provides: foo, Conflicts: foo) installed for multiple architectures. But the bulk of the time spent on multiarch has been spent discussing with various parties on how to go forward with multiarch. The release team commented on the schedule of the merge to ensure it makes it into Wheezy, and the Debian project leader also commented on the problems encountered so far. While not the best course of action I could have hoped for, it certainly helped since Guillem started pushing some reviewed commits. Out of the 66 commits that were in my pu/multiarch/full branch one week ago, 20 have been merged in the master branch already. Python-django security update and RC bug Since python-django s maintainer did not manage to prepare the required security updates, I stepped in and prepared version 1.2.3-3+squeeze2 for Squeeze and 1.0.2-1+lenny3 for Lenny. Unfortunately this security update is an example of how an inactive maintainer is likely to result in a severe delay for the release of security updates. Furthermore in this specific case, the security team did not want to release the Squeeze security update until the Lenny one had been investigated (which required some time since upstream no longer supports the version in Lenny) but they did not make this very clear. Later another release critical bug had been filed against the package (#646634) but after investigation, it turned out to be a local configuration problem so I downgraded it. I still forwarded the test suite failure to upstream authors since the test could be enhanced. In any case, co-maintainers for python-django are welcome. I really preferred the situation where I can quietly sit down as backup maintainer :-) WordPress packaging WordPress sounds similar to python-django. I m also only a backup maintainer but Giuseppe has been inactive for many months and I had to step in August because I wanted the new upstream version. I discovered a bit late that I was not subscribed to wordpress bugs and thus the release critical bug #639733 (that I introduced with my new upstream version) went unattended for a rather long time. Once aware, though, I quickly fixed it. I also took the opportunity to start a discussion on debian-devel about how to deal with embedded javascript libraries and proposed a mechanism of opportunistic replacement with symlinks . WordPress is my testbed package for this mechanism, you can check out its debian/dh_linktree that implements the replacement logic. The discussion has not been very interesting but at least I learned that Debian now requires that each source package shipping minified javascript files includes the original files too. It s somewhat of a pain since it s not a license requirement in many cases (many of those libraries are not under the GPL), but just a Debian requirement that many upstreams are not complying with. WordPress is affected and Jakub Wilk thus opened #646729 which is going to be a long-standing RC bug. To give good measures, I spent several hours investigating the case of each javascript file in the WordPress source package and I filed a new ticket on the upstream bugtracker. Dropbox packaging work A few months after the introduction of nautilus-dropbox to Debian and Ubuntu, I can say that the decision to only support the download of dropbox in the postinst has been a mistake. Because of this decision I had to make the postinst fail if the download failed. Even if the error message is relatively clear, this lead to many (mostly automated) bug reports on the Ubuntu side. Various other problems cropped up on top of this (trying to start dropbox while the package was not configured would result in an error because the user did not have the required rights to install the software, reinstalling the package while dropbox was running would result in a failure too, etc.). I have fixed all those issues in the version 0.7.0-2 of the package. Now if the user has to install dropbox, it will use PolicyKit to request the root rights. The postinst will no longer fail if the dropbox download fails since it can be run later by the user. And I fixed the download code to remove the replaced file before unpacking a new file (insead of overwriting the existing file). All this work has been forwarded upstream. The Debian Administrator s Handbook Update I m glad to tell you that the translation will happen because we reached the minimal funding goal on October 22th with the help of 380 supporters. Now the fundraising continues, but this time the goal is the liberation of the resulting book. For this to happen, we need to reach 25000 EUR in the liberation fund. So far we re at 37% of this goal with 9400 EUR in the liberation fund (which means that 59% of the money raised has been put in the liberation fund).

Click here if you want to contribute towards the liberation of this book. With (less than) 27 days left, it s going to be a challenge to meet the goal, but we do like challenges, don t we? Misc work

Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

30 October 2011

Vincent Sanders: Software that adds another dimension to things

I think that it will come as no surprise to my fellow software engineers if I note that I almost never write new software any more. I maintain, I augment, I refactor, I debug but very, very rarely do I start something new.

This probably has something to do with my maturity as a code monkey, my immediate reaction is to seek out a solution to a problem that already exists and perhaps extend it to fulfil my requirements.

Partially this comes from my innate laziness but also over time I have discovered that I am a "finisher" the role I invariably end up in involves doing all the final bits to make the client accept a project. Because I know my reaction is to always finish something I start, I avoid starting things.

Anyhow, enough introspection, a couple of months ago I was talking on IRC about my 3D printer and was asked "can you print the Debian logo?". So I hunted around for software that would let me convert a bitmap into a suitable 3d format. The result was rather disappointing, the few tools I could find were generally python scripts which simply generated a matrix of cuboids, one for each pixel their heights corresponding to the pixel value.

I used one such script to generate a file for the Debian swirl and imported it into the OpenScad 3d modelling application. I got an inkling of the issues involved after the scene render took over half an hour. The resulting print was blocky and overall I was not terribly happy with the outcome.

So I decided I would write a program to convert images into a 3D representation. I asked myself, how hard can it be?

Sometimes starting from an utterly naive approach with no knowledge of a problem can lead to new insights. In this case I have spent all my free coding time for a month producing a program which I am now convinced has barely even scratched the surface of the possible solutions.

Having said that I have started from a blank editor window and a manual gcc command line compilation and progressed to an actually useful tool and, arguably, of more import to me I have learned and implemented a load of new algorithms which has actually been mentally stimulating and fun!

The basic premise of the tool is to take a PNG image, quantise it into a discrete number of levels , convert that as a height map into a triangle mesh, index that mesh (actually a very hard problem to solve efficiently), simplify the indexed mesh and output the result in a selection of 3D file formats.

The mesh generation alone is a complex field which it appears often devolves into the marching cubes algorithm simply out of despair of anything better ;-) I have failed to implement marching cubes so far (though I have partially implemented marching squares, an altogether simpler algorithm)

The mesh indexing creates an indexed list of vertices from the generated mesh and back annotates it with which faces are connected to which vertices. This effectively generates a useful representation of the meshes topology which can then be used to reduce the complexity of the mesh, or at least describe it. To gain efficiency I implemented my first ever bloom filter as part of my solution. I also learned that generating a plausible hash for said filter is a lot harder than it would seem. In the end I simply used the FNV hash which produces excellent results for very little computation cost.

The mesh simplification area is awash with academic research, most of which I ended up skipping and simply went for the absolute simplest edge removal algorithm. Implementing even this and maintaining a valid mesh topology was challenging.

By comparison the output of the various formats was positively trivial, mainly littered with head scratching over the bizzare de-facto "extensible" formats where only one trivial corner is ever actually implemented.

All in all I have had fun creating the PNG23D project and have actually used it to generate some useful output. I have even printed some of it to generate a lithophane of Turing. I now look forward to several years of maintaining and debugging it and doing all the other things I do instead of writing new software ;-)

28 October 2011

Kai Wasserb ch: Google recommends using OpenStreetMaps/OpenLayers

Maybe you've already heard, that Google is starting to charge a fee for using the Maps API (Google may decide you're an non-profit organization and waive the fees). That is, in my opinion, Google telling you to look for better alternatives. And thanks to the awesome effort of the OpenStreetMaps community, there is such an alternative. Thus today's addition to the mini-tips series will tell you, how you get an OpenStreetMaps-powered map on your website.
  1. First you need to determine the coordinates, where you want to center, the map. You can do this with OpenRouteService.org. Just search for the address, right-click on the marker highlighting the result, and set it as e.g. your start point, that'll give you the coordinates in the start field on the left. Alternatively you can just move your pointer over a place on the map and have a look at the lower right corner, where the current coordinates for the pointer position are displayed.
  2. Then you need to add a few things to the website, where the card is to be shown:
    • Include the OpenLayers script in the <head> of your webpage:
      <script src="http://www.openlayers.org/api/OpenLayers.js" type="text/javascript" />
      
    • Add a short ECMAScript snippet right after that, which will initialize your map:
      function osmInit()  
        document.getElementById("ol_map").style.display = 'block';
        var lonLat = new OpenLayers.LonLat(13.376096, 52.518598) // Center of the map
        .transform(
          new OpenLayers.Projection("EPSG:4326"),   // transform from WGS 1984
          new OpenLayers.Projection("EPSG:900913")  // to Spherical Mercator Projection
        );
        var map = new OpenLayers.Map("ol_map");
        var osmlayer = new OpenLayers.Layer.OSM();
        map.addLayer(osmlayer);
        map.setCenter(lonLat,
          16 // Zoom level
        );
       
      
      The first line is not needed, but I find it nice to hide the map on default, in case the user has disabled ECMAScript they haven't a large empty area somewhere on the website. The first line changes then the display value from "none" to "block". The rest is, I hope, already explained by the code itself.
    • Now only two more changes are needed, first you need to add onload="osmInit();" to your <body> tag. And afterwards an element (say a <p>) with the ID ol_map somewhere in the website <body>. You should set a width and height for that tag (style is your friend), otherwise the map might be bigger than you like.
  3. Done. Enjoy your free (not just as in free beer) map!
You can, of course, customize your map further by adding additional layers and changing the available options (e.g. the zoom level). If you would want to mark a location on your map, you'd need to add something like
var markers = new OpenLayers.Layer.Markers("POI1");
map.addLayer(markers);
markers.addMarker(new OpenLayers.Marker(lonLat));
to your ECMAScript snipped after you added the OpenStreetMaps layer. These three lines would mark the location given by the variable lonLat. For further information, you can check out the OpenStreetMaps wiki and the OpenLayers documentation. In case you wonder, where my coordinates from this example point, you can check. Oh, and don't forget to consider a donation to OpenStreetMaps, so the project can pay the bills for its servers.

Matthew Palmer: rsync for LVM-managed block devices

If you ve ever had to migrate a service to a new machine, you ve probably found rsync to be a godsend. It s ability to pre-sync most data while the service is still running, then perform the much quicker sync the new changes action after the service has been taken down is fantastic. For a long time, I ve wanted a similar tool for block devices. I ve managed ridiculous numbers of VMs in my time, almost all stored in LVM logical volumes, and migrating them between machines is a downtime hassle. You need to shutdown the VM, do a massive dd netcat, and then bring the machine back up. For a large disk, even over a fast local network, this can be quite an extended period of downtime. The naive implementation of a tool that was capable of doing a block-device rsync would be to checksum the contents of the device, possibly in blocks, and transfer only the blocks that have changed. Unfortunately, as network speeds approach disk I/O speeds, this becomes a pointless operation. Scanning 200GB of data and checksumming it still takes a fair amount of time in fact, it s often nearly as quick to just send all the data as it is to checksum it and then send the differences.1 No, a different approach is needed for block devices. We need something that keeps track of the blocks on disk that have changed since our initial sync, so that we can just transfer those changed blocks. As it turns out, keeping track of changed blocks is exactly what LVM snapshots do. They actually keep a copy of what was in the blocks before it changed, but we re not interested in that so much. No, what we want is the list of changed blocks, which is stored in a hash table on disk. All that was missing was a tool that read this hash table to get the list of blocks that had changed, then sent them over a network to another program that was listening for the changes and could write them into the right places on the destination. That tool now exists, and is called lvmsync. It is a slightly crufty chunk of ruby that, when given a local LV and a remote machine and block device, reads the snapshot metadata and transfers the changed blocks over an SSH connection it sets up. Be warned: at present, it s a pretty raw piece of code. It does nothing but the send updated blocks over the network , so you have to deal with the snapshot creation, initial sync, and so on. As time goes on, I m hoping to polish it and turn it into something Very Awesome. Patches Accepted , as the saying goes.
  1. rsync avoids a full-disk checksum because it cheats and uses file metadata (the last-modified time, or mtime of a file) to choose which files can be ignored. No such metadata is available for block devices (in the general case).

26 October 2011

Russ Allbery: Review: Scrum and XP from the Trenches

Review: Scrum and XP from the Trenches, by Henrik Kniberg
Publisher: InfoQ
Copyright: 2007
ISBN: 1-4303-2264-0
Format: Trade paperback
Pages: 127
For those not already familiar with it, Scrum is a project management methodology that's most often seen in the agile software development community. It attempts to do for project management what agile techniques do for software development (and, other than test-driven development, I think is more useful than most of agile). I'm not going to explain it here in detail, but if you do software development or project management of software-style projects (including integration and computing infrastructure), it's worth your attention. It provides some nice techniques for dealing with some of the banes of technical projects, particularly constantly changing requirements and the difficulties of estimates. Scrum and XP from the Trenches is subtitled "How we do Scrum" and is exactly what it says on the tin. It is a fairly detailed description of how Crisp (an agile Java development company in Sweden) does Scrum (with a little discussion of Extreme Programming). It doesn't explain what Scrum is, although it does provide a few pointers to resources if you've never heard of it (but this isn't the sort of book you'd be buying if you'd never heard of it). It also doesn't try to convince you that Scrum is wonderful, which is a nice trap to avoid. It gets right down to business and packs quite a bit of relentlessly practical content into a small number of pages. I've been doing Scrum for about a year, across two large projects, both times not entirely following the full Scrum model. Both times, most of the team was learning Scrum during the project. I've gone to some Scrum training and read a bit, largely on-line, but I haven't made an exhaustive study of it. I think I'm exactly the ideal audience for this book, although my guess is that people who have been doing Scrum for longer will also get something out of it. This is not an introduction; I don't think one is likely to get much benefit out of it without a couple of Scrum projects under one's belt so that one has a feel for the problems and failure modes. As Kniberg accurately points out, Scrum is a framework, not a precise rule set, and it can't be applied blindly in one situation using analysis from another. For that audience, though, this book is great. Kniberg has used Scrum in a variety of different situations and tried multiple solutions to many of its problems, so he can provide guidance not only on his preferred solutions but on the other solutions he's tried and why they don't work. I started taking extensive notes just a few pages in; it was exactly the book I needed to read for the full-project retrospective on how Scrum worked for a just-concluded work project, and for planning what changes we'll want to make for the next project. He goes through every step of the Scrum cycle and offers advice on each step, particularly for the sprint planning (which was extremely helpful). He then walks through release planning and advance estimation (one of the best parts of the book; I've seen the concept of a focus factor before, but this is one of the best and clearest explanations I've seen), talks about XP (Extreme Programming) techniques and testing, and then discusses how to coordinate multiple Scrum teams in a larger organization. The chapter on testing was the reason why I bought the book; the rest of the book lives up to it. As mentioned, my Scrum exposure is not broad, so Kniberg may just be following different orthodoxies. That said, he broke with the ones I was previously familiar with in some useful places. One is to attach a time unit to story points, something that I was told was close to anethema, but as presented here looks very useful. Another, not so much a break but an idea that I'd not encountered elsewhere, is downtime between sprints, an brilliant solution to a problem that I'd not even fully articulated but which had been plaguing our Scrum process. He also has some suggestions about communication that, while fairly obvious and common-sense stuff, still were things that hadn't occurred to me and that we should adopt. I paid for a hard-copy version of this book from Lulu because I like things like this on paper and am happy to support the author financially, but you can download a PDF file and read it for free from InfoQ and see whether it works as well for you as it did for me. If you're ramping up on Scrum, seeing some problems, and looking for good ideas on how to fix them, I highly recommend it. I'd love to read two or three more case studies like this to get some more perspectives on how to make Scrum work. The paper book will likely be making a circuit around our office to serve as fodder for our upcoming Scrum projects. Rating: 8 out of 10

25 October 2011

Sylvestre Ledru: Removal of sun-java6 from Debian

As said in one of my previous blog post, we, Debian, were going to remove sun-java6 packages from the archive once critical security issues would come. It was supposed to happen sooner and later and here we are. The version 29 (see the #645881 Debian bug) of the Oracle proprietary JVM has been released. It fixes some critical issues but, because of the DLJ removal, Debian is no longer able to distribute it. Unfortunately, we have no other choice than asking for a removal from the archive and promote the use of the openjdk 6 & 7.
The conclusion is:
apt-get --purge remove sun-java6-jre && apt-get install openjdk-7-jre

24 October 2011

Thorsten Glaser: benz wedding, fun before

My dear MirBSD co-developer Benny did not only get his Doctor title but also recently married. There will be another post detailing this, including better photos of the two Doctors and the cake (with a D mon she made herself) on the wlog, but this is some fun beforehand: No GPL cars! Apparently, it is forbidden in France to drive GPL cars. (Without safety valve but you have to admit the picture was fun. And we were like since the thing actually meant is LPG in German. Just like UTC is CUT (Coordinated Universal Time) in English, TUC (Temps Universel Coordonn ) in French )

Colin Watson: Quality in Ubuntu 12.04 LTS

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order: Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

Dirk Eddelbuettel: Kurt Elling and the Kl vers Big Band at the Green Mill

Twenty-four hours ago I was in heaven. Or to more properly recapitulate and situate this, fourty-eight hours ago I was in hell. Now this requires some context. Earlier in the year, I had tweeted briefly about a Kurt Elling live concert still streaming on NPR, and truth be told, I have listened to Kurt Elling a whole lot ever since. The most recent album The Gate is fantastic, the Grammy-winner Dedicated to You (which retakes the very famous Coltrane/Hartman album) is beyond words, and The Messenger is worth it just for 'Nature Boy' and the funkiest-ever 'April in Paris'. I also saw him as part of a larger and otherwise rather neat tour. But I needed to see the man himself. So following what one can see from his touring calendar, I jotted down two October dates. Kurt Elling at the Green Mill. Fine. That was earlier in the summer. Summer has a tendency of evaporating quickly, so here we were in October. And getting going late on Friday, so by the time we got the Green Mill, a long line had formed and we ended up paying tribute to the line for fourty-five minutes with nothing to show. Sad. Home we went, and no live music. But somehow we mustered the energy to go again last night and thought we were early, arriving about 20 minutes before the show was to start. Ha! Others had appeared at 3pm, by 5pm all seats were gone. We were literally the last ones the get in after a short spell in a shorter line, and standing room-only it was. Oh, but what a treat we got. The set for these two days was Kurt Elling with the fourteen piece Kl vers Big Band from Denmark. They played for three marvellous sets, and well over four hours (with two 25 minute breaks). Pieces often alternated between big band arrangements and smaller pieces by Elling's standard rhythm band, and covered standards as well as Elling's catalogue. As I mentioned earlier on a short post (with pictures) on Google+, it is essentially impossible to not fall for Elling's stage presence when being so close to the stage. The man is just that good. Elling and the Kl vers Big Band will now play for six nights at Birdland in New York followed by a night each in Washington, DC, and Boston. If you're around, do yourself a favour and go out to see them.

23 October 2011

Uwe Hermann: The TrekStor eBook Reader 3.0 (EBR30-a), review and dissection

The TrekStor eBook Reader 3.0, front
The TrekStor eBook Reader 3.0, front on There a many, many, e-book reader devices available these days, and they're quickly becoming pretty affordable. The currently cheapest device in Germany (that I know of) is the TrekStor eBook Reader 3.0, model number EBR30-a, at 59.- Euros via Weltbild or Hugendubel. The device has an 800x480 7" TFT (yep, no e-ink), 2100mAh battery, it can display PDFs, EPUB, and TXT files (and Adobe DRM crap, which I don't really care about), it has an accelerometer which allows for landscape/portrait switching, it can play MP3, OGG, WAV, and WMA audio files (headphone jack), it can display pictures (BMP, GIF, JPG, even PNG, though that's not mentioned in the vendor's specs), and it has 2GB internal storage for books/music/pictures. Uploading of (non-DRM) content is done by a simple file copy, it enumerates as a standard USB mass storage device with FAT filesystem. It's a relatively nice reader for the price, I've read a few PDFs (datasheets, presentations) on it in the subway/train while listening to music from the device and it's quite OK for my purposes. So much for the review part. However, I didn't really buy it for reading books on it, I was more interested in taking it apart, of course ;-) My hope was that it would turn out to be a really cheap device running Linux/U-Boot which would be perfect for playing around with embedded Linux stuff. Unfortunately, I wasn't so lucky (it seems). The TrekStor eBook Reader 3.0, opened I've posted a few photos of the device and its hardware components on my flickr account and over at randomprojects.org, together with all the information I was able to find out so far. Here's a quick summary: The TrekStor eBook Reader 3.0, CPU There are public datasheets for most of the hardware components (see randomprojects.org for links), but unfortunately the most important one (for the CPU) is not yet found/identified. I was told that the CPU/SoC is probably based on an ARM9 (ARM926EJ-S) core and the firmware running on it seems to be some uCos-based RTOS (not Linux, unfortunately). So far I was not able to find out the vendor name or website of the "FI E200" CPU/SoC (let alone any datasheets), any hints would be highly appreciated. I checked arm.com: Processor Licensees, but the only two companies whose name starts with "F" having licensed an ARM9 core are Fujitsu and Freescale, which doesn't fit, I think? I could (and probably will) check the PCB for RX/TX lines on an UART and/or JTAG pads (none are obviously labelled), and given that it's and ARM9 core there is a good chance that OpenOCD can be used and that a standard cross-gcc for ARM will work. However, that is all pretty pointless until it's clear which SoC exactly is used, and thus whether there is already Linux and/or U-Boot support for it and/or whether datasheets are available so that the respective code could be written. Without datasheets, this is going to be a pretty painful experience, not really worth investing much time, IMHO. If anyone knows more about the vendor/device and respective datasheets, please let me know. Thanks!

Niels Thykier: Testing testing migration

If you have been following Lintian s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests. The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a live data sample , which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong. The test suite has a somewhat unfair focus on auto-hint [3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations. Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve. On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):
// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    od
    return current
end
This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows. If you are interested in playing around with this, you can get sources from: Footnotes: [1] http://lists.debian.org/debian-release/2011/10/msg00178.html [2] These tests are auto-generated, so it is merely an up-scaled pattern . [3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in. [4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. :P [5] I assume the test called sat-britney-death (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the approve tpu package test case should fail. [6] A function that maps an archive into another archive erh, I mean, it maps a set of packages into another set of packages :P [7] http://en.wikipedia.org/wiki/Fixed_point_%28mathematics%29 Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing. As an example:
y depends on x
x in testing has RC bugs If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it. (Dis-)Proving my claim is an exercise left for the reader.

22 October 2011

Russell Coker: Desks

Lindsay Holmwood has written about the benefits of a standing desk and how to buy one [1]. The case for avoiding sitting is strong, but I couldn t stand up all day. One thing that s been on my list of things to do if I had an unreasonably large amount of spare time or money is to make a reclining computer station. The idea would be to take a bed and mount a TFT display above it so that it can be viewed while lying down. Then a split keyboard would be required so that each hand can be used for half the keys (this would be difficult or expensive). If the keyboard halves were aligned correctly then it would reduce the carpal tunnel problems associated with computer use (which has been a big problem for me in the past and is something that I will probably never fully recover from [2]). As far as I am aware the risk of back problems is eliminated when lying down, so two of the major problems with regular computer desks would be avoided. I don t think that lying down all day would be that great and it wouldn t work for collaborative projects. But as monitors are so cheap nowadays it would be viable to have a second monitor at a desk connected to the same computer. Then I could spend about half my computer time lying down.

20 October 2011

Pau Garcia i Quiles: The Secure Boot controversy

Red Hat s Matthew Garrett let the cat out of the bag about a month ago: when UEFI Secure Boot is adopted by mainboard manufacturers to satisfy Microsoft Windows 8 requirements, it may very well be the case that Linux and others (BSD, Haiku, Minix, OS/2, etc) will no longer boot. Matthew has written about it extensively and seems to know very well what the issues are (part I, part II), the details about signing binaries and why Linux does not support Secure Boot yet. The Free Software Foundation has also released a statement and started a campaign, which is, as usually, anti-Microsoft instead of pro-solutions. Now let me express my opinion on this matter: this is not Microsoft s fault. Facts Let s see what are the facts in this controversy: Problem The problem stems from the fact that most Linux vendors do not have the power to get their certificates in UEFI BIOS. Red Hat and Suse will for sure get their certificates bundled in server UEFI BIOS. Debian and Ubuntu? Maybe. NetBSD, OpenIndiana, Slackware, etc? No way. This is, in my humble opinion, a serious defect in the standard. A huge omission. Apparently while developing the Secure Boot specification everybody was busy talking about signed binaries, yet nobody thought for a second how the certificates will get into the UEFI BIOS. What should have been done The UEFI secure boot standard should have defined an organization (a Secure Boot Certification Authority ) that would issue and/or receive certificates from organizations/companies (Red Hat, Oracle, Ubuntu, Microsoft, Apple, etc) that want their binaries signed. This SBCA would also be in charge of verifying the background of those organizations. There is actually no need for a new organization: just use an existing one, such as Verisign, that carries on with this task for Microsoft for kernel-level binaries (AuthentiCode). Given that there is no Secure Boot Certification Authority, Microsoft asked BIOS (UEFI) developers and manufacturers to include their certificates, which looks 100% logical to me. The fact that Linux distributions do not have such power is unfortunate, but it is not Microsoft s fault at all. What can we do? Given its strong ties with Intel, AMD and others, maybe the Linux Foundation could start a task force and a Temporary Secure Boot Certification Authority to deal with UEFI BIOS manufacturers and developers. This task force and TSBCA would act as a proxy for minorities such as Linux, BSD, etc distributions. I am convinced this is our best chance to get something done in a reasonable amount of time. Complaining will not get us anything. Screaming at Microsoft will not get us anything. We need to propose solutions. Wait! Non-Microsoft certificates? Why? In addition to the missing Secure Boot Certification Authority, there is a second problem apparently nobody is talking about: what is the advantage mainboard manufacturers get from including non-Microsoft certificates? For instance: why would Gigabyte (or any other mainboard manufacturer) include the certificate for, say, Haiku? The benefit for Gigabyte would be negligible and if someone with ill-intentions gets Haiku s certificate, that piece of malware will be installable on all Gigabyte s mainboards.This would lead to manufacturer-targetted malware, which would be fatal to Gigabyte: oh, want to be immune to the-grandchild-of-Stuxnet? Buy (a computer with) an MSI mainboard, which does not include Haiku s certificate Given that 99% of desktops and laptops only run Windows, the result of this (yet unresolved) problem would be that manufacturers will only install Microsoft certificates, therefore they would be immune to malware signed with a Slackware certificate in the wild. If we are lucky, mainboard manufacturers will give us an utility to install more certificates under your own risk. The solution to the first problem looks easy to me. The solution to the second looks much more worrying to me.

19 October 2011

Yves-Alexis Perez: Debian grsec kernels

I received recently a mail about my attempt to provide Grsecurity kernels in Debian. The sender found the bug by accident, and asked me why I didn't do some more publicity here. So here we are. I won't go into details on what grsecurity is, it's fairly complex. But it's basically a hardening patch for the Linux kernel, with three main components: A lot of this touches low level stuff in the kernel, especially memory management. Ideally this patch would be pushed upstream, but Brad Spengler (grsecurity main developper) already said he wasn't interested in upstreaming it and upstream already said the patch was too huge and invasive to include it like that (especially since the original authors aren't interested in maintaining it upstream). There's an ongoing effort to split the patch and merge things little by little, but in the meanwhile having a mid-term solution would be nice. I know Debian users rebuilding grsecurity-patched kernels themselves, and I know some of them would appreciate having them included in the Debian kernel. Fortunately, the linux-2.6 source package has a nice feature which is called featureset. Basically it's a way to build some (binary) packages using a different set of patches and a different config. For example this was used to provide xen/openvz/vserver patchsets, and is now used to provide rt kernels. So I though it'd be nice to provide a grsec featureset, and starting doing the work. I have a working setup for producing those kernels, so I've opened a wishlist bug against the kernel (#605090) to have this merged. Those packages follow the sid kernel. There's an ongoing work for Squeeze, but it's a bit harder there because both the grsecurity patchset and the Debian kernel ship a whole lot of backports to the Linux kernel, meaning the grsecurity patch doesn't apply directly to the Debian source package. Basically I need to remove some of the hunks (since they are already applied to the source) and port some others (since there are some backported code not present in the vanilla 2.6.32, for example the drm code). Until the patches are merged and the bug is closed, I host some of the built packages at:

deb http://molly.corsac.net/~corsac/debian/kernel-grsec/packages/ sid/ The repository is signed by my key which you can add to your apt setup using apt-key add. If you want to rebuild the packages yourself, here's the method:

mkdir kernel-grsec
cd kernel-grsec
svn checkout svn://svn.debian.org/svn/kernel/dists/sid/linux-2.6
git clone git://anonscm.debian.org/users/corsac/grsec-patches.git
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.bz2
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.bz2.sign
gpg --verify linux-3.0.tar.bz2
cd linux-2.6
apt-get build-dep linux-2.6
export QUILT_PATCHES=../grsec-patches
quilt push -a
python debian/bin/genorig.py ../linux-3.0.tar.bz2
debian/rules orig
fakeroot debian/rules source
fakeroot make -f debian/rules.gen binary-arch_amd64_grsec_amd64 You could also do dpkg-buildpackage, pdebuild or whatever. Kernel handbook is a nice reading too if you want more information on how to rebuild Debian kernels. The quilt push -a may fail if you checkout an svn version more recent than mine. I try to keep patches up to date but I usually have some delay. Note that installing the kernel will require installing linux-grsec-base package. Binary is not yet available on my mirror but you can easily build it. Source can be found on git.debian.org. If you're interested by this, don't hesitate to mail me or the bug.

Next.

Previous.