Search Results: "reg"

2 November 2011

Andrew McMillan: Teaching our children to lie

My son is being encouraged to lie. It's a fairly regular occurrence around here, and I'm sure you've seen it yourself. It's that checkbox on the website you're visiting where you say "Yes, I am over X years of age".

How could you not trust this child?

It's well-known that there are 10 year olds on Facebook, and everyone agrees that they lied to get those accounts, but nobody really cares much. Facebook don't care. Facebook just see another pair of eyeballs, and another node in their social graph, but actually Fraser, Max (and even Heather) aren't nodes on that particular graph. Today's annoyance was that Fraser's Youtube account has been blocked. That just seems silly. It's an open website where anyone can read pretty much anything, except that to have an account on there you have to be older than X years (I haven't bothered to discover how old X is - it's not relevant). So it seems there's this website, worth many billions of dollars and these guys can only control the functionality available to signed up members with an "on/off" switch. Is it bad if under 12 year olds post comments in a public forum? Oh, we should not let them create accounts then. Is it bad if under 8 year olds see soft porn? Oh, if we stop them creating accounts they'll be safe from that! What about the 14 year old with the videos of her cat? Well we definitely, definitely don't want that! That's ludicrous! Can these people not design a website to accept a child? Someone who might want to log into the website just like mummy and daddy do? Then, when the honest child is logged in, perhaps the account can be linked to a parent's account, or perhaps it could just be restricted in different ways. Let's see what they say an account on Youtube offers you:
  1. Subscribe to your favorite channels.
  2. Rent or purchase top Hollywood movies
  3. Save videos to watch later
  4. Get recommendations based on what you've watched
  5. Share videos you like on Facebook, Twitter, and more
  6. Find your Facebook, Yahoo, Hotmail and Gmail contacts on YouTube
  7. Watch private videos from friends and family
OK, so as far as I can see:
  1. Might be useful to any age person, and is entirely innocuous in the context of the
    content of Youtube which is available to anonymous visitors of any age.
  2. Well, Fraser wouldn't be able to pay, since he doesn't have anything to pay
    with. I guess he could steal my credit card, but if he's already at that level
    of skullduggery then some silly checkbox would be "child's play" to his lying
    skillz.
  3. Entirely innocuous.
  4. Seems pretty innocuous too.
  5. Presumably he wouldn't because 10 year old's can't have identities. And this
    functionality could easily be restricted, in any case
  6. And if there are no relationships, there are no friends. Maybe there is family,
    which is fine, I guess. I think the risks here are controllable.
So there's plenty of value to a 10 year old in having an account. There's nothing that most reasonable parents would be concerned about, if there was a clear policy of limited functionality in place that parents could see and have confidence in. The Youtube thing actually seems to be related to (maybe) to Google's recent push towards "identity". Children younger than X are not allowed to have an on-line identity, because they might run amok on the intarwebs. Or see scary stuff. Or something. And yet, by denying them an account, they are removing any ability to actually apply a level of control that reflects the presumed maturity (since, after all, age is no more a direct measure of maturity than height). Truly though, what totally pisses me off about this situation, is that Fraser associated a bunch of channels with his Youtube account, and he receives a weekly digest of the activity. Unfortunately we can't log in to turn it off.

1 November 2011

Vincent Bernat: SSL computational DoS mitigation

Some days ago, a hacker group, THC, released a denial of service tool for SSL web servers. As stated in its description, the problem is not really new: a complete SSL handshake implies costly cryptographic computations. There are two different aspects in the presented attack:

Mitigation techniques There is no definitive solution to this attack but there exists some workarounds. Since the DoS tool from THC relies heavily on renegotiation, the most obvious one is to disable this mechanism on the server side but we will explore other possibilities.

Disabling SSL renegotiation Tackling the second problem seems easy: just disable SSL renegotiation. It is hardly needed: a server can trigger a renegotiation to ask a client to present a certificate but a client usually does not have any reason to trigger one. Because of a past vulnerability in SSL renegotiation, recent version of Apache and nginx just forbid it, even when the non-vulnerable version is available. openssl s_client can be used to test if SSL renegotiation is really disabled. Sending R on an empty line trigger renegotiation. Here is an example where renegotiation is disabled (despite being advertised as supported):
$ openssl s_client -connect www.luffy.cx:443 -tls1
[...]
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
[...]
R
RENEGOTIATING
140675659794088:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:
Disabling renegotation is not trivial with OpenSSL. As an example, I have pushed a patch to disable renegotiation in stud, the scalable TLS unwrapping daemon.

Rate limiting SSL handshakes Disabling SSL renegotiation on the client side is not always possible. For example, your web server may be too old to propose such an option. Since those renegotiations should not happen often, a workaround is to limit them. When the flaw was first advertised, F5 Networks provided a way to configure such a limitation with an iRule on their load-balancers. We can do something similar with just Netfilter. We can spot most TCP packets triggering such a renegotiation by looking for encrypted TLS handshake record. They may happen in a regular handshake but in this case, they usually are not at the beginning of the TCP payload. There is no field saying if a TLS record is encrypted or not (TLS is stateful for this purpose). Therefore, we have to use some heuristics. If the handshake type is unknown, we assume that this is an encrypted record. Moreover, renegotiation requests are usually encapsulated in a TCP packet flagged with push .
# Access to TCP payload (if not fragmented)
payload="0 >> 22 & 0x3C @ 12 >> 26 & 0x3C @"
iptables -A LIMIT_RENEGOCIATION \
    -p tcp --dport 443 \
    --tcp-flags SYN,FIN,RST,PSH PSH \
    -m u32 \
    --u32 "$payload 0 >> 8 = 0x160300:0x160303 && $payload 2 & 0xFF = 3:10,17:19,21:255" \
    -m hashlimit \
    --hashlimit-above 5/minute --hashlimit-burst 3 \
    --hashlimit-mode srcip --hashlimit-name ssl-reneg \
    -j DROP
The use of u32 match is a bit difficult to read. The manual page gives some insightful examples. $payload allows to seek for the TCP payload. It only works if there is no fragmentation. Then, we check if we have a handshake (0x16) and if we recognise TLS version (0x0300, 0x0301, 0x0302 or 0x0303). At least, we check if the handshake type is not a known value. There is a risk of false positives but since we use hashlimit, we should be safe. This is not a bullet proof solution: TCP fragmentation would allow an attacker to evade detection. Another equivalent solution would be to use CONNMARK to record the fact the initial handshake has been done and forbid any subsequent handshakes. If you happen to disable SSL renegociation, you can still use some Netfilter rule to limit the number of SSL handshakes by limiting the number of TCP connections from one IP:
iptables -A LIMIT_SSL \
    -p tcp --dport 443 \
    --syn -m state --state NEW \
    -m hashlimit \
    --hashlimit-above 120/minute --hashlimit-burst 20 \
    --hashlimit-mode srcip --hashlimit-name ssl-conn \
    -j DROP
Your servers will still be vulnerable to a large botnet but if there is only a handful of source IP, this rule will work just fine1. I have made all those solutions available in a single file.

Increasing server-side power processing SSL can easily be scaled up and out. Since SSL performance increases linearly with the number of cores, scaling up can be done by throwing in more CPU or more cores per CPU. Adding expensive SSL accelerators would also do the trick. Scaling out is also relatively easy but you should care about SSL session resume.

Putting more work on the client side In their presentation of the denial of service tool, THC explains:
Establishing a secure SSL connection requires 15 more processing power on the server than on the client.
I don t know where this figure comes from. To check it, I built a small tool to measure CPU time of a client and a server doing 1000 handshakes with various parameters (cipher suites and key sizes). The results are summarized on the following plot: Plot to compare computational power required by servers and clients For example, with 2048bit RSA certificates and a cipher suite like AES256-SHA, the server needs 6 times more CPU power than the client. However, if we use DHE-RSA-AES256-SHA instead, the server needs 34% less CPU power. The most efficient cipher suite from the server point of view seems to be something like DHE-DSS-AES256-SHA where the server needs half the power of the client. However, you can t really uses those shiny cipher suites:
  1. Some browsers do not support them: they are limited to RSA cipher suites2.
  2. Using them will increase your regular load a lot. Your servers may collapse with just legitimate traffic.
  3. They are expensive for some mobile clients: they need more memory, more processing power and will drain battery faster.
Let s dig a bit more on why the server needs more computational power in the case of RSA. Here is a SSL handshake when using a cipher suite like AES256-SHA: SSL full handshake When sending the Client Key Exchange message, the client will encrypt TLS version and 46 random bytes with the public key of the certificate sent by the server in its Certificate message. The server will have to decrypt this message with her private key. Those are the two most expensive operations in the handshake. Encryption and decryption are done with RSA (because of the selected cipher suite). To understand why decryption is more expensive than encryption, let me explain how RSA works. First, the server needs a public and a private key. Here are the main steps to generate them:
  1. Pick two random distinct prime numbers p and q , each roughly the same size.
  2. Compute n=pq . It is the modulus.
  3. Compute \varphi(n)=(p-1)(q-1) .
  4. Choose an integer e such that 1<e<\varphi(n) and \gcd(\varphi(n),e) = 1 (i.e. e and \varphi(n) are coprime). It is the public exponent.
  5. Compute d=e^ -1 \mod\varphi(n) . It is the private key exponent.
The public key is (n,e) while the private key is (n,d) . A message to be encrypted is first turned into an integer m<n (with some appropriate padding). It is then encrypted to a ciphered message c with the public key and should only be decrypted with the private key:
  • c=m^e\mod n (encryption)
  • m=c^d\mod n (decryption)
So, why is decryption more expensive? In fact, the key pair is not really generated like I said above. Usually, e is a small fixed prime number with a lot of 0, like 17 (0x11) or 65537 (0x10001) and p and q are choosen such that \varphi(n) is coprime with e . This allows encryption to be fast using exponentiation by squaring. On the other hand, its inverse d is a big number with no special property and therefore, exponentiation is more costly and slow. Instead of computing d from e , it is possible to choose d and compute e . We could choose d to be small and coprime with \varphi(n) and then compute e=d^ -1 \mod\varphi(n) and get blazingly fast decryption. Unfortunately, there are two problems with this: Therefore, we cannot use a small private exponent. The best we can do is to choose the public exponent to be e =4294967291 (the biggest prime 32bit number and it contains only one 0). However, there is no change as you can see on our comparative plot. To summarize, no real solution here. You need to allow RSA cipher suites and there is no way to improve the computational ratio between the server and the client with such a cipher suite.

Things get worse Shortly after the release the denial of service tool, Eric Rescorla3 published a good analysis on the impact of such a tool. He asks himself about the efficiency to use renegotiation for such an attack:
What you should be asking at this point is whether a computational DoS attack based on renegotiation is any better for the attacker than a computational DoS attack based on multiple connections. The way we measure this is by the ratio of the work the attacker has to do to the work that the server has to do. I ve never seen any actual measurements here (and the THC guys don t present any), but some back of the envelope calculations suggest that the difference is small. If I want to mount the old, multiple connection attack, I need to incur the following costs:
  1. Do the TCP handshake (3 packets)
  2. Send the SSL/TLS ClientHello (1 packet). This can be a canned message.
  3. Send the SSL/TLS ClientKeyExchange, ChangeCipherSpec, Finished messages (1 packet). These can also be canned.
Note that I don t need to parse any SSL/TLS messages from the server, and I don t need to do any cryptography. I m just going to send the server junk anyway, so I can (for instance) send the same bogus ClientKeyExchange and Finished every time. The server can t find out that they are bogus until it s done the expensive part. So, roughly speaking, this attack consists of sending a bunch of canned packets in order to force the server to do one RSA decryption.
I have written a quick proof of concept of such a tool. To avoid any abuse, it will only work if the server supports NULL-MD5 cipher suite. No sane server in the wild will support such a cipher. You need to configure your web server to support it before using this tool. While Eric explains that there is no need to parse any SSL/TLS messages, I have found that if the key exchange message is sent before the server send the answer, the connection will be aborted. Therefore, I quickly parse the server s answer to check if I can continue. Eric also says a bogus key exchange message can be sent since the server will have to decrypt it before discovering it is bogus. I have choosen to build a valid key exchange message during the first handshake (using the certificate presented by the server) and replay it on subsequent handshakes because I think the server may dismiss the message before the computation is complete (for example, if the size does not match the size of the certificate). With such a tool and 2048bit RSA certificate, a server is using 100 times more processing power than the client. Unfortunately, this means that most solutions, except rate limiting, exposed on this page may just be ineffective.

  1. However, since this rule relies on source IP to identify the attacker, the risk of false positive is real. You can slow down legitimate proxies, networks NATed behind a single IP, mobile users sharing an IP address or people behind a CGN.
  2. Cipher suites supported by all browsers are RC4-MD5, RC4-SHA and 3DES-SHA. Support for DHE-DSS-AES256-SHA requires TLS 1.2 (not supported by any browser).
  3. Eric is one of the author of several RFC related to TLS. He knows his stuff.

Bernhard R. Link: File ownership and permissions in Debian packages

As you will know, every file on a unixoid system has some meta-data like owner, group and permission bits. This is of course also true for files part of some Debian package. And it is not very surprising that different files should have different permissions. Or even different owners or groups. Which file has which settings is of course for the package maintainer to decide and Debian would not be Debian if there were not ways for the user to give their own preferences and have them preserved. This post is thus about how those settings end up in the package and what is to be observed when doing it. As you will also have heard, a .deb file is simply a tar archive stored as part of an ar archive, as you can verify by unpacking a package manually:
$ ar t reprepro_4.8.1-1_amd64.deb
debian-binary
control.tar.gz
data.tar.gz
$ ar p reprepro_4.8.1-1_amd64.deb data.tar.gz   gunzip   tar -tvvf -
drwxr-xr-x root/root         0 2011-10-10 12:05 ./
drwxr-xr-x root/root         0 2011-10-10 12:05 ./etc/
drwxr-xr-x root/root         0 2011-10-10 12:05 ./etc/bash_completion.d/
-rw-r--r-- root/root     19823 2011-10-10 12:05 ./etc/bash_completion.d/reprepro
drwxr-xr-x root/root         0 2011-10-10 12:05 ./usr/
drwxr-xr-x root/root         0 2011-10-10 12:05 ./usr/share/
--More--
(For unpacking stuff from scripts, you should of course use dpkg-deb --fsys-tarfile instead of ar gunzip. Above example is about the format, not a recipe to unpack files). This already explains how the information is usually encoded in the package: A tar file contains that information for each contained file and dpkg is simply using that information. (As tar stores numeric owner and group information, that limits group and owner information to users and groups with fixed numbers, i.e. 0-99. Other cases will be covered later.) The question for the maintainer is now: Where is the information which file has which owner/group/permissions in the .tar inside the .deb, and the answer is simple: It's taken from the files to be put into the .deb. This means that package tools could simply be implemented first by simply calling tar and there is no imminent need to write you own tar generator. It also means that the maintainer has full control and does not have to learn new descriptive languages or tools to change permissions, but can simply put the usually shell commands into debian/rules. There are some disadvantages, though: A normal user cannot change ownership of files and one has to make sure all files have proper permissions and owners. This means that dpkg-deb -b (or the usually used wrapper dh_builddeb) must be run in some context where you could change the file ownership to root first. This means you either need to be root, or at least to fake being root by using fakeroot. (While this could be considered some ugly workaround, it also means upstream's make install is run believing to be root, which also avoids some -- for a packager -- quite annoying automatisms in upstream build scripts assuming a package is not installed system wide if not installed as root). Another problem are random build host characteristics changing how files are created in the directory later given to dpkg-deb -b. For example an umask which might make all files non-world-readable by default. The usual workaround is to first fix up all those permissions. Most packages use dh_fixperms for this, which also sets executable bits according to some simple rules and has some more special cases so that the overall majority of packages does not need to look at permissions at all. So using some debhelper setup, every special permissions and all owner/group information for owner groups with fixed numbers only needs to be set using the normal command line tools between dh_fixperms and dh_builddeb. Everything else happens automatically. Note that games is a group with fixed gid. So it is not necessary (and usually a bug) to change group-ownership of files withing the package to group games in maintainer scripts (postinst,...). If a user wants to change permissions or ownership of a file, dpkg allows this using the dpkg-statoverride command. This command essentially manages a list of files to get special treatment and ownership and permission information they should get. This way a user can specify that files should have different permissions and this setting is applied if a new version of this file is installed by dpkg. Being a user setting especially means, that packages (that means their maintainer scripts) should not usually use dpkg-statoverride. There are two exceptions, though: Different permissions based on interaction with the user (e.g. asking question with debconf) and dynamically allocated users/groups with dynamic id. In both cases one should note that settings to dpkg-statoverride are settings of the user, so the same care should be given as to files in /etc, especially one should never override something the user has set in there. (I can think of no example where calling dpkg-statoverride --add without dpkg-statoverride --list in some maintainer script is not a serious bug: Either you override user settings or you are using debconf as a registry. Moral To recap, your package is doing something wrong if:

31 October 2011

Gregor Herrmann: RC bugs 2011/43

a bit late & short (I was travelling the last days) but here it is: the recent update of my RC bug acitivities:

Rapha&#235;l Hertzog: The promising OpenMediaVault failed its debut as free software project

Volker Theile is well known for his work on FreeNAS. But at one point, he decided to give up on it and to restart from scratch but this time based on Debian (instead of FreeBSD). OpenMediaVault is the name of this new Debian based product. Here s how he defines the project:
OpenMediaVault is the next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, DAAP media server, RSync, BitTorrent client and many more. Thanks to the modular design of the framework it can be enhanced via plugins. OpenMediaVault is primarily designed to be used in home environments or small home offices, but is not limited to those scenarios. It is a simple and easy to use out-of-the-box solution that will allow everyone to install and administrate a Network Attached Storage without deeper knowledge.
Even though all the work was private, he managed to attract an important following, and I must say that I was looking forward to this project. He regularly blogged on his progress, sharing some good-looking video of the resulting product (example here). The first public release (Version 0.2, codenamed Ix) happened on October 17th. I have yet to try it but I took a look on the website. As a Debian developer, I was keen on seeing the source code and how the project was managed. The GPLv3 license is presented as an important feature and I was expecting a well-managed open source project. The fact that it was a private one-man project up to now did not bother me, we re quite used to the scratch your itch kind of start for free software projects. Enough to say is that I have been very disappointed. First you come across a contributor agreement, it s frowned upon by many free software developers. But why not, maybe he bought the argument of Mark Shuttleworth and wants to give it a try. But then I looked at the subversion repository, it s obvious that it s just a dumping ground of files that are managed somewhere else in another repository. A bit like Android which is not developed in the open but released as a whole from time to time. But the worst was yet to be found on the licensing page:
Beside being freely available for personal end-users, System Builders and System Integrators, in general Installers, require a commercial license for OpenMediaVault.
Besides the fact that I m still not sure what this means, I would like to know how he reconciles this requirement with the terms of the GPLv3. It s a shame that a so promising project ends up being a disaster from a free software perspective. Have you tried OpenMediaVault already? If yes, I d be glad to read your thoughts in the comments.

9 comments Liked this article? Click here. My blog is Flattr-enabled.

Russell Coker: Links October 2011

Ron has written an interesting blog post about the US as a lottery economy [1]. Most people won t win the lottery (literally or metaphorically) so they remain destined for poverty. Tim Connors wrote an informative summary of the issues relating to traffic light timing and pedestrians/cyclists [2]. I have walked between Southgate and the Crown Casino area many times and have experienced the problem he describes many times. Scientific American has an interesting article about a new global marketplace for scientific research [3]. The concept is that instead of buying a wide range of research equipment (and hiring people to run it) you can outsource non-core research for a lower cost. Svante P bo gave an interesting TED talk about his work analysing human DNA to determine prehistoric human migration patterns [4]. Among other things he determined that 2.5% of the DNA from modern people outside Africa came from the Neandertals. Lisa wrote an informative article about Emotional Support Animals (as opposed to Service Animals such as guide dogs) for disabled people [5]. It seems that the US law is quite similar to Australian law in that reasonable accommodations have to be made for disabled people which includes allowing pets in rental properties even if such pets aren t officially ESAs. Beyond Zero Emissions has an interesting article about electricity prices which explains how wind power forces prices down [6]. This should offset the new carbon tax . Problogger has an article listing some of the ways that infographics can be used on the web [7]. This can be for blog posts or just for your personal understanding. Petter Reinholdtsen wrote a handy post about ripping DVDs which also explains how to do it when the DVD has errors [8], I haven t yet ripped a DVD but this one is worth noting for when I do. Miriam has written about the Fantastic Park ICT training for 8-12yo kids [9]. It s run in Spain (and all the links are in Spanish but Google Translation works well) and is a camp to teach children about computers and robotics using Lego Wedo among other things. We need to have more of these things in other countries. The Atlantic Cities has an interesting article comparing grid and cul-de-sac based urban designs [10]. Apparently the cul-de-sac design forces an increase in car use and therefore an increase in fatal accidents while also decreasing the health benefits of walking. Having lived in both grid and cul-de-sac based urban areas I have personally experienced the benefits of the grid based layout. Sarah Chayes wrote an interesting LA Times article about governments being taken over by corruption [11]. She argues that arbitrary criminal government leads to an increase in religious fundamentelism. Michael Lewis has an insightful article in Vanity Fair about the bankruptcy of US states and cities [12]. Ben Goldacre gave an interesting TED talk about bad medical science [13]. He starts with the quackery that is published in tabloid newspapers and then moves on to deliberate scientific fraud by medical companies. Geoff Mulgan gave an interesting TED talk about the Studio Schools in the UK which are based around group project work [14]. The main thing I took from this is that the best method of teaching varies by subject and by student. So instead of having a monolithic education department controlling everything we should have schools aimed at particular career paths and learning methods. Sophos has an interesting article about the motion sensors of smart phones being used to transcribe keyboard input based on vibration [15]. This attack could be launched by convincing a target to install a trojan application on their phone. It s probably best to regard your phone with suspicion nowadays. Simon Josefsson wrote a good article explaining how to use a GPG smart-card to authenticate ssh sessions with particular reference to running backups over ssh [16]. C ran wrote a good article explaining how to use all the screen space when playing DVDs on a wide screen display with mplayer [17]. Charles Stross has an informative blog post about Wall St Journal circulation fraud [18]. Apparently the WSJ was faking readership numbers to get more money from advertisers, this should lead to law suits and more problems for Rupert Murdoch. Is everything associated with Wall St corrupt?

30 October 2011

Dominique Dumont: Config::Model: How to test configuration models

Hello With Config::Model, configuration models are created to validate configuration data. But, before that, how can we validate a model ? This has long been a sore spot of Config::Model. During its development, I ve progressively enhanced a test script to tests the various models (and their corner cases) shipped with Config::Model. This script is able to test several models with several subtests. Currently, more that 400 unit tests are done on 5 models and about 50 configuration files. Which is great, but usable only inside Config::Model distribution. Which is not so great So I ve bitten the bullet and created a new Config::Model::Tester class from this test script. This new class is shipped with Config::Model 1.260 (also available on Debian/sid). For instance, let s see how this tester is used to test Xorg model (shipped in Config::Model::Xorg). Xorg features a small test file (t/model_test.t):
use warnings;
use strict;
use Config::Model::Tester ;
use ExtUtils::testlib;
my $arg = shift   '';
my $test_only_model = shift   '';
my $do = shift ;
run_tests($arg, $test_only_model, $do) ;
The xorg files to test are specified by t/model_tests.d/xorg-test-conf.pl:
$conf_file_name = "xorg.conf" ;
$conf_dir = "etc/X11" ;
$model_to_test = "Xorg" ;
@tests = (
      name => 'fglrx',  ,
      name => 'modern',  ,
      name => 'vesa',  ,
      name => 'xorg',  ,
      name => 'xorg-ati',  ,
);
1;
And the xorg.conf files are there:
$ ls t/model_tests.d/xorg-examples/
fglrx  modern  vesa  xorg  xorg-ati
Here s the output of the xorg-ati subtest:
$ perl -Ilib t/model_tests.t x xorg xorg-ati
ok 1 - compiled
# Beginning xorg test (t/model_tests.d/xorg-test-conf.pl)
# xorg uses Xorg model on file xorg.conf
# Beginning subtest xorg xorg-ati
ok 2 - Copied xorg example xorg-ati
ok 3 - Read wr_root/test-xorg-ati/etc/X11/xorg.conf and created instance with init() method with warning check
ok 4 - Ran dump_tree
ok 5 - Dumped xorg config tree in full mode
ok 6 - Dumped xorg config tree in custom mode
ok 7 - xorg write back done
ok 8 - Created instance xorg-test-xorg-ati-w
ok 9 - Dumped xorg 2nd config tree in custom mode
ok 10 - compare original xorg custom data with 2nd instance custom data
ok 11 - check that original xorg file was not clobbered
# End of subtest xorg xorg-ati
# End of xorg test
1..11
Now, it s quite easy to create non regression tests for bug triggered by err, unexpected, configuration files: Just drop the file in the example directory and edit the *-test-conf.pl file. This saves a lot of typing. All the best

28 October 2011

Gunnar Wolf: The book: Available for sale

This is an update to my last post regarding the Construcci n Colaborativa del Conocimiento book. The book is, as we have repeatedly stated, available online for download Both as a full PDF or chapter by chapter. In the website you will also find videos of all of the conferences held. But holding a printed book in your hands is just a different experience, isn't it? :-) Anyway, I said I would give here an update on how to get your hands on it. The main venue would be through my University's e-store. I recommend it to anybody interested in buying the book in Mexico. The book's list price is MX$300 (around US$27), but it is currently sold at half price I don't know how long will that price be offered. On the other hand, we also uploaded it to the lulu.com self-publishing service. Of course, given I have not seen the printed results, I cannot assure you the resulting product will be of the same quality as the one we got here, but I have a couple of books I have bought at lulu, and their quality is quite acceptable. So, you can also buy it from lulu.com. Note the 20% discount it shows will be permanent That's what I would get as an author, a payment I decided to forefit given we are 11 authors and it would be unfair to collect it all myself. So, the price at lulu.com is US$12.64 plus shipping Very similar to the price at UNAM. Enjoy!

Russell Coker: Capabilities vs SE Linux

In December 2010 a paper was published by Robert N.M. Watson and Jonathan Anderson from the Cambridge University and Ben Laurie and Kris Kennaway of Google about the Capsicum capabilities system [1]. It seems that the aim of the project is to allow systems that need privileges briefly when they start (such as tcpdump) a safe method of dropping privs. The main project page is here [2]. The focus of the paper is on the Chromium web browser and six different ways of constraining the Chromium sandbox are compared. For the SE Linux comparison they claim 200 lines of code changes as of Fedora 15, in Fedora 16 I couldn t find a Chromium package, so I presume that they mean 200 lines of SE Linux policy (I am not aware of anyone modifying the Chromium source for SE Linux). They note that SE Linux doesn t support separating different sandboxes, while it would be possible to have each sandbox be assigned a different MCS sensitivity label to separate them that option would be unwieldy enough that they are essentially correct in this regard. For SE Linux systems running the MLS policy the correct thing to do would be to run multiple copies of Chromium at different levels to access different sensitivity levels of data, this would normally be done by polyinstantiating the home directory. One thing to note however is that there is no requirement that only one security method be implemented. I can t think of any technical reason why it would be impossible to run SE Linux and Capsicum on the same system. SE Linux could constrain daemons and restrict the access to Capsicum services while Capsicum could be used to give minimum privileges to parts of Chromium. I m not sure that such a combination would offer anything that the MLS users would desire, but it seems that everyone else (the vast majority of computer users) would be served well by a combination of SE Linux and Capsicum. It s disappointing that the paper didn t mention Posix 1003.1e capabilities, but given the lack of use that Posix capabilities get that s understandable. It s also disappointing when someone develops something new and different nowadays and doesn t provide a virtual machine image for it. Installing and configuring something that requires application and kernel changes is a lot of work and most people who are idly curious about the technology won t go to the effort. By today s standards it s not that difficult to share a 1GB filesystem image via Bittorrent.

27 October 2011

MJ Ray: RISE Faces Demutualisation Threat at EGM

Photo of UWE Bristol

UWE Frenchay, Bristol: Venue of the RISE EGM

The RISE co-op is the sole shareholder in the Social Enterprise Mark CIC and its members have been called to an Extraordinary General Meeting during the lunch break of next Tuesday s Knowing and Growing conference at UWE Bristol. The RISE board has proposed Four Special Resolutions that would dissolve the co-op and transfer all assets as windfalls to the SEM CIC and a trust, ignoring RISE Ltd s Memorandum of Association. software.coop is calling on other RISE members to attend the EGM and oppose this demutualisation attempt. Update: dissolved but not demutualised (see below) RISE is constituted as a common ownership co-op and its Memorandum of Association contains a clause that directs the assets to be transferred to another common ownership social enterprise organisation if the co-operative is dissolved. However, unlike the CIC asset lock, there is no independent regulator enforcing it and, unlike in many co-ops, there has been no requirement for new members to pledge to obey the RISE common ownership clause at an individual level and there has been no member education about common ownership in the last three years. This demutualisation is the wrong solution for RISE because: software.coop will vote against the demutualisation, in favour of social enterprise, and calls on other RISE members to show solidarity with the co-operative and common ownership social enterprise movements. Update 2 November: rise has announced its dissolution but they ve got to give further consideration to where the assets go because the demutualisation resolutions were defeated. I fear that they ll still try some way to bail out the Social Enterprise Mark despite the rejection, but I hope they ll do the right thing and give the assets to good common ownership social enterprise like the RISE Memorandum of Association requires.

Russell Coker: Donating old Hardware

On a recent visit to my local e-waste disposal place I noticed an open PC on the top of the pile with a pair of DIMMs that were begging to be removed. I also noticed three PCI Ethernet cards that were stacked in a manner that made them convenient to grab possibly some nice person deliberately placed them so someone like me could take them. The DIMMs turned out to be 3G of DDR2-800 RAM and were regarded as good by Memtest86+ a nice upgrade for one of my test systems that previously only had 1G of RAM. If you have old hardware to dispose of then please try to take the RAM to your local computer users group meeting. In any such gathering there s always someone who wants old RAM, anything better than PC-133 will find good home unless it s very small (128M sticks of DDR-266 and 256M sticks of anything faster probably won t get any interest). RAM is small and light so you can carry it in your pocket without inconvenience. Ethernet cards of all vintages are in demand due to people reusing old desktop systems as routers and PCIe video cards are in great demand, PCI and PCIe cards are small enough that it s usually not a great inconvenience to transport them. Hard drives larger than about 100G are in demand as are ATX power supplies, these are really inconvenient to transport unless you travel by car. For computer systems, anything that can use DDR2-800 RAM will probably be of use to some member of a computer users group, if you offer it on the mailing list then you can expect that someone will want to collect it from you at your home or a meeting. There are organisations such as Computerbank that take donations of old hardware and make systems for disadvantaged people [1], it s worth considering them if you have hardware to dispose of. But for me the hardware I use every day is quite close to the minimum specs for donations that Computerbank will accept so there s no possibility of me discarding systems that are useful to them. I ve created a page listing hardware that I need, if anyone in my area has such hardware that they don t need then please let me know [2].

26 October 2011

Jose Luis Rivas Contreras: and issues on vim with $TERM= screen

Because of tmux I run with $TERM= screen instead of my regular rxvt-unicode, but I had a big issue with VIM because of this, it kept writing and F\n and H\n each time I pressed <end> and <home> keys respectively. I finally found the way to solve this, making VIM use $TERM= xterm internally writing this on my ~/.vimrc:
if &term == "screen"
set term=xterm
endif
So, this way, even in my servers without XTerm installed (but ncurses-term) I can use VIM as I m used to.

24 October 2011

Colin Watson: Quality in Ubuntu 12.04 LTS

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order: Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

Gregor Herrmann: managing $HOME

I'm not the first one who thought about managing their dotfiles & scripts in a (D)VCS in order to be able to have them in sync across different machines, & I even haven't re-invented the wheel for doing so. short report on my experiences:

after looking around a bit I decided to try movein (info page with links to downloads & a README, blog post, git repo) because it looked liked it would easily do what I'd like to do without getting in my way too much; which means e.g.: no symlink farm, & only one simple command to manage the whole setup.

movein is a simple shell script that allows to setup & manage multiple modular repos tracking different sets of files quite easily, taking advantage of the powers of git & mr. & being written in shell means I also was able to make a few changes, which stew kindly accepted & applied to his tree already.

in fact I'm just starting to use it for real, but it looks good so far, & I think I'm going to stick with it for the time being.

2 minor observations: </promotion_time>

23 October 2011

Gregor Herrmann: RC bugs 2011/42

this week my activities were split between some freshly-reported pkg-perl bugs, & looking at some of the older RC bugs. for most of the ones in the latter category I found again ready-to-be-applied patches, many of them from Ubuntu again. details:

Luca Falavigna: Stats, more stats and, guess what? Even more stats!

We all love stats, don t we? So, here we go! Let s start with a graph: NEW graph It shows the number of packages in the NEW queue since last year. You can see a big drop during April 2011, and a reasonably low rate during the last six months. You could think fellow Debian Developers stopped to upload NEW packages. Sorry, you re wrong! :) Since Squeeze release, 3.832 .changes files with NEW components were processed by dak, with an average of 14,85 NEW packages per day. On the FTP Team side, we had 3.732 accepts (14,47 per day), 339 rejects (1,31 per day) and 178 comments to maintainers (0,69 per day).
Who were the most prolific maintainers who got a NEW processing? Here is our special top ten:
  1. Debian Haskell Group (362 packages)
  2. Debian Perl Group (343 packages)
  3. Debian Java Maintainers (161 packages)
  4. Debian Ruby Extras Maintainers (124 packages)
  5. Debian Multimedia Maintainers (100 packages)
  6. Debian Fonts Task Force (96 packages)
  7. Debian Med Packaging Team (79 packages)
  8. Debian Install System Team (61 packages)
  9. Debian Javascript Maintainers (54 packages)
  10. Debian Python Modules Team (50 packages)
That s bad packaging teams cannot bake cookies!
Let s do the same with Changed By, this time:
  1. Ben Hutchings (159 packages)
  2. Joachim Breitner (138 packages)
  3. Clint Adams (134 packages)
  4. Jonas Smedegaard (124 packages)
  5. TANIGUCHI Takaki (97 packages)
  6. Nicholas Bamber (61 packages)
  7. Alessio Treglia (60 packages)
  8. maximilian attems (54 packages)
  9. David Paleino (51 packages)
  10. Torsten Werner (45 packages)
Much better now go and heat up your ovens, we know who you are ;)
Another nice aspect to look at is the speed of NEW processing. Some maintainers were very happy for a fast NEW processing, someone even complained for having been too quick! :) So, let s find out which upload was the quickest ever. Try to gamble a bit before reading the answer, to see whether you are near to the real value ;) Alessio Treglia, you probably already know, because your gwc_0.21.16~dfsg-1 upload has been processed in 41 seconds (yes, forty-one seconds!). Here s an excerpt from ftp-master log to certify it:
20110516120252 process-upload dak Processing changes file gwc_0.21.16~dfsg-1_amd64.changes
20110516120258 process-upload dak Moving to new gwc_0.21.16~dfsg-1_amd64.changes
20110516120339 process-new tolimar NEW ACCEPT: gwc_0.21.16~dfsg-1_amd64.changes
Alex was the super-fast FTP Team member behind the quickest accept, do you want to beat him? Join FTP Team ;)

22 October 2011

Adnan Hodzic: My DebConf11 summary and its after effects

Even though some three months have passed since DebConf11 has successfully ended, I still wanted to give you just a glimpse on how some parts were played through the eyes of a lunatic (read: organizer). Of course, blog post can t come close near of explaining anything but at least it should give some insight and hopefully some pointers to the future DebConf organizers. If you don t feel like reading this much text you can listen to last episode of This Week In Debian podcast and after it just head to Beginning of the end part. DebConf11 First of all I d like to apologize if I offended or hurt anyone in this whole process as it wasn t my intention and during DebConf organization this is nothing irregular, for you to get hurt or you hurting somebody else. One thing that got stuck with me this whole time is when Mart n Ferrari approached me after I did bid proposal of Bosnia/Herzegovina for DebConf11 back on DebConf9 and told me you have no idea what you just got yourself into I played cool and said something along the line of coure I do and boy did I lie. During DebConf organization you re bound to make some of your decisions instantaneously in which you ll lose something, the most you can do is assess what s the thing you can cut the cord on, even though if that thing or a person for that matter might have meant something to you. You ll lose things, relationships, friendships, contacts at the times sense of humor and maybe even common logic. Some of it comes back to you, some the very next day and for some it may take longer. Some of it was lost irreversibly, but even in that case it wasn t lost and could even be traced to some other cause. One thing that I m thankful for is that I didn t lose, not lose but even get distanced from those close to me, and that I didn t lose guy who was with me in this from very first day, my right hand, my partner in crime Velimir ( aroundthfur ). It s not even losing I m talking about here, because even by losing you re gaining something and this is part of your everyday life, it s just that your life frame kinda gets stuck on fast forward button. So once the ride s over you may feel bit dizzy asking yourself what just happened. During this time I was also contacted by Google recruiter regrading software engineer internship but to make a long story short I didn t make it due to my heavy involvement and my lack of time (read: fail). But I don t find this as losing either, first time I was contacted by them, next time they ll be contacted by me maybe even for different position this time (project manager? Big Smile. Even this DebConf wasn t our first bid we lost DebConf7 bid to Edinburgh, but it is perseverance that got us here. The moral I m trying to tell here is if you have a goal never back down from it. What I found to be our biggest problem and what I personally found as most painstaking in all of this was having and reporting to so many different sides and different groups while trying to keep them in same communication channel. Even my right hand (Velimir) was apart from me (Vienna), then place where two of us are coming from is not the place where DebConf is getting organized in. As in the end we ended up having all the parties separated and pulling to their own side, from Government of Republika Srpska to NGO to Debian to local team and to those who remained as undeclared . Option of cutting the cord in this case was missing as these were the components that DebCon11 was made of. Of course this meant you ll be missing workforce and there were days you d wake up driving >= 200 km/h (literally) with one hand on the wheel, other on the phone and your other right hand (Velimir) holding your RedBull. But you wouldn t do this just because you had to and there was no one else to do it, you d do it because you re responsible for the sake of the project and won t be just sitting there because by some unwritten hierarchy rules you re supposed to sit and supervise things. I d advise you do the same thing when you see there s no one else to do something, even if it means making coffee for one of the teams. I m not trying to scare you away from organizing your own DebConf, on the contrary after all I ve been through in past two years if someone asked me would I do it if I knew it was like this, without a doubt my answer would still be fsck yea . All the previous DebConf organizers told me about all of this, but why didn t I give up? For the same reason you won t give up and for the same reason why we believe in the same goal. If anything this is one of the stories you ll want to tell your kids about so When I was your age I used to sleep naked in the snow would turn to When I was your age I organized DebConf11 in Maybe this blog post wasn t even necessary and some things weren t supposed to be said, I guess sometimes I just can t shut up Smile For those who were on DebConf11 I hope you enjoyed and had a great time, otherwise I m saying all of this in vein. As organizer all you see is problems and you miss most of the confrence, but by what I know there are 11 candidate bids for DebConf13 and for me hearing that tells me the results of our efforts. DebConf11 after effects I could write a whole novel about this, but to keep it as short as possible, for last two years as a side project I was working on an idea of Government or some of its institutions migrating to Linux. At first I was somewhat loud about it, then after Microsoft heard about it and after they tried stopping the idea by trying to scare me by trying to interfere with my private life; as that didn t work its lobbyist came even near of obstructing the whole conference within the Government. For the sake of the conference, I convinced the Government that by supporting DebConf it doesn t mean they need to move to Linux and publicly stopped talking about it. I also convinced them that our only goal was to have successful conference and promote alternative options and open ideas. I wasn t lying as I saw this as new opportunity of them concluding on their own why they should or shouldn t not move, the better conference was the more chances of success we had. That s why I tried pushing as many representatives from various companies as in this case we would use reverse psychology where basically no one or few know what Linux or Debian for that matter is, but everybody knows who Google is, so if you have participants from i.e: Google or Austrian E-Health care system talking about how they are using your technology is better way to explain what s it all about really. Eventually Microsoft even had their first ever conference in Bosnia/Herzegovina and you wanna take a wild guess where it was held? Smile In the end we had a great conference, after the conference we were the ones that were approached by some big local companies interested in future co-operation and in the end a meeting with Mark Shuttleworth and President along with the core of Government was scheduled. Topic? Migration to Linux. For me personally this meeting went better then I could possibly even picture it, many topics were discussed and basically it was up to us/me to make a draft of the project plan and submit request for proposal to the Government. There was still some lobbying but it seemed as it all disappeared, runway was clear and open for the lift off. Necessary contacts with potential future partners were made and before any meetings were scheduled I announced I ll be taking a short break. I missed this years vacation, for matter of a fact I missed going to a vacation for last two years. I decided going to Dubrovnik for few days, just to recollect my thoughts, recompile my kernel and switch to lower gear as I was stuck in 6th gear for way too long. Beginning of the end Being on this vacation , even if it was only for few days this gave me an opportunity to look at it all from a different prospective, different angle and that s when it came to me. I realized that even before I was finished with one project without even realizing I was already involved with another one, one that would take much more time and one that might leave me wondering ~5 years later with question what if? After all if get involved with something I m not going to back out, it s either failure is not an option or I won t start doing it at all. Even if it brought me everything I wanted, how would you know if that s what you wanted if you haven t thought about it first? Having DebConf for the first time in this region already showed some (positive) results, but for seeing the actual results it may take a bit longer. We sowed the seeds now the best thing we can do is leave them as it is and see if anything grows out. Of course I could lean over them and protect them from any natural or any other type of hazards, this would also mean when I m not around it wouldn t know how to protect itself on its own. Besides all, interfering with natures way of doing things is rarely a good option. I consider myself to still be relatively young and threatening my well being over safe keeping some seeds and watching over if they grow into something or not doesn t really seem that attractive to me. Maybe one day I ll decide to leave the land, maybe I won t even get to see if anything grows out after all. Maybe some day someone will find this fruit, recognize its worth and claim it as it s own, maybe over time everybody forgets everything and even that this land once was a fertile ground. Maybe, maybe not. I m still a student, I was supposed to graduate last year but I took less classes last year so I wouldn t snap under all the pressure. That s why this year I have less classes and am supposed to graduate in May, that s less then a year until then and I just want to devote some time to myself, figure out what I want with me life, learn the things I want to learn, do the things I want to do. I want to turn back to working on Eclipse, I have some good plans for Eclipse and Android developers, lately I ve been slacking off way too much but I think I have a good excuse Smile In the end, all I know is that I still want to change the world, whether it s doing that inside of Google or in my own company, or some third place; or I ll just be laughing at this statement only time will show. I m hoping that now with all of this out of me I can finally finish the DebConf11 chapter of my life.

Russell Coker: Desks

Lindsay Holmwood has written about the benefits of a standing desk and how to buy one [1]. The case for avoiding sitting is strong, but I couldn t stand up all day. One thing that s been on my list of things to do if I had an unreasonably large amount of spare time or money is to make a reclining computer station. The idea would be to take a bed and mount a TFT display above it so that it can be viewed while lying down. Then a split keyboard would be required so that each hand can be used for half the keys (this would be difficult or expensive). If the keyboard halves were aligned correctly then it would reduce the carpal tunnel problems associated with computer use (which has been a big problem for me in the past and is something that I will probably never fully recover from [2]). As far as I am aware the risk of back problems is eliminated when lying down, so two of the major problems with regular computer desks would be avoided. I don t think that lying down all day would be that great and it wouldn t work for collaborative projects. But as monitors are so cheap nowadays it would be viable to have a second monitor at a desk connected to the same computer. Then I could spend about half my computer time lying down.

21 October 2011

Russell Coker: Dedicated vs Virtual Servers

A common question about hosting is whether to use a dedicated server or a virtual server. Dedicated Servers If you use a dedicated server then you will face the risk of problems which interrupt the boot process. It seems that all the affordable dedicated server offerings lack any good remote management, so when the server doesn t boot you either have to raise a trouble ticket with the company running the Data-Center (DC) or use some sort of hack. Hetzner is a dedicated server company that I have had good experiences with [1], when a server in their DC fails to boot you can use their web based interface (at no extra charge or delay) to boot a Linux recovery environment which can then be used to fix whatever the problem may be. They also charge extra for hands-on support which could be used if the Linux recovery environment revealed no errors but the system just didn t work. This isn t nearly as good as using something like IPMI which permits remote console access to see error messages and more direct control of rebooting. The up-side of a dedicated server is performance. Some people think that avoiding virtualisation improves performance, but in practice most virtual servers use virtualisation technologies that have little overhead. A bigger performance issue than the virtualisation overhead is the fact that most companies running DCs have a range of hardware in their DC and your system (whether a virtual server or a dedicated server) will be on a random system from their DC. I have observed hosting companies to give different speed CPUs and for dedicated servers different amounts of RAM for the same price. I expect that the disk IO performance also varies a lot but I have no evidence. As long as the hosting company provides everything that they offered before you sign the contract you can t complain. It s worth noting that CPU performance is either poorly specified or absent in most offers and disk IO performance is almost never specified. One advantage of dedicated servers in this regard is that you get to know the details of the hardware and can therefore refuse certain low spec hardware. The real performance benefit of a dedicated server is that disk IO performance won t be hurt by other users of the same system. Disk IO is the real issue as CPU and RAM are easy to share fairly but disk performance is difficult to share and is also a significant bottleneck on many servers. Dedicated servers also have a higher minimum price due to the fact that a real server is being used which involves hardware purchase and rack space. Hetzner s offers which start at E29 per month are about as cheap as it s possible to get. But it appears that the E29 offer is for an old server new hardware starts at E49 per month which is still quite cheap. But no dedicated server compares to the virtual servers which can be rented for prices less than $10 per month. Virtual Servers A virtual server will typically have an effective management interface. You should expect to get web based access to the system console as well as ssh console access. If console access is not sufficient to recover the system then there is an option to boot from a recovery device. This allows you to avoid many situations that could potentially result in down-time and when things go wrong it allows you to recover faster. Linode is an example of a company that provides virtual servers and provides a great management interface [2]. It would take a lot of work with performance monitoring and graphing tools to give the performance overview that comes for free with the Linode interface. Disk IO performance can suck badly on virtual servers and it can happen suddenly and semi-randomly. If someone else who is using a virtual server on the same hardware is the target of a DoS attack then your performance can disappear. Performance for CPU is generally fairly reliable though. So a CPU bound server would be a better fit on the typical virtual server options than a disk IO bound server. Virtual servers are a lot cheaper at the low end so if you don t need the hardware capabilities of a minimal dedicated server (with 1G of RAM for Hetzner and a minimum of 8G of RAM for some other providers) then you can save a lot of money by getting a virtual server. Finally the options for running a virtual machine under a virtual machine aren t good, AFAIK the only options that would work on a commercial VPS offering are QEMU (an x86 CPU instruction emulator), Hercules (a S/370 S/390, and Z series IBM mainframe emulator), and similar CPU emulators. Please let me know if there are any other good options for running a virtual machine on a VPS. Now while these emulators are apparently good for debugging OS development they aren t something that are generally useful for running a virtual machine. I knew someone who ran his important servers under Hercules so that x86 exploits couldn t be used for attacking them, but apart from that CPU emulation isn t generally useful for servers. Summary If you want to have entire control of the hardware or if you want to run your own virtual machines that suit your needs (EG one with lots of RAM and another with lots of disk space) then a dedicated server is required. If you want to have minimal expense or the greatest ease of sysadmin use then a virtual server is a better option. But the cheapest option for virtual hosting is to rent a server from Hetzner, run Xen on it, and then rent out DomUs to other people. Apart from the inevitable pain that you experience if anything goes wrong with the Dom0 this is a great option. As an aside, if anyone knows of a reliable company that offers some benefits over Hetzner then please let me know. What I would Like to See There is no technical reason why a company like Linode couldn t make an offer which was a single DomU on a server taking up all available RAM, CPU, and disk space. Such an offer would be really compelling if it wasn t excessively expensive. That would give Linode ease of management and also a guarantee that no-one else could disrupt your system by doing a lot of disk IO. This would be really easy for Linode (or any virtual server provider) to implement. There is also no technical reason why a company like Linode couldn t allow their customers to rent all the capacity of a physical system and then subdivide it among DomUs as they wish. I have a few clients who would be better suited by Linode DomUs that are configured for their needs rather than stock Linode offerings (which never seem to offer the exact amounts of RAM, disk, or CPU that are required). Also if I had a Linode physical server that only had DomUs for my clients then I could make sure that none of them had excessive disk IO that affected the others. This would require many extra features in the Linode management web pages, so it seems unlikely that they will do it. Please let me know if there is someone doing this, it s obvious enough that someone must be doing it. Update: A Rimuhosting employee pointed out that they offer virtual servers on dedicated hardware which meets this criteria [3]. Rimuhosting allows you to use their VPS management system for running all the resources in a single server (so no-one else can slow your VM down) and also allow custom partitioning of a server into as many VMs as you desire.

19 October 2011

Steve Langasek: Debian: not stale, just hardened

Rapha l Hertzog recently announced a new dpkg-buildflags interface in dpkg that at long last gives the distribution, the package maintainers, and users the control they want over the build flags used when building packages. The announcement mail gives all the gory details about how to invoke dpkg-buildflags in your build to be compliant; but the nice thing is, if you're using dh(1) with debian/compat=9, debhelper does it for you automatically so long as you're using a build system that it knows how to pass compiler flags to. So for the first time, /usr/share/doc/debhelper/examples/rules.tiny can now be used as-is to provide a policy-compliant package by default (setting -g -O2 or -g -O0 for your build regardless of how debian/rules is invoked). Of course, none of my packages actually work that way; among other things I have a habit of liberally sprinkling DEB_MAINT_CFLAGS_APPEND := -Wall in my rules, and sometimes DEB_LDFLAGS_MAINT_APPEND := -Wl,-z,defs and DEB_CFLAGS_MAINT_APPEND := $(shell getconf LFS_CFLAGS) as well. And my upstreams' build systems rarely work 100% out of the box with dhauto* without one override or another somewhere. So in practice, the shortest debian/rules file in any of my packages seems to be 13 lines currently. But that's 13 lines of almost 100% signal, unlike the bad old days of cut'n'pasted dh_* command lists. The biggest benefit, though, isn't in making it shorter to write a rules file with the old, standard build options. The biggest benefit is that dpkg-buildflags now also outputs build-hardening compiler and linker flags by default on Debian. Specifically, using the new interface lets you pick up all of these hardening flags for free:
-fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wl,-z,relro
It also lets you get -fPIE and -Wl,-z,now by adding this one line to your debian/rules (assuming you're using dh(1) and compat 9):
export DEB_BUILD_MAINT_OPTIONS := hardening=+pie,+bindnow
Converting all my packages to use dh(1) has always been a long-term goal, but some packages are easier to convert than others. This was the tipping point for me, though. Even though debhelper compat level 9 isn't yet frozen, meaning there might still be other behavior changes to it that will make more work for me between now and release, over the past couple of weekends I've been systematically converting all my packages to use it with dh. In particular, pam and samba have been rebuilt to use the default hardening flags, and openldap uses these flags plus PIE support. (Samba already builds with PIE by default courtesy of upstream.) You can't really make samba and openldap out on the graph, but they're there (with their rules files reduced by 50% or more). I cannot overstate the significance of proactive hardening. There have been a number of vulnerabilities over the past few years that have been thwarted on Ubuntu because Ubuntu is using -fstack-protector by default. Debian has a great security team that responds quickly to these issues as soon as they're revealed, but we don't always get to find out about them before they're already being exploited in the wild. In this respect, Debian has lagged behind other distros. With dpkg-buildflags, we now have the tools to correct this. It's just a matter of getting packages to use the new interfaces. If you're a maintainer of a security sensitive package (such as a network-facing daemon or a setuid application), please enable dpkg-buildflags in your package for wheezy! (Preferably with PIE as well.) And if you don't maintain security sensitive packages, you can still help out with the hardening release goal.

Next.

Previous.