It may come as no surprise to some of you that I actually enjoy -- yes, enjoy -- watching political conventions. I've spent some time watching them over the past two weeks.

McCain has yet to speak tonight as I write this, but I've got to get out a few comments.

First of all, both parties of course were attacking the presidential candidate of the other. But did you notice a key difference? The Democrats attacked McCain on policy, while the McCain people attacked Obama on biography. VP nominee Palin even went so far as to mock Obama's work 20 years ago as a community organizer. I think that is incredibly telling. It seems to me to indicate that the Republicans know that their policies aren't working, and are grasping at straws to find something else. There is plenty in McCain's biography that the Democrats could go after -- his involvement in one of the worst scandals in the Senate, for instance -- but they aren't.

What about the other difference? Hope vs. fear. Obama's speech last week was truly inspiring. I don't think I've ever been inspired by a politician before. Make no mistake about it, he set the bar for himself incredibly high. I don't think anybody since Kennedy has done that. Obama realizes that there are things we must do as a country, and he also emphasized that the "ask not what your country can do for you" theme. Watching Giulianni and Palin was an exercise in fear-mongering. It's a constant stream of "9/11 MUSLIM OIL SHORTAGE TERRORISM NUCLEAR BIN LADEN IRAN GAS PRICES RED ALERT PANIC PANIC PANIC MEXICANS 9/11 9/11 9/11 SCARY WAR WAR WAR." It's as if they have discovered M-x spook RET in Emacs.

The Democrats, on the other hand, are talking about reducing health care costs, making sure everyone is covered, ending war, making the economy better for working-class people. It's an interesting disconnect that Obama highlighted well: McCain has said that the American economy has done well over the past 8 years. And indeed, by some "traditional" measures, such as productivity or GDP, it has. But the vast majority of Americans are not better off today than they were 8 years ago. Obama argues that the way McCain measures economic success is way off; our economy is not doing well if the vast majority of regular Americans are not doing well.

Also telling was how both candidates promised to reach across party lines (Bush promised that too). One actually is working to do it. Take abortion, for example. Obama said that he doesn't agree with the pro-lifers, but surely everyone can agree that providing better funding and support for alternatives to abortion such that it becomes incredibly rare is a good thing. Where's McCain looking for common ground? And then, of course, we have Palin wanting her family to be off limits -- but only when it's convenient to her.

I've voted in the past, but usually as a vote against someone rather than a vote for someone. This year, I'll make a single vote and do both.

My previous post titled AppArmor is Dead [1] has inspired a number of reactions. Some of them have been unsubstantiated opinions, well everyone has an opinion so this doesn’t mean much. I believe that opinions of experts matter more, Crispin responded to my post and made some reasonable points [2] (although I believe that he is overstating the ease of use case). I take Crispin’s response a lot more seriously than most of the responses because of his significant experience in commercial computer security work. The opinion of someone who has relevant experience in the field in question matters a lot more than the opinion of random computer users! Finally there is the issue of facts. Of the people who don’t agree with me, Crispin seems to be the first to acknowledge that Novell laying off AppArmor developers and adding SE Linux support are both bad signs for AppArmor. The fact that Red Hat and Tresys have been assigning more people to SE Linux development in the same time period that SUSE has been laying people off AppArmor development seems to be a clear indication of the way that things are goind. One thing that Crispin and I understand is the amount of work involved in maintaining a security system. You can’t just develop something and throw it to the distributions. There is ongoing work required in tracking kernel code changes, and when there is application support there is also a need to track changes to application code (and replacements of system programs). Also there is a need to add new features. Currently the most significant new feature development in SE Linux is related to X access controls - this is something that every security system for Linux needs to do (currently none of them do it). It’s a huge amount of work, but the end result will be that compromising one X client that is running on your desktop will not automatically grant access to all the other windows. The CNET article about Novell laying off the AppArmor developers [3] says ‘“An open-source AppArmor community has developed. We’ll continue to partner with this community,” though the company will continue to develop aspects of AppArmor‘ and attributes that to Novell spokesman Bruce Lowry. Currently there doesn’t seem to be an AppArmor community, the Freshmeat page for AppArmor still lists Crispin as the owner and has not been updated since 2006 [4], it also links to hosting on Novell’s site. The Wikipedia page for AppArmor also lists no upstream site other than Novell [4]. The AppArmor development list hosted by SUSE is getting less than 10 posts per month recently [6]. The AppArmor general list had a good month in January with a total of 23 messages (including SPAM) [7], but generally gets only a few messages a month. The fact that Crispin is still listed as the project leader [8] says a lot about how the project is managed at Novell! So the question is, how can AppArmor’s prospects be improved? A post on linsec.ca notes that Mandriva is using AppArmor, getting more distribution support would be good [9], but the most important thing in that regard will be contributing patches back and dedicating people to do upstream work (Red Hat does a huge amount of upstream development for SE Linux and a significant portion of my Debian work goes upstream). It seems to me that the most important thing is to have an active community. Have a primary web site (maybe hosted by Novell, maybe SourceForge or something else) that is accurate and current. Have people giving talks about AppArmor at conferences to promote it to developers. Then try to do something to get some buzz about the technology, my SE Linux Play Machines inspired a lot of interest in the SE Linux technology [10]. If something similar was done with AppArmor then it would get some interest. I’m not interested in killing AppArmor (I suspect that Crispin’s insinuations were aimed at others). If my posts on this topic inspire more work on AppArmor and Linux security in general then I’m happy. As Crispin notes the real enemy is his employer (he doesn’t quite say that - but it’s my interpretation of his post).

• [1] http://etbe.coker.com.au/2008/08/23/apparmor-is-dead/


• [2] http://blogs.msdn.com/crispincowan/archive/2008/09/02/go-ahead-make-my-day.aspx


• [3] http://news.cnet.com/8301-13580_3-9796140-39.html


• [4] http://freshmeat.net/projects/apparmor/


• [5] http://en.wikipedia.org/wiki/Apparmor


• [6] http://forge.novell.com/pipermail/apparmor-dev/


• [7] http://forge.novell.com/pipermail/apparmor-general/


• [8] http://forge.novell.com/modules/xfmod/project/memberlist.php?group_id=1864


• [9] http://linsec.ca/blog/2008/09/03/crispin-responds-to-allegations-that-apparmor-is-dying/


• [10] http://www.coker.com.au/selinux/play.html

Share This

I’m now up to five different terms for “independent variable” (X’s) in my methods lecture for today (thanks in part to a Polmeth post that just reminded me of one I’d forgotten to put on the list), and I’m probably missing some. You’d think we could narrow that down a little. The others: covariate, predictor, explanatory variable, and regressor. Funnily enough, I only came up with “dependent variable” and “regressand” for Y, but I’m surely missing some there too.

I hate the Plesk server control panel webapp but our webmaster cooperative still maintains one (but only one) server that uses it. That non-free software means that it’s under our “limited support agreement” which essentially says “you understand that the lack of copyright permissions means this system may break in ways we’re legally forbidden to fix and you have to go beg the copyright holder for help”. Recently, a site broke because it wanted to use a PEAR module and Plesk’s PHP configuration includes an open_basedir which forbids access to /usr/share/pear - that’s stupid. If someone has enough permission to install dangerous stuff into that folder, they definitely have enough permission to edit the PHP configuration. The httpd.include file which contains that PHP configuration has a comment at the top that helpfully tells you to create a vhost.conf file alongside it, instead of editing the httpd.include which gets regenerated by Plesk (wiping out any changes at some near-random time in the future). It doesn’t mention that you need to run /usr/local/psa/admin/sbin/websrvmng -u --vhost-name=your.domain after creating it, else no amount of reloading the Apache httpd configuration will make it use the new file. I found the idea from this Re:morse.nl blog post and then read the –help output to pick the best options for me. It’s not obvious on SWSoft’s Plesk help, which is also stupid. Finally, you can’t edit the httpd.include template (to mention the websrvmng command to future sysadmins, or to change the open_basedir across all sites) because it is a string inside websrvmng and not a template file - very very stupid. I didn’t find the source code for websrvmng and I doubt we have permission to modify it, so I guess this is another thing we can’t fix in a reasonable way, like the limited support agreement says. I could recompile PHP to ignore those open_basedir settings or add /usr/share/pear to it, but that’s one hell of a work-around. It seems like having non-free software (like Plesk) rewrite the configuration files of free software (like PHP) is a good way to cripple the free software. One day I want us to be like Bristol Wireless and have a policy of refusing to deal with such non-free-software systems, but that’s a decision I’ve never won in any organisation I’ve joined yet.

Back from Argentina, back from DebConf. As always, the ~3 weeks I spent there were really great, in as many fronts as I can imagine or describe. But I won't go into that now - For the purposes of this posting, the single thing that I got out of DebConf was looking with envy at all the people that had something that used to be called a sub-notebook some time ago, and now morphed into the more modern(?) name netbook.
Several people were seen with their tiny Asus Eee machines, of various models. And I definitively decided I want one - I was quite close to buying one in Argentina, as they are readily for sale there (surprisingly, in Mexico Asus sells motherboards, but no netbooks)... But I've always prefered waiting or paying a little premium for having an on-country seller and warranty.

Back home, Pooka told me that several stores in Mexico do sell the Acer Aspire One. After a little research, I decided to go for it. Office Depot sells the AAO for MX$4500 (around US$450). The only model they carry comes with Windows XP installed (instead of Linpus Linux), which is a shame I thoroughly repeated to the vendor - But it does come with 1GB RAM and 120GB HDD, much better for my needs than the other model, with 512MB RAM and 8GB SSD. This is, after all, a full (although very small) machine. It has an Intel Atom 270 CPU - I haven't yet measured how it fares, but it feels quite responsive so far for typical desktop tasks.
But what made me really happy about it is the Debian support. The only tricky part was to get the installer going, as it does not have a CD drive to boot from (and I didn't want to completely overwrite my only available USB stick's data). Don't try installing Debian Etch, as its kernel will not support the built-in Realtek RTL8101E network card (maybe etch-and-a-half's kernel does?). My greatest ally in this was, of course, the wiki.debian.org article on the Acer One - I rebooted with Lenny's debian-installer, and everything was smooth from that point on. Propietary firmware is required for the wireless AR5007 card and webcam, but -exactly as documented in the wiki- they are covered respectively by madwifi and linux-uvc.
I did a very regular install, with basically the default desktop and notebook setup. I continue to be amazed... Everything just works! It is not even fun, there are no funny drivers to recompile, no bang-your-head-against-the-wall... Even suspend-to-RAM. It just works.
The only glitch I found so far is that, after suspend-to-RAM, the madwifi module must be removed and reloaded to have wireless network. This is a well-known glitch that can be easily worked around. But besides that, it is... as easy as it gets. And, at such a price, and under 1Kg weight... This computer will get used to go out with me quite often! Battery life is just 2 hours, but for most situations, it's more than enough.

First of all, I don't want to blame individual persons. This is just a note of how disappointed I'm about some parts of Debian that are not complying to licenses when it comes to distributing software. debian-cd embedds copy of syslinux without source A couple of years ago, I took over the maintenance of the syslinux package since its previous maintainer was MIA. The takeover was motivated by to the fact, that having started to take care about live systems, I also started to use syslinux on a daily basis and that the syslinux package in Debian was horribly outdated. At that time, I found out that debian-cd, the toolkit to build the official Debian Installer images, doesn't take syslinux out of the archive at build-time of the image, but rather embedds a copy of the required binaries inside the package itself. So I asked Steve to update the binary what he did. Before the etch release, I needed to ask the debian-cd team to sync again their embedded syslinux copy to match the one I had uploaded to the archive, as it was again outdated. On the other hand, live-helper, the toolkit to build the official Debian Live images, was always using the package out of the archive and did never had that problem. After we have released the Debian Lenny Live Beta 1 images last week, we got reports from people trying them on Apple MacBooks and failed because syslinux, taken out of the archive (version 3.71), is broken on that hardware. Some people stated that Debian Installer images do work. That is because debian-cd has an embedded copy of syslinux (version 3.63) which doesn't have that regressions. Since syslinux version 3.71 is present in testing/lenny as well as unstable/sid, and stable/etch has version 3.31, that means... if debian-cd is embedding a syslinux binary with a different version, it must contain the sources for it (it also needs to contain the sources for it anyway, even if it would embedd binaries of the current version, however, it would be a tiny bit less arguable if its sources would be at least present in the Debian archive). So I checked debian-cd, and surprise, it doesn't contain syslinux sources. Syslinux is licensed under the GNU General Public License, version 2 or later. That means, that if you distribute the binaries and sources together, you can remove both at the same time if you decide to no longer distribute the binary (that is what Debian uses to do). This let me to bug #497270. The option to distribute the binary with a note on how to get the sources, valid for three years, could theoretically be done, but isn't used so far (since it has pretty bad practical implications of keeping sources arround even after having stopped to distribute the binaries). Debian sarge release has incomplete source images Checking fo the consequences for embedding bootloader binaries in debian-cd, I just saw that the Debian sarge release does ship syslinux version 2.04 in its images, but is shipping syslinux 2.11 in its source images. This is another violation of the GPL and I've filled this as bug #497471. ...and syslinux is just one bootloader, only used for i386 and amd64. Someone still needs to check for all the other bootloaders for the other architectures we support (and those also for the etch release). cdimage.debian.org distributes images without sources While browsing arround on cdimage.debian.org, I also found the kde4beta livecds made by the Debian KDE team back in November 2007. Although these images are nice, there is no source available for them at all. They use packages that are not available anymore in Debian since a long time. This time, this doesn't only violate the GPL as in the previous two cases, but almost any copyleft license under which we distribute software in main and that is included on these images. I've filled this as bug #497462. Again, this is not about blaming individual persons. But I'm pretty disappointed by these things. In Debian, we spend a big chunk of time checking licenses of packages before we start distributing them. We have our beloved NEW queue where, after the Debian Developer who has initial uploaded a package and has checked the sources, also ftp-masters are re-checking each and every package to ensure that our archive is kept legal. The NEW queue is a mesurement that consumes a big deal of our time, making uploading packages new packages slow, but this is the prize we pay for ensuring our freedom. And we do also make a big fuss about cluebating upstreams that don't respect licenses (be it intentionally or by accident). However, it appears that as good as our package checks are, we spend little to no time to check our resulting products made from these packages. Update: kde4beta livecds have been removed now from cdimage.debian.org, see #497462. Update 2: debimg does the same crap.

qemubuilder+distcc. I've been playing with regexp in C this morning. That sounds like a pretty remote step towards qemubuilder+distcc, but yes, it is.

When volume becomes high enough you start to observe patterns in SPAM pretty easily. I think that this is primarily because people like to see patterns, whether they are present or not. The trick is determining whether they are real patterns or not, and then to a lesser extent whether they are useful patterns. For example I host mail for a business domain. That means that incoming messages come primarily from existing customers, and very rarely from potential new ones. In practise that means that email is expected to arrive from 9am til 6pm (+/-2hours) Email received at 2AM? Either it is somebody working remotely, a foreign contact, or much more likely it is SPAM. Now clearly you cannot dump all messages received at unusual times of the day, but it is a surprisingly robust SPAM indicator for that particular domain. All heuristics are fallable, but some are useful regardless.. I'd love to know what people can learn from their SPAM. This week I'm handling approximately 80,000 messages a day, per MX, which isn't huge (ie. 2-3 million a month). ObQuote: Highlander

If all goes according to plan, Helen, Ilenia and I will be in Vienna, Austria sometime in the next week or two, to register our daughter as a US citizen. As always, we're interested in meeting anyone local for alcohol/food/coffee/whatever, within the limits of our time there. Send email with a mobile number if you're interested in meeting up at some point. I'm not exactly a frequent traveler, but Ilenia and I both really enjoy meeting people when we go places, so I signed up for this to see if it helps put me in contact with people, either in places I go, or people visiting Innsbruck: http://www.dopplr.com/traveller/davidnwelton

There’s one thing that the Apple Mail client gets right which I’ve never seen anything else try to do - the way it formats messages. Most mail clients seem to offer plain text and HTML as user selectable options and do exactly what they’re told regardless of the content of the message. If HTML is enabled they always send a mail with both text/plain and text/html renditions of the message. Normally the plain text version is a fixed, 80 column version. This is wasteful of bandwidth, especially since very few users actually use any formatting at all, and means that mail programs that don’t do HTML have to treat the mails as though the fixed layout the sending system chooses is important even when it results in poor layout (for example, on mobile devices with small screens). What Apple Mail does here is to only enable the more complex formatting options if they add information that can’t be represented in the less complex formats. By default mails are sent in text/plain with the format=flowed option to let the reader know it can safely reflow the text and no HTML alternative is generated. If something that can’t be represented using format=flowed is included in the message then a HTML alternative is generated - transparently and without user intervention. This is good partly because it’s nice to see format=flowed used, it’s a nice technical solution to the problem, but mostly because it’s great user interface design. Most Apple Mail users will never notice if it is or isn’t generating HTML e-mail, they’ll just see that it’s doing what they expect and won’t have to deal with an option that they probably don’t understand or have much of a view on. Other users won’t be troubled with HTML generated by Apple Mail users unless there is some content in the formatting. It’d be good to see more MUAs implementing similar behavior, at least optionally.

As per the original press release:

In Quebec, access to public markets is the rule while contracts attribution without invitation to tender is the exception. A public market should be transparent, fair and most importantly, open to all. The solutions as well as the propositions must be evaluated objectively on known and accepted criteria. Furthermore, the regulation implies that public markets have to enhance the local economic development as well as the Quebec technologies. From February to June 2008, FACIL has noticed sales of proprietary software for more than 25 million dollars. These purchases were made for products offered by large multinational enterprises, with no regard to suppliers in Quebec.

Much to my surprise, news about this lawsuit have spread as far as Vietnam! Something tells me that this court case might have lasting repercussions...

Xorg 1.4 in experimental is supposed to have input device hotplugging.Does anyone have a Howto for Debian? I tried it, but I couldn't get it to hot-plug my USB mouse, so I'm back to using the regular mouse driver for it again, using the /dev/input/mice in-kernel-hack for hotplugging.P.S. on a recent kernel, you might want to add

blacklist snd_pcsp

to a custom file in /etc/modutils/, in order to avoid your PC speaker showing up as regular audio device. You don't want your regular apps to consider your legacy PC speaker as audio device usually.P.S. No, my blog doesn't have comments. Just send me an email (you know, 'legacy' email) via erich AT debian org.

I've updated my simple Simple SDL based perl game, so that:

• It has a name.
• It has an intro screen which uses that name!
• It has multiple balls.
• It gets harder, ie. more balls are added.
• The collision detection is much improved.

I still need to work on the rebound-angle but otherwise it is as complete as it will probably ever become. It would also be nice if the balls could collide with each other, and be different colours.. Regardless it was a fun diversion for a few hours, and probably tells me that I shouldn't attempt to waste more time doing gamy things, and that maths is too hard for me these days. ObQuote: Interview with the vampire

The Debian Live team is pleased to announce the first beta of Debian Lenny's Live images. Although we missed releasing images for Etch along with the installer images, we are now prepared to release live images within the regular Lenny release process. This is the first official release of Debian Live and the whole team has been working hard during the past 2.5 years to make Debian's own live systems become a reality. Nevertheless, we do need your help to find more bugs and improve the live systems, so please try them out. The images are available at:

	http://cdimage.debian.org/cdimage/lenny_live_beta1/

Please also have a look at the known issues listed below. Main features 100% Debian The build process of Debian Live basically consists of creating a Debian chroot, installing one or more kernels along with live-initramfs (a set of hooks into initramfs-tools for handling booting from read-only media) and generating a bootable image from that. This process is handled by live-helper, a collection of shell scripts that allow us to automate and customize this process. Considerable care is taken to ensure that the resulting live system is not tainted by the host system and that installed packages are not modified morethan absolutely necessary. This ensures that Debian Live really is Debian, and not "just another" a Debian-based live system. Flavours Although live-helper is a toolkit to produce your very own live systems with only a few steps, we also provide prebuilt images that are meant to be used as reference systems for end-users. Currently, this consists of the three major desktop environments (GNOME, KDE and Xfce), as well as a small 'standard' image without a graphical environment. For the desktop environments, package selection is performed by 'tasksel' with the respective desktop task, whilst the 'standard' image contains only packages of "Priority: standard" or greater, notwithstanding a handful of live-specific packages (console-common, eject, file, kbd, live-initramfs, locales, sudo and vim-tiny). Image types Debian Live offers prebuilt images for CD/DVD discs, USB sticks (or any HD-media-like device), tarballs (for PXE netboot) as well as a bare squashfs image to boot from the web directly. Live Magic Live Magic is an GUI frontend around the live-helper scripts, offering a subset of the features of live-helper in an easy-to-use graphical user interface. live-magic 1.0 was recently uploaded to sid and is the recommended version. It currently supports 7 languages. Live Installer Live Installer is a special udeb for the Debian Installer that (optionally) replaces a part of d-i in order to install the system from the live image instead of to bootstrapping it from .deb packages. This way, a live system can be easily installed to the harddisk, ensuring that the look and feel of the installation (including preseeding) works the same as the regular installer process. Unfortunately, live-installer does still have a few minor bugs left and is thus not included in our builds yet; we hope to be able to include it in the next beta. Known issues in this release

• The prebuilt images for gnome-desktop and kde-desktop are a bit too big to fit on a CD. Although the automatic installation of Recommends was disabled to build the images, they are still too big. This will need further tweaking as they are supposed to fit with the next beta.
  
  
• The rescue flavour, containing system rescue and forensic related packages, is missing in this beta release.
  
  
• There will be a DVD image with the next beta that includes all three desktop environments so that you can choose at boot time which system you would like to start.
  
  
• Due to time constraints, the prebuilt images for Beta1 are only covering i386 and amd64; with the next beta, powerpc and sparc will follow (if you wish to test these architectures earlier, please build them yourself).
  
  
• The new desktop artwork is not yet included.
  
  
• The syslinux menu is still the old, prompt-based one. A freshly made new syslinux vgamenu using the official lenny desktop artwork is on the way (the same as d-i media will use in Lenny).
  
  
• No live-installer included yet (if you wish to test live-installer earlier, please build it yourself).
  
  
• No win32-loader included yet.
  
  
• Sid packages used - At the time of building the live images, both live-helper 1.0.0-1 and live-initramfs 1.139.1-1 from sid were used. Since the latter has not yet migrated to testing, it was included manually as a local package. This is undesirable, as the release is supposed to be self-contained - however, it is just a convenient workaround as both were granted freeze exception and will migrate soon.

Plans for next Beta release We are looking forward to upload Beta2 in about two weeks from now (maybefollowed by a third beta) with one final RC after that which should be identical to the final release. Some more details about the open things we would like to address in Beta2 and later can be found at the wiki page. The Debian Live team is still looking for more contributors for new features (post-lenny, though) as well as documentation writers for the manual (always). If you care about live systems, please join and help! Thanks Last but not least, our thanks goes to everyone who has contributed and to all maintainers that have kindly fixed live-specific bugs in their packages.

The Problem: A problem with virtual machines is the fact that one rogue DomU can destroy the performance of all the others by inappropriate resource use. CPU scheduling is designed to allow reasonable sharing of computational resources, it is unfortunately not well documented, the XenSource wiki currently doesn’t document the “credit” scheduler which is used in Debian/Etch and CentOS 5 [1]. One interesting fact is that CPU scheduling in Xen can have a significant effect on IO performance as demonstrated in the paper by Ludmila Cherkasova, Diwaker Gupta and Amin Vahdat [2]. But they only showed a factor of two performance difference (which while bad is not THAT bad). A more significant problem is managing virtual memory, when there is excessive paging performance can drop by a factor of 100 and even the most basic tasks become impossible. The design of Xen is that every DomU is allocated some physical RAM and has it’s own swap space. I have previously written about my experiments to optimise swap usage on Xen systems by using a tmpfs in the Dom0 [3]. The aim was to have every Xen DomU swap data out to a tmpfs so that if one DomU was paging heavily and the other DomUs were not paging then the paging might take place in the Dom0’s RAM and not hit disk. The experiments were not particularly successful but I would be interested in seeing further research in this area as there might be some potential to do some good. I have previously written about the issues related to swap space sizing on Linux [4]. My conclusion is that following the “twice RAM” myth will lead to systems becoming unusable due to excessive swapping in situations where they might otherwise be usable if the kernel killed some processes instead (naturally there are exceptions to my general rule due to different application memory use patterns - but I think that my general rule is a lot better than the “twice RAM” one). One thing that I didn’t consider at the time is the implications of this problem for Xen servers. If you have 10 physical machines and one starts paging excessively then you have one machine to reboot. If you have 10 Xen DomUs on a single host and one starts paging heavily then you end up with one machine that is unusable due to thrashing and nine machines that deliver very poor disk read performance - which might make them unusable too. Read performance can particularly suffer in a situation when one process or VM is writing heavily to disk due to the way that the disk queuing works, it’s not uncommon for an application to read dozens or hundreds of blocks from disk to satisfy a single trivial request from a user, if each of these block read requests has to wait for a large amount of data to be written out from the write-back cache then performance will suck badly (I have seen this in experiments on single disks and on Linux software RAID - but have not had the opportunity to do good tests on a hardware RAID array). Currently for Xen DomUs I am allocating swap spaces no larger than 512M, as anything larger than that is likely to cause excessive performance loss to the rest of the server if it is actually used. A Solution for Similar Problems: A well known optimisation technique of desktop systems is to use a separate disk for swap, in desktop machines people often use the old disk as swap after buying a new larger disk for main storage. The benefit of this is that swap use will not interfere with other disk use, for example the disk reads needed to run the “ps” and “kill” programs won’t be blocked by the memory hog that you want to kill. I believe that similar techniques can be applied to Xen servers and give even greater benefits. When a desktop machine starts paging excessively the user will eventually take a coffee break and let the machine recover, but when an Internet server such as a web server starts paging excessively the requests keep coming in and the number of active processes increases so it seems likely that using a different device for the swap will allow some processes to satisfy requests by reading data from disk while some other processes are waiting to be paged in. Applying it to Xen Servers: The first thing that needs to be considered for such a design is the importance of reliable swap. When it comes to low-end servers there is ongoing discussion about the relative merits of RAID-0 and RAID-1 for swap. The benefit of RAID-0 is performance (at least in perception - I can imagine some OS swapping algorithms that could potentially give better performance on RAID-1 and I am not aware of any research in this area). The benefit of RAID-1 is reliability. Now there are two issues in regard to reliability, one is continuity of service (EG being able to hot-swap a failed disk while the server is running), and the other is the absence of data loss. For some systems it may be acceptable to have a process SEGV (which I presume is the result if a page-in request fails) due to a dead disk (reserving the data loss protection of RAID for files). One issue related to this is the ability to regain control of a server after a problem. For example if the host OS of a machine had non-RAID swap then a disk failure could prevent a page-in of data related to sshd or some similar process and thus make it impossible to recover the machine without hardware access. But if the swap for a virtual machine was on a non-RAID disk and the host had RAID for it’s swap then the sysadmin could login to the host and reboot the DomU after creating a new swap space on a working disk. Now if you have a server with 8 or 12 disks (both of which seem to be reasonably common capacities of modern 2RU servers) and if you decide that RAID is not required for the swap space of DomUs then it would be possible to assign single disks for swap spaces for groups of virtual machines. So if one client had several virtual machines they could have them share the same single disk for the swap, so a thrashing server would only affect the performance of other VMs from the same client. One possible configuration would be a 12 disk server that has a four disk RAID-5 array for main storage and 8 single disks for swap. 8 CPU cores is common for a modern 2RU server, so it would be possible to lock 8 groups of DomUs so that they share CPUs and swap spaces. Another possibility would be to have four groups of DomUs where each group had a RAID-1 array for swap and two CPU cores. I am not sure of the aggregate performance impact of such a configuration, I suspect that a group of single disks would give better performance for swap than a single RAID array and that RAID-1 would outperform RAID-5. For a single DomU it seems most likely that using part of a large RAID array for swap space would give better performance. But the benefit in partitioning the server seems clear. An architecture where each DomU had it’s own dedicated disk for a swap space is something that I would consider a significant benefit if renting a Xen DomU. I would rather have the risk of down-time (which should be short with hot-swap disks and hardware monitoring) in the rare case of a disk failure than have bad performance regularly in the common situation of someone else’s DomU being overloaded. Failing that, having a separate RAID array for swap would be a significant benefit. If every process that isn’t being paged out could deliver full performance while one DomU was thrashing then it would be a significant benefit over the situation where any DomU can thrash and kill the file access performance of all other DomUs. A single RAID-1 array should handle all the swap space requirements for a small or medium size Xen server One thing that I have not tested is the operation of LVM when one disk goes bad. In the case of a disk with bad sectors it’s possible to move the LVs that are not affected to other disks and to remove the LV that was affected and re-create it after removing the bad disk. The case of a disk that is totally dead (IE the PV header can’t be read or written) might cause some additional complications.

• [1] http://wiki.xensource.com/xenwiki/Scheduling


• [2] http://www.cs.ucsd.edu/~dgupta/papers/per07-3sched-xen.pdf


• [3] http://etbe.coker.com.au/2008/05/22/xen-and-swap/


• [4] http://etbe.coker.com.au/2007/09/28/swap-space/

Share This

So it looks like the project I’ve been laboring on has been getting some attention:

Ever thought it was difficult to write software for Linux? For multiple distros? InternetNews reports that the LSB is making a push for their next release (due out later this year) that should help make all that much easier.

They even link to our project status page. Cool! Of course, good publicity invites criticism. This time, there seem to be two themes. William Pitcock seems to have the most succinct summary:

To put things simply, the LSB sucks. Here s why:

• The LSB spec depends on RPM. I mean, come on. Seriously. Why do they need to require a specific package manager? If package handling is really required, then why not create a simple package format that can be converted on demand into the system package format? Or why care about packages at all?
• The LSB spec invents things without consulting distros. Like the whole init scripts thing. But that s not as bad as depending on RPM or requiring a specific layout.

(See also Scott James Remnant.) Let’s take this one part at a time. Today’s topic: packaging. Part of William’s problem may be that he doesn’t understand the spec. The LSB doesn’t require a specific package manager, or a specific package format. It doesn’t even require that the distribution be set up using package management at all! The spec only requires that LSB-compliant software be distributed so that any LSB-compliant distribution can install it. That could be tarballs with POSIX-compliant install scripts, an LSB-compliant install binary, a shar archive, a Python script with embedded base64 binaries, whatever. One of the options allowed is an RPM package, with a number of restrictions. The restrictions are key, because they effectively define a subset of RPM that acts as, to quote William again:

…a simple package format that can be converted on demand into the system package format…

The difference being, of course, that we didn’t reinvent the wheel and create our own; we used a popular format as the basis for ours. Scott raises another concern:

While much of the LSB can be hacked into a different distribution through compatibility layers and tools, such as alien, what ISV or other vendor wants to provide a support contract against a distribution that has such kludges?

I’m not sure if he’s referring specifically to packaging or to the standard in general. As regards packaging: the reason we specify a strict subset is because we can test that subset, and we’ve tailored it to the needs of tools such as alien. The theory goes that alien isn’t a kludge when it comes to LSB packages. But, as already mentioned, if vendors aren’t comfortable with supporting RPM, they have a number of other options. As it turns out, most of them are doing just that; the feedback we’re getting from most ISVs is that packaging (whether LSB-subset RPM, full RPM, or Debian) is just not worth the effort. Coming up: part 2…

Probably still affected by jetlag, I've decided at the end of last week while going through my Debian mailinglists backlog, to (try to) fix one RC bug per week from now on until Lenny is released. I guess I was also influenced by Neils talk about Lenny at DebConf8 :-) At that time, which it is merely nine days ago, there were 378 RC bugs open in Lenny. Today we have 428 :-(

Also I decided I will blog about this regulary, in the hope that others pick up similar procedures for them, so that we can release Lenny sometime soon. We, as a project, need to gain speed in fixing these issues, this will only happen if more people start fixing these bugs. After that, we can happily go back to adding more features! ;-)

So for now I closed (or contributed to closing) three bugs: #492299: downgraded & closed, generic bug report, users didnt provide moreinfo, #495587 (bug in atd, caused by a bug in lsb-base, suggested to forcemerge), #438885 (downgraded, as all blocking bugs for this one have been fixed).

For 475036 (removal of kernel-package requested by maintainer) I updated status in the bugreport and notified the kernel-team, that they are one of the two last users of this package in Debian. Not sure which change is more appropriate now: fix kernel-package, so that it can stay in Lenny, or change the linux-2.6 build-system.

The other package which uses kernel-package is cdfs-src, but I'm not sure we should with that package anyway, see 482075.

That's it for this week. Oh, I forgot to mention why I choose one RC bug per week: it's a goal I can fullfill. If I'd choose more, I might not manage to keep up with my plan and stop it altogether. So I choose something which I hopefully can manage over a some period of time ;-) I'm aware this is not very much, but at least something.

What's your plan to help releasing Lenny on time?

I took a note of the various talks I attended at DebConf, with the aim of writing a few notes about each so that I could try to explain to my work mates exactly what I get out of going. However I think a lot of the benefit is about general face to face contact with fellow DDs and random conversations that happen. Having worked from home for over 3 years I'm finding it quite nice to be back in an office with people working on the same things as me; there are conversations that you will have over a quick cup of coffee or in the corridor that you wouldn't bother typing on IRC or email or picking up the phone for. Debian is a lot like a large organisation full of telecommuters and DebConf provides a vital opportunity for us to have those "minor" conversations that prove quite useful and to remind each other what we're actually like in person. I didn't end up writing a lot of notes during the talks so these are just snippets after the fact. If I'd been more organised I'd have done a write up on a daily basis rather than leaving it all until the week after. A lot of the below is fragmented thoughts, but the longer I leave it the less relevant it gets and the more chance I forget something. It'd have been up faster if I hadn't been fighting Movable Type to try and get some sort of headings. Sunday
Introduction, Marga The "Welcome to DebConf" bit. Marga got up and talked about how at DebConf4 (Brazil) she hadn't been a DD and now she was and how it had been a life changing event. She'd thought she was to blame for DebConf8 in Argentina, but trawling mail archives had shown it was originally Mart n's idea. She hadn't realised at the start quite how much work it would be. Steve's DPL talk, Steve McIntyre The DPL welcomes us to DebConf and talks about Debian's achievements, some of the results of his team survey (the teams largely say they need more manpower. Not really a huge surprise.). NM/User Survey BOF, Paul Wise Paul emailed debian-user, debian-newmaint and various other places (I think) asking about people's experiences as a Debian user or applicant to New Maint. His final results aren't yet published and I got the impression a disappointing number of our users had replied. The NM problems these days seem to be largely down to a lack of AMs, at least according to the stats on nm.debian.org. Meet SPI board - bdale, maulkin, jimmy, michael, others A chance to meet those of the SPI board who were present. The usual "If you're involved with Debian you should seriously consider joining SPI". Some discussion about the fact that although SPI was originally mostly Debian these days other projects are making more and more use of it - OFTC, PostgreSQL and Madwifi for example. Not a huge amount of new information for me. I've been an SPI member for some time and I think they perform a vital role for Debian. They've had issues in the past but it's really good to see them overcome and more projects on board. Monday
http://hp.com/go/debian, bdale Bdale talks about the history of Debian and HP. And makes the point that HP aren't doing any of this to be "nice"; their primary responsibility is to their shareholders. Not surprising, but it's good that even with that criteria at the fore front Debian ends up a good idea for a large multinational. Managing 666 packages, Mart n Ferrari Mostly centered around the pkg-perl experience and the tools/workflow they have to help manage things. Impressive web tool (DEHS) to help track packages with new upstreams, issues or pending uploads. Quality Assurance in Lenny + 1 I can't remember a lot of this, which is a bit bad. I remember a discussion about whether we should release with orphaned packages, or if they should be removed. The argument for not removing them seemed to revolve around a theory that if they weren't buggy then what was the harm, the argument for was that if they were unmaintained and not really used than how did we actually know they weren't buggy, and was it reasonable that they potentially held up library transitions etc? Bugs in large packages Don Armstrong and ways in which the BTS can help people with lots of bugs. This (thankfully) doesn't affect me, but it was interesting to hear Don talk about his work on the BTS and his various ideas. I was sorry I'd missed his talk on SOAP access to it; I shall have to find time to watch the video. Tuesday
Debian and Ubuntu, Mark Shuttleworth The usual interaction with Debian stuff, including how we interact with our upstreams. If a bug is raised with an Ubuntu package then it might well be appropriate to raise it with Debian and upstream as well, but each set of people should be able to use their own bug reporting systems rather than having to go to a foreign system. I don't think it was actually said, but I guess the idea is that launchpad is the thing that will do this gluing together. He also talked about Ubuntu as an "experiment" rather than a fork of Debian; something that has potentially different ideas about how to do things but that these ideas may lead to things that should be fed back into Debian. The point was made that even if you consider Ubuntu a fork they've forked 9 times now; it hasn't been a case of forking once and diverging wildly since thast point. I have a few thoughts on all the whole Debian/Ubuntu stuff which I should really try to form into something coherant and post. I think it's important to take Mark's talk in the context of bdale's "Shareholders come first" point. Derivatives round table Didn't stay for most of this. Ubuntu, Debian-EDU (which I hadn't realised was a derivative rather than a team within Debian, but apparently there are various things it was much easier for them just to patch now and try to work towards a generic solution later), someone from Extramadura and someone from the Munich government Linux stuff. I'm sure I've missed someone off. tdebs/i18n This was a useful BOF. It helped that we said no to the usual video team requirement that if you want to speak you had to wait for the microphone. This works fine for a normal talk, but really impedes the flow of conversation in something like a BOF. tdebs are something that have partly come from a need of Emdebian to reduce installed package size and partly from the desire of the i18n team to be able to update package translations without having to do a full binary NMU of a package. We had some of the FTP team, the release team, the i18n team and Emdebian there and I think some progress has been made. There'd been various conversations had before this meeting which I think helped people think about the issues and objections so that we had answers by the time we all sat down together - an excellent example of the use of DebConf. DebConf9 C ceres Various bits of discussion about next year's DebConf in Extramadura. Temperature is likely to be an issue for me as they reckon it could easily get up to 40C! I'll melt! Venue sounds good; some queries regarding late night hacklabs and net connections to same, but I'm sure the networking team will sort it all out as usual. ;) DebConf10 bids Venezela and New York were both pitching. Lots of people don't want to travel to the US or think they'll have problems doing so (I hadn't realised there was such an issue for people from South America). I suspect various people will have issues with Venezela as well; I heard expressions of doubt over their political stability expressed. However I'd be happy to go to either (and preferably both! DebConf10 + 12 perhaps?) Speaking of which, what ever happened to the Sarajevo bid? If they're reading, have you considered DebConf11? Wednesday
A day trip to an Argentinian ranch and lunch was an asido. Fantastic food and a pleasant afternoon spent sitting in the sun chatting to people. That evening was the formal, but thanks to having stayed up until after 4 on Tuesday I had to wimp out and go to bed fairly early. Not until I'd seen the DPL dance though. :) Thursday
Internationalization in Debian I forget most of this other than pretty graphs from Christian regarding what's translated and what other languages we might have; e.g. you can consider South America to be covered largely by Spanish and Portuguese, or you can consider it to not be covered very much at all by the local indigenous languages. dh-make-webapp: yeah right! Mainly what I took away from this was "It'll never happen thanks to the fact that every web app author uses a different system" and that more work needs to be done on the web apps best practices document. I don't really do web apps, but having them easily installable on my systems is a good thing so I hope some standardisation within Debian comes from this. Best practises in team-maintaining packages Various team members got up to talk about work flow in their teams. Not unsurprisingly different things seem to work for different people. Predictable PRNG in the Vulnerable Debian OpenSSL Package Lucian giving his Black Hat talk about the OpenSSL issues we faced earlier this year. Some queries about automated testing and that it should have found this, which originally I agreed with until I thought about it - you'd have to have run > 65536 tests and compared all the data to find this; something like the FIPS tests against a single run wouldn't have shown any issues. Lucian wanted feedback on ensuring he was presenting the Debian side of things acceptably as well. Keysigning Run by Don Armstrong. Nothing to really remark except I think key signing attendance is dropping off after things like the Helsinki experience. Also I've now managed to get my new RSA key signed by keys other than my own (and I'll try to get sorted out to sign keys myself soon). Friday
LessWatts Intel's drive towards reducing power usage. Odd to hear such things from someone who wasn't Matthew Garrett and I think I'd heard most of it before. Debian technical policy update Sorry Manoj, I should have been paying more attention. Debian Policy is something I think is important, but I'm not enough of a nitpicker to really feel I can contribute a whole lot. Virtualisation in Debian - Present and future This was depressing. What I took away was "We had 2 forms of virtualisation in etch (Xen + vserver) and we're supporting neither for Lenny". As someone running a Xen machine this is problematic. The host does have the hardware support that would allow kvm, but I'm worried about smooth migration (it's a colo box). There was some mention of Xenner, which isn't packaged for Debian. I took a look at doing so but there are some worrying items in the TODO list (like, oh, locking to enable reliable SMP) and I've spectacularly failed to get it compiling so far. I should have another go. Emdebian update Neil Williams about where Emdebian is and waved his Balloon3 about. Some interest from the Openmoko guys and discussion about why it might be better than OpenEmbedded. And equally what it couldn't do that OpenEmbedded can manage (mainly very small installs; a minimal Emdebian install is 24MB+ and with X that goes up to 75MB+ I believe). netconf I'd never actually seen Martin talk about netconf before. Am impressive flow of control diagram. It all sounds quite complex for little benefit at present, but if he pulls it off then I can see it being a much more flexible replacement for the likes of Network Manager. Saturday
Lenny - The Road To Release Neil played with putting the heads of the release team on random images. And tried to tell us all how to behave during the release process in terms of what should and shouldn't be uploaded etc. Most of it is just common sense, but apparently everything he talked about he'd experienced. Also pointed out we were supposed to release in 3 weeks and didn't even have the appropriate kernel in testing yet so it's probably not going to happen. Colour me surprised. Sustainable Computing Low power consumption/affordable computing. Interesting points about the OLPC being used in the middle of nowhere with no chance of an Internet connection being close by compared to neighbours in large cities where people can't necessarily afford top of the line computers or individual connections - in this sort of scenario mesh networking is really viable in terms of getting people connected up. Multi winner voting Manoj needs pretty diagrams, probably drawn by Martin Krafft. ;) Semi-interesting discussion about multi-winner voting (e.g. SPI board elections) and how the current Debian voting system is completely tied to Debian's infrastructure eg the LDAP setup. Manoj would like to make it more generic so it could be used by other groups as well as for different sorts of votes in Debian.

Apparently the LUG’s website is still off-line and there’s no idea when it will return, so let me use this site to help announce that the LUG’s regular meeting at the Knights Templar public house, near Isambard Walk, Bristol, will happen tomorrow Saturday 23 August from noon-ish until whenever people drift away. Software Freedom Day happens between this meeting and the next one. Is anyone interested in running a Bristol-area event and do you know of a suitable/available venue? I’m finding the SFD official site somewhat dysfunctional (I’m not in the Northern Caucasus, damnit), so I’ve not found a nearby team registered yet…

Venezuela’s biggest and most important ISP, CANTV, was my first serious email provider, since it is tied to my ADSL connection, and I previously used several different dial-up providers (T-Net, IFX Networks) with no specific interest in email. CANTV mail service used to be provided with the highest standards of quality in the country, particularly as per malware and UCE control, and general abuse surveillance. Hey, even one of the former insider guys is quite known inside our FOSS community! I’ve previously (1, 2) blogged about the huge problems CANTV is facing with their mail service. They don’t answer, and sometimes even bounce, mails to abuse or postmaster, and temporarily suspended or degraded their abuse statistics web site. All in all, they are providing a poor customer service and are being listed in several blacklists all around the world. Sometimes I get warnings from mailing lists since their border MTAs are refusing to deliver messages to the mailboxes. So if you sent me my signed PGP public key, please forgive my ISP for probably bouncing your message. I thought I could live with this problem, since it’s probably temporary, but today a friend from El Salvador told me that all of his mail to cantv.net addresses is bouncing with this message.

Diagnostic-Code: smtp; 550 5.7.1 ...>... Mail from
    168.243.89.79 refused: Vea http://abuso.cantv.net/bl/dul.html

So after taking a look at the whois record for that block, and doing a lookup on the reverse entry for 168.243.89.79, I think that a proper conclusion would be Venezuela’s biggest ISP is currently blocking most mail from El Salvador. And probably other countries. Terrible news. And please use my at debian.org mail address if you need to get in touch.