It's the X Development Summit in Edinburgh this week, so I've been hanging out with the graphics crowd. There hasn't been a huge amount of work done in the field of power management in graphics so far - Intel have framebuffer compression and there's the lvds reclocking patch I wrote (I've cleaned this up somewhat since then, but it probably wants some refactoring to avoid increasing CPU usage based on its use of damage for update notifications). That still leaves us with some fun things to look at, though.

The most obvious issue is the gpu clock. Intel's chipset implements clock gating, where unused sections of chip automatically unclock themselves. This is pleasingly transparent to the OS, and we get power savings without any complexity we have to care about. However, there's no way to control the core clock of the GPU - it's part of the northbridge and fiddling with the clocking of that would be likely to upset things. Nvidia and Radeon hardware is more interesting in this respect, since we can control the gpu clock independently of anything else. The problem is trying to do so in a reasonable way.

In an ideal universe, we can change these clock frequencies quickly and without any visual artifacts. That way it's possible to leave it in the lowest state and clock it up as load develops. There's a couple of problems with this - non ideal hardware, and the software in the first place. Jerome's been testing a little on Radeon and discovered that changing the memory clock through Atom results in visual corruption. It's conceivable that this is due to some memory transaction cycles getting corrupted as the clock gets changed. If we could ensure that the reclock happens during the vertical blank interval, that's something that could potentially be avoided (of course, then we have the entertainment of working out when the vertical blank interval actually is when you have a dual head output...). The other problem is that 3D software tends to consume as many resources as are available. Games will produce as many frames per second as possible. Demand-based clocking will simply ramp the gpu to full speed in that situation, which isn't necessarily what you want in the battery case (as the number of frames per second goes up, so does the cpu usage - even more power draw) but is probably pretty reasonable in the powered case.

Handwavy testing suggests that this can save a pretty significant amount of power, so it's something that I'm planning on working on. Further optimisations include things like making sure that we're not running any PLLs that aren't being used at the time (oscillators chew power), not powering up output ports when you're not outputting to them and enabling any hardware-level features that we're currently ignoring. And, ideally, doing all of this without causing the machine to hang on a regular basis.

I recently announced that German and French reached 100% completion in unstable for po-debconf translations (by the way, both are now 100% in lenny as well). However, l10n efforts also happen for many languages in that area, so it's time to highlight them as well:

• Third is Portuguese at 98.3%. Nearly all complete. It would need some overview by the team, lead by Miguel Figueiredo, to complete the currently incomplete files.
• Czech if 97.85%. The effort is a very longstanding one by Miroslav Kure, nearly alone. Miroslav has small difficulties to cope with changes in eth->lenny release cycle which explains why, despite they once were virtually 100%, cs could never reach it. This will happen for squeeze
• Vietnamese is 84.97%. Clytie Siddall fights with changes and her real life conditions and still succeeds in keeping the l10n effort up. I'm admirative
• Dutch is 80.72%. The Dutch team is an "old" team in Debian with a thorough, but sometimes slow, QA process. I have no doubt about the quality of their work. The manpower seems fairly short, though, with people involved in many other areas in Debian and outside.
• Swedish was slowly vanishing until Martin Bagge recently resurrected it. They even passed over Russian again at 78.76%. Congratulations.
• Russian is constantly progressing. Kudos are due to Yuri Kozlov who efficiently leads the team. They'll make it at 100%, I don't doubt it. 78.71% for now.
• Galician is the longstanding one-man effort by Jacobo. He already described in details his work process and took great advantage of my NMU campaigns to progress constantly. 76.80%: will he pass over Swedish and Russian before lenny?
• Spanish is an "old" team. I expected to see a good progress, which doesn't really happen. The NMU campaigns do not benefit them and I don't see that many bug reports. What's happening there? Tis team is one I'm most worried about.
• Finnish took great advantage of the NMU campaing with Esko Ar j rvi following it very closely. He bumped the stats from less than 40% to 66.5%
• Japanese slowly goes its own way. The effort is less constant than others, but they successfully manage to stay around 60%
• Basque is also constantly progressing. Under the lead of Piarres Beobide, the team reached 60% after starting below 40%
• Brazilian Portuguese is more focused on DDTP these days (they have impressive stats, thanks to Felipe can de Wiel's work) so they just manage to stay around 55%

Other translation efforts are more hectic. I'd like to point the slow decrease of the work on Catalan, probably due to low manpower and other areas of interest for the old timers. I'll try to soon make such report for other areas of translation effort in Debian.

Some time ago I wrote a small shell script that counted the number of dependencies of each package. It used to take lots of time to perform such a simple operation, so today I decided to rewrite it from scratch in perl and even made it able to sort the packages according to the number of pre-/dependencies.

---

So, here are 20 packages from sid, which are in the top 20 of most number of pre-/dependencies:

texlive-full:103
libjifty-perl:82
brdesktop-gnome:74
rhythmbox:72
evolution:66
xorg-dev:65
epiphany-gecko:58
gnome-control-center:55
epiphany-webkit:55
iceweasel-l10n-all:55
anjuta:53
gnucash:52
gnome-core-devel:52
brdesktop-common:52
gnome-desktop-environment:51
mplayer:51
beagle:50
dtc-common:49
totem-gstreamer:49
gnome-power-manager:49

Note that ORed dependencies are, correctly, counted once only.

Here's a usertagged list of bugs I filed when I computed the values with the shell version of the script to see what was the reaction of the maintainers. Here's the list of the 20 packages with the more dependencies on sid as of ~2008-05-02.

103 texlive-full
79 brdesktop-gnome
68 rhythmbox
65 xorg-dev
60 evolution
57 epiphany-gecko
54 gnome-control-center
53 epiphany-webkit
51 screenkast
51 iceweasel-l10n-all
51 beagle
50 vlc-nox
50 python-gnome2-extras
50 brdesktop-common
50 anjuta
49 dtc-common
48 totem-gstreamer
47 python-gnome2-desktop
47 awn-applets-c-core
46 gnome-main-menu

So, it looks like we are only going up but not down :(

The FSF is celebrating the birth of the idea that is GNU. There is a video which I have mirrored here for local users. So I plan to sing "Happy Birthday to GNU" on 27th September 2008 as well. It seems like some people are not quite so happy. One correspondent wrote that it was not the 25th birthday of the "GNU Operating System" since that would mean the GNU+HURD which he feels is still not ready. In my opinion the term "GNU Operating System" represents in loose terms "a free alternative to Unix" (free as in mukta). And that idea was born 25 years ago.² I think that the HURD was just the "concept of a notion of an idea" 25 years ago. (The first article about the HURD was on the net about 1990 or so). Moreover, in comparison with Unix of 25 years ago, the GNU+HURD combination could be considered "ready". Thus, there is indeed much to celebrate. For example,

1. There is not one --- there are many free alternatives to Unix. In fact, there are perhaps no non-free Unixes visible anymore! Does this owe a lot to the resolution made by RMS 25 years ago? Absolutely. Of course, it also owes a lot to the resolve of the hackers who followed up on this and took it well beyond what he had in mind.
2. One can run the GNU tools + HURD combination to get an environment as good or better than what was available as Unix 25 years ago. Does a general member of the public really need to do this? No. There are currently a number of far superior solutions. Does the software community need this? Yes. (See below.)

I think that there are a lot of people who disagree with RMS and FSF today on a number of issues --- and these detractors may even be right. Perhaps this makes some people begrudge FSF celebrating the 25th anniversary/birthday of GNU. I offer the following points for their consideration:

1. The vision and resolve RMS and the FSF put behind the vision --- starting 25 years ago --- have got us here.
2. The FSF's astuteness in getting a rather well-known person like Stephen Fry ¹ to celebrate GNU. If any of the distros had managed it, it would have been considered a coup!
3. While Linux may be the dominant player in Unix today, the alternatives like *BSD, Solaris and even the HURD have an important role to play in the development of free software.
4. That chocolate cake looks really delicious. :-)

So here's to (at least) another 25 years for GNU!

My previous post titled AppArmor is Dead [1] has inspired a number of reactions. Some of them have been unsubstantiated opinions, well everyone has an opinion so this doesn’t mean much. I believe that opinions of experts matter more, Crispin responded to my post and made some reasonable points [2] (although I believe that he is overstating the ease of use case). I take Crispin’s response a lot more seriously than most of the responses because of his significant experience in commercial computer security work. The opinion of someone who has relevant experience in the field in question matters a lot more than the opinion of random computer users! Finally there is the issue of facts. Of the people who don’t agree with me, Crispin seems to be the first to acknowledge that Novell laying off AppArmor developers and adding SE Linux support are both bad signs for AppArmor. The fact that Red Hat and Tresys have been assigning more people to SE Linux development in the same time period that SUSE has been laying people off AppArmor development seems to be a clear indication of the way that things are goind. One thing that Crispin and I understand is the amount of work involved in maintaining a security system. You can’t just develop something and throw it to the distributions. There is ongoing work required in tracking kernel code changes, and when there is application support there is also a need to track changes to application code (and replacements of system programs). Also there is a need to add new features. Currently the most significant new feature development in SE Linux is related to X access controls - this is something that every security system for Linux needs to do (currently none of them do it). It’s a huge amount of work, but the end result will be that compromising one X client that is running on your desktop will not automatically grant access to all the other windows. The CNET article about Novell laying off the AppArmor developers [3] says ‘“An open-source AppArmor community has developed. We’ll continue to partner with this community,” though the company will continue to develop aspects of AppArmor‘ and attributes that to Novell spokesman Bruce Lowry. Currently there doesn’t seem to be an AppArmor community, the Freshmeat page for AppArmor still lists Crispin as the owner and has not been updated since 2006 [4], it also links to hosting on Novell’s site. The Wikipedia page for AppArmor also lists no upstream site other than Novell [4]. The AppArmor development list hosted by SUSE is getting less than 10 posts per month recently [6]. The AppArmor general list had a good month in January with a total of 23 messages (including SPAM) [7], but generally gets only a few messages a month. The fact that Crispin is still listed as the project leader [8] says a lot about how the project is managed at Novell! So the question is, how can AppArmor’s prospects be improved? A post on linsec.ca notes that Mandriva is using AppArmor, getting more distribution support would be good [9], but the most important thing in that regard will be contributing patches back and dedicating people to do upstream work (Red Hat does a huge amount of upstream development for SE Linux and a significant portion of my Debian work goes upstream). It seems to me that the most important thing is to have an active community. Have a primary web site (maybe hosted by Novell, maybe SourceForge or something else) that is accurate and current. Have people giving talks about AppArmor at conferences to promote it to developers. Then try to do something to get some buzz about the technology, my SE Linux Play Machines inspired a lot of interest in the SE Linux technology [10]. If something similar was done with AppArmor then it would get some interest. I’m not interested in killing AppArmor (I suspect that Crispin’s insinuations were aimed at others). If my posts on this topic inspire more work on AppArmor and Linux security in general then I’m happy. As Crispin notes the real enemy is his employer (he doesn’t quite say that - but it’s my interpretation of his post).

• [1] http://etbe.coker.com.au/2008/08/23/apparmor-is-dead/


• [2] http://blogs.msdn.com/crispincowan/archive/2008/09/02/go-ahead-make-my-day.aspx


• [3] http://news.cnet.com/8301-13580_3-9796140-39.html


• [4] http://freshmeat.net/projects/apparmor/


• [5] http://en.wikipedia.org/wiki/Apparmor


• [6] http://forge.novell.com/pipermail/apparmor-dev/


• [7] http://forge.novell.com/pipermail/apparmor-general/


• [8] http://forge.novell.com/modules/xfmod/project/memberlist.php?group_id=1864


• [9] http://linsec.ca/blog/2008/09/03/crispin-responds-to-allegations-that-apparmor-is-dying/


• [10] http://www.coker.com.au/selinux/play.html

Share This

I’m now up to five different terms for “independent variable” (X’s) in my methods lecture for today (thanks in part to a Polmeth post that just reminded me of one I’d forgotten to put on the list), and I’m probably missing some. You’d think we could narrow that down a little. The others: covariate, predictor, explanatory variable, and regressor. Funnily enough, I only came up with “dependent variable” and “regressand” for Y, but I’m surely missing some there too.

I’ve been seeing a bit about Canada recently, what with plastic-bottled water (more on that next month), the airline I used for my last trip (Zoom) going bust and stranding passengers and being a fan of Co-op Radio’s documentaries, so I was particularly interested to see this news:

“FACIL, a non-profit association, which promotes the collective appropriation of Free Software, contests the Quebec government purchasing methods for software [...]” (first seen at Lawsuit, etc - Diary of a CrazyFrench)

In England, discrimination against free software has been less obvious, with the whole national, county and district government system being officially open to free software (or rather, open source that meets gov.uk’s own definition of the term), but very biased towards large service providers. Like most countries, England doesn’t seem to have large pure-free-software service providers yet, so this usually means that the best we can do is get some mix of free and non-free software. This only gets beyond token gestures towards free software when lots of the project development and leadership is internal, as in nearby Bristol City Council, or the Department for Innovation, Universities and Skills. I’m surprised that Quebec has been overt enough to make a lawsuit possible and I’m very encouraged that FACIL is the group to do this. I’ll suggest to SPI and a few other groups that they should join to show support, but I don’t know whether the “political interventions” policy of FACIL will cause problems for non-profits.

While sitting on a bench in Hyde Park, watching the leaves fall from the trees, thinking that fall is just around the corner, I started to write on the back of an old receipt. One that says 108,- for a Banana Passion Split and a bottle of Coca Cola. It’s just something I do when I’m nervous, excited, lonely, happy or all of the above. Sometimes I spend so long thinking about how to express myself, that I forget what I was going to write. I spend too long trying to find the right adjectives. And I wish I could find the right words to describe some of the best moments. My trip to London was more than just Big Ben, The London Eye, Madame Tussauds and Oxford Street. A lot more. Another receipt says 590,- for a haircut. It’s from July 15th, 2008. My ramblings on the back, blue ink, some words crossed out, some misspelled; at least 1 of the 268 Underground stations has a staircase with 78 steps, a shop on Oxford Street sells salt from the Dead Sea, and some very big soap bubbles were seen in Camden Town. There are some keywords at the bottom of the second receipt. Just to remember all the other things that happened. Like the time we almost got lost on the Underground, the elevator with no buttons, the Kanye West sunglasses in white and pink, and the lack of a proper breakfast - even though I don’t mind white bread and strawberry jam. There was just one small thing I wasn’t too happy about; you can’t write on receipts from Tesco. They have THANK YOU FOR SHOPPING WITH US printed all over it. And it took me a while to find that piece of paper in my bag. This time I saw fall in London, and a white pigeon. I will make sure I have my notebook with me next time.

Back from Argentina, back from DebConf. As always, the ~3 weeks I spent there were really great, in as many fronts as I can imagine or describe. But I won't go into that now - For the purposes of this posting, the single thing that I got out of DebConf was looking with envy at all the people that had something that used to be called a sub-notebook some time ago, and now morphed into the more modern(?) name netbook.
Several people were seen with their tiny Asus Eee machines, of various models. And I definitively decided I want one - I was quite close to buying one in Argentina, as they are readily for sale there (surprisingly, in Mexico Asus sells motherboards, but no netbooks)... But I've always prefered waiting or paying a little premium for having an on-country seller and warranty.

Back home, Pooka told me that several stores in Mexico do sell the Acer Aspire One. After a little research, I decided to go for it. Office Depot sells the AAO for MX$4500 (around US$450). The only model they carry comes with Windows XP installed (instead of Linpus Linux), which is a shame I thoroughly repeated to the vendor - But it does come with 1GB RAM and 120GB HDD, much better for my needs than the other model, with 512MB RAM and 8GB SSD. This is, after all, a full (although very small) machine. It has an Intel Atom 270 CPU - I haven't yet measured how it fares, but it feels quite responsive so far for typical desktop tasks.
But what made me really happy about it is the Debian support. The only tricky part was to get the installer going, as it does not have a CD drive to boot from (and I didn't want to completely overwrite my only available USB stick's data). Don't try installing Debian Etch, as its kernel will not support the built-in Realtek RTL8101E network card (maybe etch-and-a-half's kernel does?). My greatest ally in this was, of course, the wiki.debian.org article on the Acer One - I rebooted with Lenny's debian-installer, and everything was smooth from that point on. Propietary firmware is required for the wireless AR5007 card and webcam, but -exactly as documented in the wiki- they are covered respectively by madwifi and linux-uvc.
I did a very regular install, with basically the default desktop and notebook setup. I continue to be amazed... Everything just works! It is not even fun, there are no funny drivers to recompile, no bang-your-head-against-the-wall... Even suspend-to-RAM. It just works.
The only glitch I found so far is that, after suspend-to-RAM, the madwifi module must be removed and reloaded to have wireless network. This is a well-known glitch that can be easily worked around. But besides that, it is... as easy as it gets. And, at such a price, and under 1Kg weight... This computer will get used to go out with me quite often! Battery life is just 2 hours, but for most situations, it's more than enough.

The current method of patching into place a series of ./configure variables as $(DEB_HOST_GNU_TYPE).cache files is becoming a pain - it blocks native builds of Emdebian packages for a start. So, I've been experimenting with a different method of support.
Centralised cache value support
Initially, I've collated all the current cache values needed (i.e. beyond the ones already provided for specific architectures) into /etc/dpkg-cross/cross-config.cache and have a script in emdebian-tools called emcache that will manage the central values by comparing them to the real values used in each build tree.
So, cache values, what are they?

1. Cross-builds fail if ./configure tries to compile a test program and then try to execute it because it will have been compiled with the cross-compiler - instead, the cross-build passes the value that the test was trying to calculate so that ./configure does not need to try to execute any cross-compiled code.


2. The values are derived from native builds, written to a $(DEB_HOST_GNU_TYPE).cache file that is passed to ./configure via debian/rules.


3. The values turn out to be mostly (completely?) architecture-independent, so are eminently suitable for collation into a central cache file.


4. Quite package-specific. There is some overlap but most are customised for a specific package. This led to a development of the central cache file: package-maintained cache files to support extensibility.

Package maintained cache files
I'm now experimenting with cache files written (and maintained) by packages during normal builds and dropped into /etc/dpkg-cross/cross-config.d/ - which would later migrate into dpkg support. Ideas welcome!
The basic idea will be that IF cross-building, the build process would check for /etc/dpkg[-cross]/cross-config.d/$source_package and, if found, add that to the cache value list for that build.
Cache files are shell scripts so this it be manageable to bring some intelligence to the process.
Where does MIPS come in?
Well, all the builds, so far, have been ARM or armel (where the cache values are likely to be identical) and i386 (which doesn't need a cross-compiler at all and therefore does not need cache files). MIPS is the first chance to check the cache values and ensure that the above criteria really are true.
Now that we have a volunteer looking at MIPS, the current MIPS builds are a suitable test bed.
What does this have to do with Debian?
Once Lenny is released, I'll be looking to implement this support via the long term mass bug filing for cross-build support into the packages themselves. It is possible that the process can be entirely automated - as long as the script knows which values are relevant. A normal cache file, post-build, can be very, very long whereas most cache files for Emdebian are just a few lines long - the longest is about 20 lines.
The current process is outlined on the Emdebian website, including a list of packages that currently need cache files to cross-build.
Oh, and just a note - when I start this phase of the long term mass bug filing, I'll first do delayed NMUs for the existing bug reports - hence the delay until Lenny is released.
In order to speed things along, I have also been doing some RC bugs - downgrading some, fixing others etc. (I've finished 6 since coming back from DebConf, so far, IIRC.) All efforts to get Lenny released will help Emdebian too, so if you want to help Emdebian, then help Lenny get out of the door and let's get to work on Squeeze.

(Squeeze is a great name for a release with lots of fixes for embedded stuff, don't you think?)

Among other things, my webmaster cooperative is an agent for the Phone Co-op. Particularly with Free Software, cost-efficient ways of downloading data is a good companion service to have and the Phone Co-op has given a good ADSL and calls service to my own home and office for a few years now. When there are problems, they’ve worked pretty well to fix them. Just last week, I had a phone call to tell me when routing problems in LINX were fixed… which was a great help. I don’t remember an ISP doing that for me for years. I think the last one was my university’s dial-up service, 10 or more years ago. Another unusual thing for a UK ISP is the excellent public position on the Phorm ad-snooper that I mentioned previously: We strongly oppose these practices, especially the loss of privacy [...] Phorm is not operating on our users’ lines and we have required and received assurances from our suppliers that this will remain the case. Our Chief Executive Vivian Woodell gave an interview to the FT (accept the first two cookies, then block the rest, else you only get two paragraphs) which has a pretty good explanation of how the business side of the cooperative works. We’re recruiting for Line Rental (PSTN and ISDN) until the end of the year, including special offers like residential Unlimited Anytime Calls for 7.95 a month (normally 9.99) at the moment, as well as the great climate-neutral and ethical policies - if your landline contract renewal comes up, give it a look. The Phone Co-op was ten years old in June, but there are still big challenges: I’d love to see a cooperative pay-as-you-go mobile phone service and the mobile contract offers still look a bit expensive to me (no bundled data on phones, 25 per month for a 3G data card). I’d also love to see support for some daring innovative solutions like customers owning their own tails (tails are the last link to their house, where most of the UK has a choice of one provider: BT), using that as a way to upgrade small towns and villages to fibre-optic, or offer resident-owned wifi. Update: As if by magic, Ofcom has started a social web consultation about mobile phone services. Go comment.

Michael Janke is writing a series of posts about estimating availability of systems, here is a link to the introduction [1]. He covers lots of things that people often miss (such as cooling). If you aren’t about to implement a system for reliability then it’s an interesting read. If you are about to implement a system where reliability is required and you have control of the system (not paying someone else to run it and hope for the best) then it’s an essential read. It will probably also be good to give this URL to managers who make decisions about such things. Interesting summary of the connections between the Iraq war and the oil industry in the Reid Report [2]. The suggestion made by one of the sources she cites is that the intention of the war was to reduce the supply of Iraqi oil to increase prices. Sam Varghese has written an essay about this which summarises where the Iraqi oil goes [3]. It seems that half of Iraq’s oil goes to US military use, the other half is used domestically, and some oil is imported as well! So because of the US occupation the country with the second largest known oil reserves is importing petroleum products! If the US military was to cease operations world-wide then the oil price would drop significantly, this doesn’t just mean the occupation of Iraq and the various actions in South America, but also the bases in Germany and Japan. Interesting paper by Alexander Sotirov and Mark Dowd about Bypassing Browser Memory Protection in Windows [4]. This paper is good for people who are interested in computer security but don’t generally use Windows (such as me), if you want to learn about the latest things happening in Windows land then this is a good place to start. A well researched article by Rick Moen about the unintended effects of anti-gay-marriage laws [5]. Maybe some of the “conservatives” who advocate such laws should get themselves and their spouses tested. It would be amusing if someone like Rush Limbaugh turned out to be involved in a “gay marriage”. What Sysadmins should know about exposure to hazardous materials [6]. High-level overview of the issues, probably a good start for some google searches to get the details. Diamond John McCain is an interesting blog about the 73 year old (who was born in Panama) candidate in the US presidential election [7]. Update: Corrected my statement about Iraq’s oil reserves based on a comment by Sam.

• [1] http://lastinfirstout.blogspot.com/2008/07/estimating-availability-of-simple_04.html


• [2] http://blog.reidreport.com/2008/08/war-for-iraqi-oil.html


• [3] http://gnubies.com/comment/oil.html


• [4] http://taossa.com/archive/bh08sotirovdowd.pdf


• [5] http://linuxmafia.com/faq/Essays/marriage.html


• [6] http://blogs.techrepublic.com.com/networking/?p=637


• [7] http://diamondjohnmccain.blogspot.com/

Share This

Sorry, ever since Debconf I've been busy, ill and otherwise distracted. I'm still catching up... Stuff that's happened: Dude, you're married! For the second year running, I had to leave Debconf a day early to attend a wedding party. My good friend Matthew married Oxana last Monday. I've just put my photos of the happy event online today. After a successful Debconf where I managed to evade the normal plague, it hit hard on the Tuesday morning and I've been struggling with it ever since. Just about over it now, but I've been feeling crap for over a week. On the way to Buenos Aires to catch my flight back to England, I received a text message telling me the sad news that my uncle, John Bolton had passed away. This means that the eldest two of my mum's three brothers have died this year, in the space of just a few months, and it's been a very hard time for the family. The funeral happened earlier this week, so I was away and mostly off-line for a couple of days travelling there and back. I'm trying to get back up to speed on a whole host of different things at the moment; if you're waiting on a response from me about Debian-related stuff, please feel free to prod me via email or IRC.

Currently, I'm together with Marc Brockschmidt evaluating which virtualization to use on our new server. We want that our virtual systems feel like real systems, and we want an open source solution. So, vserver and the like is out of the game, as well as VMware. The two remaining solutions we looked at are Xen and kvm. Xen has of course the advantage of the matured. Also, we have experience with running xen servers - and with the issues that can happen, like the chances to disconnect dom0 from xend, and then reboot the server the hard way. However, the most serious disadvantage is that development has practically stalled with the 2.6.18-kernel. Of course, even of today one could install a new server based on Etch, but that doesn't really feel right. There is some development ongoing to run domUs with newer kernels (like in Lenny), but there isn't currently any new kernel available for dom0. kvm is a more recent addition to the virtualization camp, and is basically "qemu on steroids". All looks rather promising, development happens with the recent kernels. However, kvm lacks a few features of e.g. Xen. This includes the ability to reboot dom0 (and the hardware) and just let the domUs survive. Or to have a nice management script where one could just say "xm shutdown $domU", and have basically the power button be pressed on the virtual machine. Or to just attach and detach to the virtual console whenever one wants. Nothing of that is impossible with kvm, one could attach the command-terminal to some pipe, and the linux console to some other, and attach and distach via own scripts. But - all of that should be expected to be available from some solution that calls itself enterprise ready. (And - writing own scripts has always the possibility to make own mistakes.) However, among all the worst possible issue is that kvm is underdocumented (or rather: There are lots of different places where some parts of the documentation is hidden - including the great remark in the man page "The other options are similar to those of qemu."). So, what to do? Invest more time into a solution that seems like a dead end. Or put up with the incompletness of another solution?

Reportedly, the USA is falling behind the rest of the world in bandwidth:

The 2008 median real-time download speed in the U.S. is a mere 2.3 megabits per second. This represents a gain of only 0.4 mbps over last year s median download speed. It compares to an average download speed in Japan of 63 mbps, the survey reveals. US also trails South Korea at 49 mbps, Finland at 21 mbps, France at 17 mbps, and Canada at 7.6 mbps, and the median upload speed was just 435 kilobits per second (kbps), far too slow for patient monitoring or to transmit large files such as medical records.

But don’t tell Chris Blizzard’s commenters. He writes about Comcast’s annoucement of a 250GB/month bandwidth cap, and gets an earful from commenters from Canada and Europe:

A boo hoo hoo. Major Canadian ISPs have had a limit of 60 GB for months, if not years. Oh wait probably the same way as most of the world manages on 10-20GB, for far more money than you re paying for $250. Not a lot of sympathy from this corner Yep, no sympathy from here either in Australia, with the only _independant_ ISP left, $280 AUD gets you 100GB. $50 with a major telco (the rest of the ISPs here) gets you 5GB. eg with my current ISP, a 8 MB line with a 300 GB monthly cap costs 20 GBP/month. A 8 MB line with unlimited bandwidth costs 160 GBP/month. Quite a difference! I pay the equivalent of $40 a month for 30GB, and extra GB on top are $3 each. That s with Plus Net (http://www.plus.net). I m in South Africa paying about $130 for a 10GB cap.

So who’s really better off? By my calculations, if a Canadian ISP provides 7.8 mb/s with a 60 GB cap, that’s about 17.5 hours per month of sustained maximum bandwidth before you’ve blown your limit. By contrast, an American ISP with 2.3 mb/s and a 250 GB cap gives you about 247 hours per month of sustained maximum bandwidth. Perhaps part of the answer is that only one country–Canada–shows up in the list of “faster countries” and in the comments section of Chris’s post. That could explain the apparent disconnect; maybe Great Britain and Australia are worse off than the USA, while Finland and Japan are better off. Still, this does bring the question to mind: which is better, raw speed, or the ability to actually use it without fear?

I just read an interesting post about latency and how it affects web sites [1]. The post has some good ideas but unfortunately mixed information on some esoteric technologies such as infiniband that are not generally applicable with material that is of wide use (such as ping times). The post starts by describing the latency requirements of Amazon and stock broking companies. It’s obvious that stock brokers have a great desire to reduce latency, it’s also not surprising that Google and Amazon analyse the statistics of their operations and make changes to increase their results by a few percent. But it seems to be a widely held belief that personal web sites are exempt from such requirements. The purpose of creating content on a web site is to have people read it, if you can get an increase in traffic of a few percent by having a faster site and if those readers refer others then it seems likely to have the potential to significantly improve the result. Note that an increase in readership through a better experience is likely to be exponential, and an exponential increase of a few percent a year will eventually add up (an increase of 4% a year will double the traffic in 18 years). I have been considering hosting my blog somewhere else for a while. My blog is currently doing about 3G of traffic a month which averages out to just over 1KB/s, peaks will of course be a lot greater than that and the 512Kb/s of the Internet connection would probably be a limit even if it wasn’t for the other sites onn the same link. The link in question is being used for serving about 8G of web data per month and there is some mail server use which also takes bandwidth. So performance is often unpleasantly slow. For a small site such as mine the most relevant issues seem to be based around available bandwidth, swap space use (or the lack therof), disk IO (for when things don’t fit in cache) and available CPU power exceeding the requirements. For hosting in Australia (as I do right now) bandwidth is a problem. Internet connectivity is not cheap in any way and bandwidth is always limited. Also the latency of connections from Australia to other parts of the world often is not as good as desired (especially if using cheap hosting as I currently do). According to Webalizer only 3.14% of the people who access my blog are from Australia, they will get better access to my site if hosted in Australia, and maybe the 0.15% of people who access my blog from New Zealand will also benefit from the locality of sites hosted in Australia. But the 37% of readers who are described as “US Commercial” (presumably .com) and the 6% described as “United States” (presumably .us) will benefit from US hosting, as will most of the 30% who are described as “Network” (.net I guess). For getting good network bandwidth it seems that the best option is to choose what seems to be the best ISP in the US that I can afford, where determining what is “best” is largely based on rumour. One of the comments on my post about virtual servers and swap space [2] suggested just not using swap and referenced the Amazon EC2 (Elastic Computing) cloud service and the Gandi.net hosting (which is in limited beta and not generally available). The Amazon EC2 clound service [3] has a minimum offering of 1.7G of RAM, 1EC2 Compute Unit (equivalent to a 1.0-1.2GHz 2007 Opteron or 2007 Xeon processor), 160G of “instance storage” (local disk for an instance) running 32bit software. Currently my server is using 12% of a Celeron 2.4GHz CPU on average (which includes a mail server with lots of anti-spam measures, Venus, and other things). Running just the web sites on 1EC2 Compute Unit should use significantly less than 25% of a 1.0GHz Opteron. I’m currently using 400M of RAM for my DomU (although the MySQL server is in a different DomU). 1.7G of RAM for my web sites is heaps even when including a MySQL server. Currently a MySQL dump of my blog is just under 10M of data, with 1.7G of RAM the database should stay entirely in RAM which will avoid the disk IO issues. I could probably use about 1/3 of that much RAM and still not swap. The cost of EC2 is $US0.10 per hour of uptime (for a small server), so that’s $US74.40 per month. The cost for data transfer is 17 cents a GIG for sending and 10 cents a gig for receiving (bulk discounts are available for multiple terabytes per month). I am not going to pay $74 per month to host my blog. But sharing that cost with other people might be a viable option. An EC2 instance provides up to 5 “Elastic IP addresses” (public addresses that can be mapped to instances) which are free when they are being used (there is a cost of one cent per hour for unused addresses - not a problem for me as I want 24*7 uptime). So it should be relatively easy to divide the costs of an EC2 instance among five people by accounting for data transfer per IP address. Hosting five web sites that use the same software (MySQL and Apache for example) should reduce memory use and allow more effective caching. A small server on EC2 costs about five times more than one of the cheap DomU systems that I have previously investigated [4] but provides ten times the RAM. While the RAM is impressive, I have to wonder about CPU scheduling and disk IO performance. I guess I can avoid disk IO on the critical paths by relying on caching and not doing synchronous writes to log files. That just leaves CPU scheduling as a potential area where it could fall down. Here is an interesting post describing how to use EC2 [5]. Another thing to consider is changing blog software. I currently use Wordpress which is more CPU intensive than some other options (due to being written in PHP), is slightly memory hungry (PHP and MySQL), and doesn’t have the best security history. It seems that an ideal blog design would use a language such as Java or PHP for comments and use static pages for the main article (with the comments in a frame or loaded by JavaScript). Then the main article would load quickly and comments (which probably aren’t read by most users) would get loaded later.

• [1] http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it


• [2] http://etbe.coker.com.au/2008/08/27/killing-servers-virtualisation-and-swap/


• [3] http://www.amazon.com/b/?node=201590011


• [4] http://etbe.coker.com.au/2008/05/28/xen-hosting/


• [5] http://www.protocolostomy.com/2008/08/27/more-adventures-in-amazon-ec2-and-ebs/

Share This

As I start this blog post, I'm on the train between Amersfoort and Osnabr ck, on my way to Kiel, where there'll be a meeting of the m68k porters, kindly organized by m68k kernel maintainer Christian Steigies. I took the 8:40 train in Mechelen this morning, and if all goes well, I'll be arriving in Kiel at 17:22 tonight. Some train trip... but I certainly prefer that to any flight. Anyway. The days of Debconf were very nice. Debconf is a lot of things to me. Travel. Hacking. Mao, evolving into a drinking game. Beer. Attending talks. Giving talks. Meeting people. Face-to-face non-flaming. Gesturing for an extra knife at luunch. Hangover. Whiskey. Talking to the person next to you—over IRC. Getting killed before knowing what the rules of the assassins game are. Filing bugreports in person. Pictures. Kilts. Streaming video. Not being sick, hopefully. Name badges. Flying. Sleeping. Yes, sleeping. Most of all, though, this year, Debconf was just great. Thanks. You know who you are. After debconf, I flew to Buenos Aires, where I slept for one night in a Youth Hostel somewhere downtown. They were affordable, but the bed wasn't great—the mattress sorely needed replacement. Since I had a day in BA, and since there were apparently not enough people on the schedule, I'd agreed to hold a talk, and came up with the idea of a 'debian secrets' talk—about Debian-specific commands, such as dpkg-divert and update-alternatives etc—so that people could learn how to use their Debian system more efficiently. As I was sitting in Andreas' talk, who was right before me on the schedule, suddenly Dag walked up to me and said hi. This was unexpected; Dag is a fellow Belgian, who's involved with the CentOS project, and who maintains a positively huge RPM repository at his site—if you maintain an RPM-based system somewhere, you'll probably know about that site. So while I recognized him, I immediately wondered what he was doing there. Turned out he was invited to hold a talk at the Free Software event to which Debianday was attached (and which would not start for another day or two), and that when he saw my name on the schedule, he decided to attend. Fun. He learned a thing or two from my talk, and was talking later about writing some tools for RPM-based systems that would perform the same or similar functionality implemented by some of the Debian tools I talked about. Great. We had a picture together (in front of the DebConf banner—hah) that I still should upload, and then went out for lunch together with some other CentOS guys. The talk itself seems to have been well-received, and I'm glad about that; I only gave it because there was a need for more talks, not because I felt confident I was very knowledgeable about that subject. In fact, I did have to ask on the debconf-discuss mailinglist for some input (which I received) so that I could make sure the talk would actually be useful to people. That helped a lot, I guess. After that, we played some mao in the lobby of the DebianDay venue. While doing so, I overheard Dag talking about CentOS, advocating it to some of the people there, which I must say felt pretty weird on a Debian event. Not that there's anything wrong with it, of course. Except that Dag was supposed to be writing his talk slides. How did that work out, Dag? ;-) Eventually, I got in a cab to the EZE airport, and flew home. That wasn't fully without issue, but I did get there. And now, I've been, eh, overloaded. Still have to follow up on a question a customer asked me, but I've barely been home or at the office, just enough to sleep really. Should do something about that, I guess... Two days ago, I also managed to forget my camera somewhere. For 12 hours, I was worried, even though I had a pretty good idea of where it was; but since I'm getting quite good at taking pictures now, and since I really like doing so as well, I really didn't want to lose my camera. Fortunately, when I called this morning to the place where I thought I'd lost it, they told me it'd been found, and where to get it. In other words, it's safe, it's taken care of, Philip went to get it, and I just need to arrange to get it when I get back. Which is a relief. Let's not be so stupid anymore in the future... Finishing up now, a day later, and I'm at Christian's office, playing with his and my coldfire board. As it happens, it appears that Freescale has brought out a new BSP for these boards, so that's nice. Let's see whether we can get those to boot again...

Some interesting events that I’ve heard about recently, with a bit of a focus on software, cooperatives and the South-West:-

• Bristol Knowledge Unconference: Friday 5th September 2008 *
• Tour of Britain, Burnham-on-Sea, Somerset, Tuesday 9th September - other UK locations 7-14 September, itv4 7pm highlights daily *
• NoseRub DevCamp, Cologne, 13.-14. September 2008
• BathCamp, Bath, Somerset, 13-14th September
• Bristol Anarchist Bookfair, St Werburgh’s Centre, Bristol, 13-14th September
• Software Freedom Day, Global, 20 September 2008
• Scripting Enabled, London, 19th and 20th September
• Listening to the Social Entrepreneur, London Docklands, 9 Oct 2008 *
• LinuxWorld, London Olympia, 23-25 Oct 2008 *
• enterprise week, UK, part of Global Entrepreneurship Week, 17-23 November
• Cooperatives-SW/RISE Trade Fair, Bristol, 1st December *

I’m pretty sure I’ll go to the *d ones - which others do you think I should go to? Will you be at any of these? Any comments to make on/to any of them? Any more to add to the list?