Interestingly, the just finished Extremadura QA + ftpmaster + i18n meeting also turned into a quite strong Bug "Smashing" party. Several participants were digging around the list of RC bugs affecting both lenny and unstable. Most often, it turned out to many bug downgrades (for many of which I'll take the blame). It's "funny" to see how sometimes our bug reporters tend to inflate their reports, most often because they feel that "bug should absolutely be fixed", but completely forgetting about the definition of 'serious' or 'grave'. I also feel that, in some way, several maintainers begin to be quite demotivated by the length of the process (which I personnally consider as being caused by several factors: decreasing (wo)manpower, haircutting about freeness of every single byte on Debian CD's or even shyness wrt correct actions needed to deal with bugs). And several of these bugs just needed a very simple action by the maintainer. And, therefore, just because we were about 15 people with a lot of time to devote to Debian for 3 days, we ended up digging in that pile of mud....removing most of the dust and sometimes very debatable RC bug reports, and finally having that count of bugs affecting both lenny and unstable go down again. This comforts me in the feeling that an aggressive approach is a good key to bug squashing. After all, if I'm sometimes too aggressive and inappropriately downgrade an RC bug, I'm sure that someone will correct me (and, anyway, I'm not that mad: sometimes I seek for advices). Ah, and "aggressive" also sometimes mean proposing to tag some bugs "lenny-ignore" (and, that needs to be very convincing with our release managers, believe me). So, what about you? Have you already smashed an RC bug today? If not, do it. Kill those nasty beasts, they deserve it. No mercy.

I’m getting increasingly annoyed by the state of some aspects of hardware support in Linux. My laptop (old Dell Latitude D610, Intel-based with ATI graphics) used to suspend/resume correctly with Linux 2.6.24 (ie, > 95% success rate). Later changes made the success rate drop significantly, and added a problem with kacpid taking 100% CPU because of an interrupt storm. And now, with 2.6.28-rc6, it’s completely broken. (partially documented in bug 11563, but I admit I gave up on this bug, because I’m going to change my laptop soon). My desktop used to wake-on-lan correctly with 2.6.24 (but required some hacks, because it wouldn’t wake up if the NIC was DOWNed before shutdown), but changes in the r8169 driver broke it. (documented in bug 9512). As a result, I’m forced to run old kernel versions on the two systems I have at home. I can understand that those issues aren’t considered high priority (not everybody use WoL), but the fact that in both cases, they are regressions, worries me a bit. How many things are routinely broken during each kernel release cycle? Hardware support is difficult, of course, but are we really doing everything we could to make it suck less? Some things I really would like to see:

• Distro packages for development kernel versions. For Debian, there’s this repository, but it doesn’t always contain the latest kernel versions. Maybe that’s something that should be moved to a kernel.org umbrella and generalized to all distributions, to provide beta-testers with easy-to-install packages. git bisect isn’t that user-friendly.
• Funding for driver developers to buy specific hardware. Many drivers cover a wide range of chips/cards, and developers often only have on a small subset of them, making it difficult to debug issues specific to one chip.
• Better/updated bugzilla on kernel.org. Many bug logs are totally confusing, and cover different issues. They could maybe benefit from new or specifically developed bugzilla features (and more bug triagers, of course).

Hacking the PTS using SOAP I've finally found some time, thanks to the Extremadura QA + ftpmaster + i18n meeting, to release the first draft of the SOAP interface to the PTS. You probably already got the idea, which is quite simple after all. The PTS, as always, gathers information about (source) packages from various sources and melds them together into web pages. With a SOAP interface you just gain the ability of accessing such information from your programs via SOAP. A proof of concept is overdue:

    $ cat ./test.py 
    #!/usr/bin/python
    import SOAPpy
    url = 'http://packages.qa.debian.org/cgi-bin/soap-alpha.cgi'
    ws = SOAPpy.SOAPProxy(url)
    print ws.versions(source="ocaml")['unstable']
    print ws.uploaders(source="ocaml")[1]['name']
    $ ./test.py 
    3.10.2-3
    Stefano Zacchiroli

Everything is still in alpha version, but already working. Some links which you might find useful:

• (fancy) API reference (blame epydoc if you think it's too fancy)
• general information about the SOAP interface (from wiki.d.o), including pointers to other doc / tools

Please let me know if / how you are using of the SOAP interface, it will help for future developments. How it works Just a few comments on how it works. You might remember that a while ago I've made all PTS pages XHTML-valid. Well, on top of it I've implemented something along the lines microformats, that just make a clever use of ingredients already available in XHTML like classes and unique identifiers. Having that, a "reshuffling" of the information already available on the web pages (which are now kinda "semantically" tagged) can be obtained by evaluating a handful of XPaths on the (not anymore) final XHTML pages. That's precisely what the CGI implementing the SOAP API is currently doing. This way one can avoid implementing two different access paths to the information collected by the PTS: one for rendering, and one for SOAP (no, reusing the rendering one for SOAP was not an option, given that it was originally written in XSLT). The only annoyance I've encountered is that XPath is completely unaware of the "CSS-like" semantics of XHTML classes, which states that classes are space-separated list of class names, to be interpreted as sets. That means that to check whether an element belongs to a given class you need to fiddle with substring matches on the class attribute (which is quite crappy).

Let's start with a caution: this is a work-in-progress and is aimed at Squeeze (for the tools) and Squeeze+1 for the actual TDebs. Nothing about TDebs will happen within Debian any time soon. DebConf9 could be an important step in the work - as long as we've released Lenny by then.
Various steps and decisions have been discussed and tested during the meeting, to the point where we now have an early draft specification for tdebs.
See also details of the ftpmaster meeting and progress on i18n and tdebs.
The draft spec includes links to the current resources, including links to git repositories for the changes being tested for dpkg and debhelper.
These changes will supersede the existing experimental scripts in the emdebian-tdeb package.
The spec will continue to be updated during the rest of this meeting and I'll continue working on the dpkg and debhelper support.
(A trivial patch for CDBS support is already done, dependent on the debhelper support.)
Next on my ToDo list is Emdebian Grip - including a few tweaks to em_installtdeb.

Yay, long time since the last ftpmaster blog post. Time to write a new one. It’s not that I haven’t done anything since my last post, it just wasn’t only ftpmaster related. Right now I’m somewhere in Extremadura, Spain, at the Ftpmaster / I18N / QA Meeting. This meeting is near to the location of DebConf9, so yesterday we visited that venue. Still have to upload the pictures I took. It’s pretty nice, but I fear we might have a problem with wheelchairs. While the venue is very well prepared for wheelchair users, like having ramps or electric elevator platforms for every little staircase, I see one big problem in the size of the elevator going from the ground floor up to the first level. We don’t have exact measurements currently, just our rough guess from looking at it, but this is something we definitely have to look in. Something similar applies to the sleeping place. They do have a room for disabled people, but we couldn’t look into it when we where there, as it was in use (And it is pretty unpolite to just move into an occupied room). So we need to get an exact floor plan and hopefully also pictures for it later on. Luckily, the sleeping place is a much smaller issue, if it doesn’t fit we could think about placing disabled people in a Hotel. More ftpmaster related we had a lot of discussions about one of our main agenda points for this meeting, tdebs. The more we discuss it, the more we all hate it. Its a topic that touches nearly everything in Debian and will redefine quite a lot of places/actions/behaviours, so it needs a lot of thought. Which is what we start here. We wont be able to get anything substantly implemented here, but I think we can get to a common understanding of a specification and a Todo. After all the schedule currently has “lenny + 2” for this, its not a light weight step. But thats not all, would be boring if we would have only one topic. Us ftpteam also intends to/does work on other issues, like a redesign of dak and the database below it. And today we issued a statement about the AGPL. Not ftpmaster related, but also a nice work done here in Extremadura, is the setup for mirrors of packages.debian.org. This enables us to mirror the part of packages.debian.org to other .debian.org hosts thats responsible for the web view, and as such allows us to have redundancy in case of errors, load balancing and all those nice things. Comments: 2

What started with a little host naming theme riddle, and spread as a meme across various planets, has come full circle today, as I installed a new machine, khyber.madduck.net in the racks hosted at the LayerOne colocation site in Z rich. Init Seven, my favourite Swiss ISP, are sponsoring the rack space and connectivity: a routed /28 IPv4 subnet and a /64 IPv6 subnet. The machine is a Transtec Calleo 331 1U server with two quad-core AMD Opteron 2354 processors, 16Gb of RAM, and 4Tb of disk space, configured into a 2Tb RAID1 with LVM on top. It will be used to host a couple of KVM instances for a number of projects I currently operate off my main, personal server, pulse.madduck.net. Unfortunately, Transtec (as well as most other vendors I checked out) uses nVidia s MCP55Pro chipset, which had me sceptical long before I sent the order. nVidia is known to suck at thinking open-source, and the Linux kernel driver for their network chips, forcedeth, had to be reverse-engineered, because nVidia failed to cooperate. Apart from that, nVidia s chips are also often simply crap. Many of their motherboards come with broken APICs and the network chips often wreak havoc on subnets. I find it curious that a search for MCP55 or MCP55Pro on the nVidia website yields no results. Given these expectations, I insisted on a try-and-buy contract, which are not default in Switzerland (as opposed to the EU). My fears proved right, it wasn t long until I encountered the first problems with nVidia s crap: the network chip is unable to deal with segmentation offloading for larger packets, e.g. those with IPv6 payload in a bridging context, and generic segmentation offloading has to be turned off on the interface:

ethtool -K eth0 gso off

This fixes the issue, and so I stuffed it into /etc/network/interfaces and decided to keep the machine, especially after having invested the time into the server to figure out how to work the watchdog and the IPMI card. The folks behind Transtec s support hotline are helpful, but it took me a bit of time to get the information I wanted out of them. At least they deemed it perfectly natural that I wanted to run Debian, and never questioned that. Let s document some of the more internal specs here for posterity: The BIOS gives the impression of two separate watchdogs, one on the IPMI card (a SIMSO+) and one on the mainboard itself (a SuperMicro H8DMU+). Transtec told me that SuperMicro disabled the IPMI watchdog (called BMC watchdog in the BIOS) with firmware 1.29, leaving only the one on the motherboard. It took a bit of time to find out that this was a W83627THF and supported by the w83627hf_wdt driver, which I configured with module options timeout=120 nowayout=1. I left the BIOS setting at disabled since that would otherwise interfere with booting e.g. the awesome grml live system by rebooting the machine after the set amount of time. It also took a bit of time to get SOL working with the IPMI card. In the end, I found it to work best by configuring the console redirection in the BIOS to use COM2 with 38400/n/8, using hardware flow control, redirection-after-POST always, and vt100 emulation ANSI and vt-utf8 emulation were horrific. I instructed the bootloader to pass console=tty0 console=ttyS1,38400n8 to the kernel (order matters), and configured agetty in /etc/inittab like this:

T0:2345:respawn:/sbin/getty -8hL ttyS1 38400n8 vt100

To guard against the worst, I also left a grml CD in the drive, to be able to boot a rescue system via SOL in case everything else was broken. The following kernel boot parameters enable me to use the rescue system via IPMI:

vga=normal noeject noprompt console=tty0 console=ttyS1,38400n8 forensic

Michael Prokop of the grml project tried to create a customised ISO image with these options for me, but I kept getting medium errors trying to read it, independent of the burner, CD-writing-software, operating system used to burn, medium, and drive used to read, so it s possible that the ISO image is faulty. Michael used the following command line to create it (supporting files are here):

grml-live -t ~/templates-madduck -c GRMLBASE,GRML_MEDIUM,AMD64 -o \
  /grml-live/grml64-medium_2008.11-madduck -a amd64 -s sid -v \
  2008.11-madduck -g grml64-medium -r Schluchtenscheisser -V -b

Until we figure out the problem with this, the grml64 0.2 CD in the drive will do. The machine is now operational, but the IPv6 prefix I have still arrives at vera.madduck.net, a Xen-Instance hosted by Init Seven, which I will no longer need, once I find the time to migrate all the services it hosts. NP: This Will Destroy You: Young Mountain

Cooperatives-SW is working in partnership with the Regional Infrastructure for Social Enterprise (RISE) to hold a Trade Fair with support from CDA (BRAVE Ltd.), Social Enterprise Works, Co-opportunity and Watershed Media Centre. It will be held on Monday 1 December at the Watershed in Bristol. TTLLP will be there and I should post lots of news about it on Tuesday. I’ll try to microblog the event, but that depends how busy it is. Thanks to the VAT rate cut on the same day, I’ll be the only one there, leaving workers in the office just in case any of our customers have problems with their OSCommerce or Drupal online shops. (It seems there shouldn’t be any, but we hope for the best and prepare for the worst.) If you’d like to meet up on Monday, send me a comment however you prefer. Update: If you want to attend the event, registration details are on this page - it looks like registration is required, but I’ll be disappointed if they turn people away from the exhibition space between 1 and 2pm if there’s room.

It's interesting to be here in Casar de Caceres for the joined i18n+QA+FTPmasters Extremadura meeting. Today, I've been sitting all day long in the biblioteca, which is used as hacklab, and just watched all great people around playing around with the concept of "tdebs". That weird beast can be described in two words as a concept of deb packages that would be able to distribute translation data in a more independent way then "regular" deb packages. Don't smash me as the concept is currently being twisted around in all directions by the nice folks here, to bring it to a great concept well suited for "side" material of our packages. To get an idea, you can read the original concept that popped up during the 2006 i18n Extremadura meeting, but got transformed drastically by ideas exchanged at Debconf 8 All this is made possible because the concept is now a very important issue for Emdebian and is strongly pushed by Neil Williams, whose nickname shall now be "MisterT", by general agreement of the folks here. What's really interesting right now is to see live discussions happening all the way round and see the concept slowly emerging from this. Something we could probably never do in regular mailing lists discussions or on IRC. We should really grant tons of rewards to the Extremadura region, and particularly out tireless C sar who again organised that meeting. It's still hard to say what will emerge from this and we willprobably need some more such live meetings to have real implementations of these tdeb beasts, but this is really how I like to see free software development going on. And now people begin to think about coding stuff and prepare patches, which is a good sign, in general.

Some time ago Bill Joy (who is famous among other things for being a co-founder of Sun) [1] wrote an article for Wired magazine titled “Why the future doesn’t need us” [2]. He wrote many sensible things but unfortunately focussed on the negative issues and didn’t receive a good response. On reading it today I thought more highly of the article than I did in 2000 when it was printed, largely due to having done some background research on the topic. I’ve recently been reading Accelerating Future [3] which has a more positive approach. Now a talk by Bill Joy from 2006 has been published on the TED.com web site [4]. He starts by talking about “super empowered individuals” re-creating the 1918 flu. He also claims that “more technology super-empowers people more“. Such claims seem to be excessively hyped, I would be interested to see comments from someone who has a good knowledge of current bio-technology as to the merits of those claims. He talks briefly about politics and has some good points such as “the bargain that gives us civilisation is the bargain to not use power” and “we can’t give up the rule of law to fight an asymmetric threat - which is what we seem to be doing“. He mentions Moore’s law and suggests that a computer costing $1000 then (2006) might cost $10 in 2020. He seems to be forgetting the cost of the keyboard and other mechanical parts. I can imagine a high-end CPU which cost about $800 a couple of years ago being replaced by a $2 CPU in 2020, but I don’t expect a decent keyboard to be that cheap any time before we get fully automated nano-factories (which is an entirely separate issue). Even the PSU (which has a significant amount of government regulations for safety reasons) will have a floor cost that is a good portion of $10. Incidentally the keyboard on my EeePC 701 sucks badly, I guess I’m spoiled by the series of Thinkpad keyboards that I keep getting under warranty (which would cost me a moderate amount of money if I had to pay every time I wore one out). I will make a specific prediction, that by 2015 one of the better keyboards will comprise a significant portion of the entire cost of a computer system (more than a low end computer unit) - such that in some reasonable configurations the keyboard will be the most expensive part. It would be good if PCs could be designed to use external PSUs (such as the Compaq Evo models that took laptop PSUs). Then the PSU, keyboard, and monitor would be optional extras thus giving a small base price. Given that well built PSUs and keyboards tend not to wear out as fast as people want to replace computers, it seems that financial savings could be provided to most customers by allowing them to purchase the computer unit without the extra parts. People like me who type enough to regularly wear out keyboards and who keep using computers for more than 5 years because they still work are in a small minority and would of course be able to buy the same bundle of computer, PSU, and keyboard that new users would get. In 2020 a device the size of an iPaQ H39xx with USB (for keyboard and mouse) and the new DisplayPort [5] digital display interface (that is used in recent Lenovo laptops [6]) would make a great PDA/desktop. You could dock a PDA the way some people dock laptops now and carry all your data around with you. Bill cites an example of running the Mac interface on an Apple ][ (does anyone know of a reference for this) as an example of older hardware being more effective with newer software. It's a pity that often new software needs such powerful hardware. EG it's only recently that hardware developments have overtaken the developments of OpenOffice to make it deliver decent performance. He has an interesting idea of using insurance companies to replace government regulation of food and drugs. The general concept is that if you can convince an insurance company that your new drug is not a great risk to them so that they charge a premium you can afford then you could sell it without any further testing. Normally I'm in favor of government regulation of such things, but given the abject failures of the US government this idea has some merit. Of course there's nothing stopping insurance companies from just taking a chance and running up debts that they could never hope to repay (in a similar manner to so many banks). Finally I think it's interesting to note the camera work that the TED people used. My experience of being in the audience for many lectures (including one of Bill Joy's lectures in the late 90's) is that a speaker who consults their notes as often as Bill does gives a negative impression to the audience. Note that I'm not criticising Bill in this regard, often a great talk requires some significant notes - very few people can deliver a talk in a convincing manner entirely from memory. It seems to me that the choices of camera angle are designed to give a better impression than someone who was seated in the audience might receive - there's no reason why a video of a talk should be spoiled by seeing the top of the speaker's head while they consult their notes!

• [1] http://en.wikipedia.org/wiki/Bill_Joy


• [2] http://www.wired.com/wired/archive/8.04/joy.html


• [3] http://www.acceleratingfuture.com/michael/blog


• [4] http://blog.ted.com/2008/11/what_i_worry_ab.php


• [5]
  
• [6] http://lenovoblogs.com/insidethebox/?p=161

Space and place has long been important to hackers. Whether it was/is the university lab, the workplace, the hacker con, or the particularly high-tech city, hackers congregate and meet face to face, often and everyday. One semi-new development has been the explosion and proliferation of hacker labs and spaces, such as Noisebridge located in the Mission district of San Francisco Foulab in Montreal. I recently got back from San Francisco and was able to spend a few nights at Noisebridge and was jaw droppingly impressed. The space is, well, spacious and nice (and located right by the Bart, a real +++++), but more important is that it is a thriving collective with all sorts of geeky participants and they have just souped up the space with all sorts of equipment, from the usual suspects (lots of computers) to lots of electronic gear such as oscilloscopes. What I also was impressed by was not only the blizzard of events but the open and accessible nature of the organization, which seemed to sit in some contrast to NYC’s hacker collective, NYC Resistor. Like so many organizations in this metropolis, they apparently are lacking in space to grow and the word on the streets (which I cannot confirm or deny as I have had very little contact with them but have heard this repeatedly) is that the organization has had a tough time letting in new members. Some folks are understanding of this given their space limitation, others have been less kind, and have referred to the group as a clique. There are already a few other initiatives under way to find a larger space so as to accommodate a more open, participatory atmosphere for a hacker space but I imagine that over time the culture and developments of NYC Resistor will also change as new spaces develops and do hope that this creates the conditions for more access rather than less (and again, I know next to nothing of the situation though I suspect space plays a real limiting role as it does with nearly everything else in NYC). And thankfully Rose White, a NYC-based sociology graduate student at CUNY, is paying close attention to the rise and development of these hacker spaces. She is well underway doing her dissertation dissertation on these hack spaces and I really look forward to her work.

The Timeless Way of Building: Diagnosis and Repair Architect Christopher Alexander's A Pattern Language is well known among programmers as the inspiration for the Design Patterns school of thought, but for some reason his other books are less widely read. My mother and father are both architects—"real" ones, not "software architects." :) My own amateur interest in the subject led me to read several of Alexander's books over the past couple of years. The Timeless Way of Building is the first volume of an open-ended series. (A Pattern Language is the second.) It is in this book that Alexander first lays out the idea of a pattern language. More importantly, he explains an entire theory and practice of design and construction, of which the pattern language is only one piece. Another piece of the Timeless Way is the practice of diagnosis and repair. Rather than a detailed plan of future projects, Alexander prefers to maintain a diagnosis of current problems. Each new project must be targeted at fixing these existing problem in the system: "At every moment we use the defects of the present state as the starting point for the definition of the new state" (p. 485). Alexander's goal is to create a type of spontaneous growth and order that is possible when change is incremental and decision-making authority is decentralized. Alexander was hired to produce a new type of master plan for the University of Oregon campus. The result was published as the third volume in the "Timeless Way" series: The Oregon Experiment. In it, he writes again about diagnosis and repair:

It is still not clear where global order will come from, without a master plan.… How is the problem solved in an organism? Essentially, the problem is solved by a process of diagnosis and local repair.… The master plan tells us what is right for the future. The diagnosis tells us what is wrong for the present. (p. 146–148)

In software, the same problems and conclusions have led to "agile" processes, which emphasize incremental change and flexible planning. But I haven't seen any software companies adopt a consistent practice of "diagnosis"—keeping track of what's wrong or unsatisfactory with the current system. This extends beyond defect tracking; it should include a constantly updated list of code that, although it works, is hard to understand, difficult to change, or otherwise likely to cause problems. (This should be easy to generate—every programmer knows certain parts of their code base that they dread working on for one reason or another.) This knowledge can then guide new development to include "repair" and address such problems before they become defects. I've applied this on a small scale to my own work, and now I wish I had used it at Amazon, where my team spent much more time maintaining and improving old code than writing new code.

I knew for a long time that rails is somewhat fun to work with. The first time I read something about it, I went to check it out, and kindof liked the idea. Web development really isn't my cup of tea, so I didn't do anything with rails; but really, the idea made somewhat sense to me. It's unfortunate that the implementation is such a piece of crap. Luckily I don't have to deal with it firsthand, but the stories I hear and read everywhere speak for themselves. I won't enumerate them all; but in case you're not familiar with them, the Debian Ruby team has a pretty clear explanation on their website. In addition to complaints by packagers, I also hear that many ruby developers recommend that you install gems inside the project you're running (or that you ship your software in such a way that it includes those libraries as part of your source). Oh, the horror. Now, however, they've really topped it. Via Koen's blog, I found some other page that contains this:

Now, in theory, to update to the latest version of RubyGems, you just need to run

sudo gem update --system

But - depending on what version of RubyGems you have installed, running that command may lie and tell you that you have nothing to update. If you re currently on RubyGems 1.1 or 1.2, you need to run a different set of commands to update RubyGems:

sudo gem install rubygems-update
sudo update_rubygems

Seriously. This is why we have distributions, packagers, and, really, people spending a whole shitload of time making sure everything works together nicely. This is why there is Debian, RedHat, CentOS, Mandrake, SuSE, and whatever. Me touching rails? Not with a 10-foot pole. Not if you pay me a million, in any currency. Stay away from my system!

I really did not like phpMyAdmin project website for some time and wanted to change it. However it is harder and harder to find time to do some bigger thing, so this task was stuck in my queue for quite a long time. One of biggest problem is that the website is hosted on SourceForge.net and they limit quite a lot what you can do there. Especially including external resources is hard - the website should include lot of information from SourceForge.net as news posts, file releases etc. The only reasonable option to get all external data into the website is to generate static content and push it to SourceForge.net web server. This decision was quite clear to me and it lead me to using Python with Genshi templating to generate the website. Yes, you read it correctly - one of most popular PHP applications is using Python to generate its website. I started to play with Genshi and feedparser about one month ago, with no clear concept of pages design, just to learn the technologies. However after creating first proposal and showing it to some friends, phpMyAdmin team members and frequent visitors to phpmyadmin IRC channel, I was quite surprised about positive feedback, so I started to work a bit harder to make it possible to have new website before we release 3.1.0. And here it comes, just check it out at http://www.phpmyadmin.net/.

I’m sceptical about Global Enterpreneurship Week after last year’s problems, but today is Social Enterprise Day, so I’ve tried to get involved with the WalesCoop Ethical Entrepreneurs Online Discussion from 1-2pm and 7-8pm today. Come join us!

My current desktop PC has an RS790-based Radeon on-board graphics controller. It also has a Radeon X1900 plugged in. Playing with my Watts Up, I found that the system was (at idle!) drawing around 35W more power with the X1900 than with the on-board graphics.

This is clearly less than ideal.

Recent Radeons all support dynamic clock gating, a technology where the clocks to various bits of the chip are turned off when not in use. Unfortunately it seems that this is generally already enabled by the BIOS on most hardware, so playing with that didn't give me any power savings. Next I looked at Powerplay, the AMD technology for reducing clocks and voltages. It turns out that my desktop hardware doesn't provide any Powerplay tables, so no joy there either. What next?

Radeons all carry a ROM containing a bunch of tables and scripts written in a straightforward bytecode language called Atom. The idea is that OS-specific drivers can call the Atom tables to perform tasks that are hardware dependent, even without knowledge of the specific low-level nature of the hardware they're driving. You can use Atom to do several things, from card initialisation through mode setting to (crucially) setting the clock frequencies. Jerome Glisse wrote a small utility called Atomtools that lets you execute Atom scripts and set the core and RAM frequencies. Playing with this showed that it was possible to save the best part of 5W by underclocking the graphics core, and about the same again by reducing the memory clock. A total saving of 9-10W was pretty significant.

The main problem with reducing the memory clock was that doing it while the screen is being scanned out results in memory corruption, showing up as big ugly graphical artifacts on the screen. I'm a fan of doing power management as aggressively as possible, which means reclocking the memory whenever the system is idle. Turning the screen off to reclock the memory would avoid the graphical corruption but introduce irritating flicker, so that wasn't really an option. The next plan was to synchronise the memory reclocking to the vertical refresh interval, the period of time between the bottom of a frame and the top of the next frame being drawn. Unfortunately setting the memory frequency took somewhere between 2 and 20 milliseconds, far too long to finish inside that time period.

So. Just using Atom was clearly not going to be possible. The next step was to try writing the registers directly. Looking at the R500 register documentation showed that the MPLL_FUNC_CNTL register contained the PLL dividers for the memory clock. Simply smacking a new value in here would allow changing the frequency of the memory clock with a single register write. It even worked. Almost. I could change the frequency within small ranges, but going any further resulted in increasingly severe graphical corruption. Unlike the sort I got with the Atom approach to changing the frequency, this corruption manifested itself as a range of effects from shimmering on the screen down to blocks of image gradually disappearing in an impressively trippy (though somewhat disturbing) way.

Next step was to perform a register dump before and after changing the frequencies via Atom, and compare them to the registers I was programming. MC_ARB_RATIO_CLK_SEQ was consistently different, which is where things got interesting. The AMD docs helpfully describe this register as "Magic field, please use the excel programming guide. Sets the hclk/sclk ratio in the arbiter", about as helpful as being told that the register contents are defined by careful examination of a series of butterflies kept somewhere in Taiwan. Now what?

Back to Atomtools. Enabling debugging let me watch a dump of the Atom script as it ran. The relevant part of the dump is here. The most significant point was:

MOVE_REG @ 0xBC09
src: ID[0x0000+B39E].[31:0] -> 0xFF7FFF7F
dst: REG[0xFE16].[31:0] <- 0xFF7FFF7F

, showing that the value in question was being read out of a table in the video BIOS (ID[0x0000+B39E] indicating the base of the ROM plus 0xB39E). Looking further back showed that WS[0x40] contained a number that was used as an index into the table. Grepping the header files gave 0x40 as ATOM_WS_QUOTIENT, containing the quotient of a division operation immediately beforehand. Working back from there showed that the value was derived from a formula involving the divider frequencies of the memory PLL and the source PLL. Reimplementing that was trivial, and now I could program the same register values. Hurrah!

It didn't work, of course. These things never do. It looked like modifying this value didn't actually do anything unless the memory controller was reinitialised. Looking through the Atom dump showed that this was achieved by calling the MemoryDeviceInit script. Reimplementing this from scratch was one option, but it had a bunch of branches and frankly I'm lazy and that's why I work on this Linux stuff rather than getting a proper job. This particular script was fast, so there was no real reason to do it by hand instead of just using the interpreter. Timing showed that doing so could easily be done within the vblank interval. This time, it even worked.

I've done a proof of concept that involved wedging this into the Radeon DRM code with extreme prejudice, but it needs some rework. However, it demonstrates that it's possible to downclock the memory whenever the screen is idle without there being any observable screen flicker. Combine that with GPU downclocking and we can save about 10W without any noticable degradation in performance or output. Victory!

I gave the code to someone with an X1300 and it promptly corrupted their screen and locked their machine up. Oh well. Turns out that they have a different memory controller or some such madness.

So, obviously, there's more work to be done on this. I've put some test code here. It's a small program that should be run as root. It should reprogram an Atom-based discrete graphics card[1] to half its memory clock. Running it again will halve it again. I don't recommend doing that. You'll need to reboot to get the full clock back. This isn't vblank synced, so it may introduce some graphical corruption. If the corruption is static (ie, isn't moving or flickering) then that's fine. If it's moving then I (and/or the docs) suck and there's still work to be done. If your machine hangs then I'm interested in knowing what hardware you have and may have some further debugging code to be run. Unless you have an X1300, in which case it's known to break and what were you thinking running this code you crazy mad fool.

Once this is stable it shouldn't take long to integrate it into the DRM and X layers. I'm also trying to get hold of some mobile AMD hardware to test what impact we can have on laptops.

[1] Shockingly enough, it's somewhat harder to underclock graphics memory on a shared memory system

So, the yum package manager (and, I assume, RPM underlying that) has a fairly neat (to my naive eyes, anyway) implementation of multiarch support -- on an amd64 system you can install i386 packages, and they go into a separate lib tree, and there's no conflicts or unhappiness. It even installs both the i386 and x86_64 versions of libraries automatically when you ask for them, which is a nice touch. However, the dependency resolver doesn't quite seem to have gotten the hang of some of the corner cases. For instance, if you want to install a noarch package that depends on an architecture-specific package, but for some reason only the i386 version is available (say, your local package repo hasn't quite built the x86_64 version yet) then yum will happily install the i386 version and say "OK, dependency resolved". Nice try, but no cigar. What makes it a bit more irritating is that once you work this out and the x86_64 package is available, you need to remove the i386 version of the package by hand in order for it to pick up the x86_64 package and install it. Just asking yum "hey, can you install this package" results in "nope, it's already installed, go away".

I have been using Serendipity on my blog for some time now. Overall, I’ve been pleased with it, but the conversion was a pain. Serendipity is a simple blog engine, and has a wonderful built-in plugin system. It can detect what plugins need upgrading, and install those upgrades, all from directly within the management interface. There’s no unzipping stuff in install directories as with WordPress.

On the other hand, Serendipity has had some long-standing issues. #1 in the list is spam protection. I’ve been using the Captha system, but as some others have commented on here periodically, it can be a hassle for legitimate humans. The Serendipity implementation in particular, I mean. Another issue was with the plugins themselves. Because the software does updates itself, and plugins tend to write things in the install directory, your Serendipity install directory has to be writable by the webserver. Morever, it is difficult to keep track of local changes — forcing me to use git on the tree, and even then, it’s difficult. Finally, there has been some long-standing weirdness with the RSS feeds. I switched to WordPress over the weekend. It doesn’t solve everything, but it is a nice improvement. It has a ton of themes available, though many of them are not as high-quality as the themes in Serendipity, and it is hard to pick out the good ones quickly. WordPress has a larger development community and seems to be getting new features and improvements at a more rapid rate than Serendipity. I particularly like how it automatically saves drafts as I’m typing. That is a wonderful feature. I can’t tell how many draft posts I’ve lost by accidentally clicking on the wrong thing. Serendipity has a ton of plugins as well. I looked for Twitter plugins today. There are maybe a dozen of them. But there is no central tracking of them, so I have no idea which ones might be nicely peer-reviewed and which ones might rm -rf / on my server. Also, I don’t know which ones are going to be supported and upgraded over time. I’m happy with the way most WordPress themes work nicely with widgets. Though I am baffled that some WP themes hard-code in things like the calendar or archives when they could just let people insert a widget for it. The Conversion Process This was a real pain. I started with a Serendipity to WordPress converter. But alas, it only supports Serendipity databases in MySQL and mine’s in PostgreSQL. I used SQuirreL SQL with the DBCopy plugin to convert it. This worked… sorta. A few tables didn’t convert, but I didn’t need those. Some others had indexes that wouldn’t convert, and after re-trying it enough times, it went. Then I ran the converter. Alas it had some bugs and had to be fixed up with current table names and some things like that. Also I hacked it to preserve ID numbers. So eventually this migrated over the posts and categories. It did not preserve the category hierarchy. It also didn’t recognize when nl2br was disabled, and munge the post appropriately. And it didn’t migrate over tags, so I wrote a Haskell program to do that. Then it was a matter of crafting a bunch of RedirectPermanent and RewriteRule entries for Apache to try to redirect the old article pages and RSS feeds to their new locations as much as possible. Final Thoughts I’m glad I did it, and I don’t plan to do this again for a long time.

Australia has slow Internet access when compared to other first-world countries. The costs of hosting servers are larger and the cost of residential access is greater with smaller limits. I read news reports with people in other countries complaining about having their home net connection restricted after they transfer 300G in one month, I have two net connections at the moment and the big (expensive) one allows me 25G of downloads per month. I use Internode, here are their current prices [1] (which are typical for Australia - they weren’t the cheapest last time I compared but they offer a good service and I am quite happy with them). Most people in Australia don’t want to pay $70 per month for net access, I believe that the plans which have limits of 10G of download or less are considerably more popular. Last time I investigated hosting servers in Australia I found that it would be totally impractical. The prices offered for limits such as 10G per month (for a server!) were comparable to prices offered by Linode [2] (and other ISPs in the US) for hundreds of gigs of transfer per month. I have recently configured a DomU at Linode for a client, Linode conveniently offers a choice of server rooms around the US so I chose a server room that was in the same region as my client’s other servers - giving 7 hops according to traceroute and a ping time as low as 2.5ms! Currently I am hosting www.coker.com.au and my blog in Germany thanks to the generosity of a German friend. An amount of bandwidth that would be rather expensive for hosting in Australia is by German standards unused capacity in a standard hosting plan. So I get to host my blog in Germany with higher speeds than my previous Australian hosting (which was bottlenecked due to overuse of it’s capacity) and no bandwidth quotas that I am likely to hit in the near future. This also allows me to do new and bigger things, for example one of my future plans is to assemble a collection of Xen images of SE Linux installations - that will be a set of archives that are about 100MB in size. Even when using bittorrent transferring 100MB files from a server in Australia becomes unusable. Most Australians who access my blog and have reasonably fast net connections (cable or ADSL2+) will notice a performance improvement. Australians who use modems might notice a performance drop due to longer latencies of connections to Germany (an increase of about 350ms in ping times). But if I could have had a fast cheap server in Australia then all Australians would have benefited. People who access my blog and my web site from Europe (and to a slightly lesser extent from the US) should notice a massive performance increase, particularly when I start hosting big files. It seems to me that the disadvantages of hosting in Australia due to bandwidth costs are hurting the country in many ways. For example I run servers in the US (both physical and Xen DomUs) for clients. My clients pay the US companies for managing the servers, these companies employ skilled staff in the US (who pay US income tax). It seems that the career opportunities for system administrators in the US and Europe are better than for Australia - which is why so many Australians choose to work in the US and Europe. Not only does this cost the country the tax money that they might pay if employed here, but it also costs the training of other people. It is impossible to estimate the cost of having some of the most skilled and dedicated people (the ones who desire the career opportunities that they can’t get at home) working in another country, contributing to users’ groups and professional societies, and sharing their skills with citizens of the country where they work. Companies based in Europe and the US have an advantage in that they can pay for hosting in their own currency and not be subject to currency variations. People who run Australian based companies that rent servers in the US get anxious whenever the US dollar goes up in value. To quickly investigate the hosting options chosen for various blogs I used the command “traceroute -T -p80” to do SYN traces to port 80 for some of the blogs syndicated on Planet Linux Australia [3]. Of the blogs I checked there were 13 hosted in Australia, 11 hosted independently in the US, and 5 hosted with major US based blog hosting services (Wordpress.com, Blogspot, and LiveJournal). While this is a small fraction of the blogs syndicated on that Planet, and blog hosting is also a small fraction of the overall Internet traffic, I think it does give an indication of what choices people are making in terms of hosting. Currently the Australian government is planning to censor the Internet with the aim of stopping child porn. Their general plan is to spend huge amounts of money filtering HTTP traffic in the hope that pedophiles don’t realise that they can use encrypted email, HTTPS, or even a VPN to transfer files without them getting blocked. If someone wanted to bring serious amounts of data to Australia, getting a tourist to bring back a few terabyte hard disks in their luggage would probably be the easiest and cheapest way to do it. Posting DVDs is also a viable option. Given that the Internet censorship plan is doomed to failure, it would be best if they could spend the money on something useful. Getting a better Internet infrastructure in the country would be one option to consider. The cost of Internet connection to other countries is determined by the cost of the international cables - which can not be upgraded quickly or cheaply. But even within Australia bandwidth is not as cheap as it could be. If the Telstra monopoly on the local loop was broken and the highest possible ADSL speeds were offered to everyone then it would be a good start towards improving Australia’s Internet access. Australia and NZ seem to have a unique position on the Internet in terms of being first-world countries that are a long way from the nearest net connections and which therefore have slow net access to the rest of the world. It seems that the development of Content Delivery Network [4] technology could potentially provide more benefits for Australia than for most countries. CDN enabling some common applications (such as Wordpress) would not require a huge investment but has the potential to decrease international data transfer while improving the performance for everyone. For example if I could have a Wordpress slave server in Australia which directed all writes to my server in Germany and have my DNS server return an IP address for the server which matches the region where the request came from then I could give better performance to the 7% of my blog readers who appear to reside in Australia while decreasing International data transfer by about 300MB per month.

• [1] http://www.internode.on.net/residential/internet/home_adsl/nakedextreme/pricing/


• [2] http://www.linode.com/


• [3] http://planet.linux.org.au/


• [4] http://en.wikipedia.org/wiki/Content_Delivery_Network

A few years ago, I posted a story about my frustrations with Blue Cross Blue Shield. They were not coughing up the dough for a 4000 dollar bill and for the life of me, I could not get them to do it and I could not even get in touch with people in the organization to help me. After that post circulated, the Public Relations director wrote me and this helped me get the access I needed to eventually get all the money (it did take almost 3 years though and I should write about that but later). Though the ethical stakes and scenario are totally different, I have had equal problems contacting someone, a live body, a person who might reply to an email, a person who might return a phone call, from the American Anthropological Association. Next week is their Annual Meeting in San Francisco, CA and I am giving a presentation. I simply want to inquire whether there is A/V equipment in the room for if there is not, I need to do some serious shuffling of my talk because it is currently so dependent on the audio visual components. Given that I coughed up $ 400 bucks (and some change) to be a member and to attend the zoo that is this conference, given that it costs money to fly there, given that it is a member supported organization, one would think that one could just get a simple answer to a simple question. I realize they are not staffed to field endless inquiries but it is a conference they are putting on and thus, I do expect an answer or basic information about the A/V equipment (which we also requested when we signed up) especially after 1.5 months of trying. Anyhow, I am frustrated and I realize this is a really silly and small annoyance. But if no one ever calls them on it, there is no way for them to every change their archaic and pathetic ways.

One month with Emacs and counting - Part 2 A while ago I've blogged about my switch from Vim to Emacs, promising a blog post series, quite a mouthful nevertheless, it's time to continue the series. The first part was about why I think that nowadays Emacs is ready to be switching to. This second part is about flawed Vim design choices which substantially contributed to my choice. (Of course all this post is under a IMO-/rant-disclaimer, it can't be otherwise, so take it cum grano salis) Flawed Vim design choices Vim started as a wonderful editor, following the path of Vi, which in turn has got right a good deal of design choices. One is the often debated modality. In spite of being a HCI PITA in general, modality is not a problem at all for software you are using every single day (and the editor is the 2nd program I use the most, with the terminal being the 1st), because you learn it as a habit. In the case of Vim, modality is good because it let you use powerful yet simple motions in command mode, which can be easily combined with operator-pending commands. This can be considered yet another incarnation of the UNIX philosophy, casted inside the editor itself. ... which brings us to the good old joke which describes Emacs as a great operating system, lacking only a decent editor. In fact, Vim is no longer the nice Unix philosophy player which it used to be, and Emacs plays much more nicely with external tools than Vim. As arguments, let's consider a few (not so) recent Vim evolutions:

• Spell checking. Starting with version 7, Vim has got support for on-the-fly spell-checking of text buffers. It is a super-nice feature, which I've advertised in the past, because spell checking is needed in many tasks we daily do with editors (mail/doc writing, code commenting), and having a single interface to do it comes handy. Unfortunately, how that is implemented in Vim is definitely not along the lines of the UNIX philosophy. We have countless different FOOspell implementations whereas Vim implemented yet another one, without relying on any pre-existing implementation. Even more so, the file format of spell files is basically ad-hoc, and not sharable with other tools. This is already a PITA for packaging (let's add to Debian another 20 binary packages vim-spellfiles-FOO), but is even worst than that: the spell files must be generated in ad-hoc ways which are seriously prone to failures (guess why in Debian we don't have yet the vim-spellfiles-FOO packages which used to exist in experimental a long time ago?). What would have been the UNIX-friendly way of achieving the same result? Well, obviously spawning FOOspell and interact with its process from the editor. Keep that in mind.
• Internal grep. A feature of Vim I've always loved is :grep (yet another instance of quick-fixing, which is also used for jumping to compile-time errors). In its original incarnation it was as simple as invoking grep and parse its output. KISS. Then, Vim started to have a problem: grep -r can take ages and in the meantime the editor was stuck. Hence, starting with Vim 7, vimgrep has born: an internal re-implementation of grep. OK, it is more portable (but how many of us do care? on how many system you use you are missing grep? damn, I used to install it also on Windoze!), and the editor retain control. Control that is exploited just to show a nice progress bar of what is being scanned ..., while the editor is still stuck. Well, to me it looks like the problem would have been solved much more nicely by adding the ability to interact with an external process (deja vu?). That process can then be grep -r. Full stop.
• Lack of debugging support (on purpose). Long standing missing feature of Vim: you can not "easily" drive a debugger from Vim, because a firm choice in Vim is to ... (OK, I confess, I'm starting to beat a dead horse) ... not have support for interaction with external processes. I found such a choice quite dumb: to me it is evident that it induces a trade-off between being feature-complete wrt programmer needs (as available external tools get available to face those needs) and not "re-implementing the wheel" inside the editor. Going back to the lack of debugging support in Vim, that has spawned projects like GdbVim, Clewn, Bram's Agide, and my own tiny teeny incarnation specific to ocamldebug called WOWcamldebug. Every single project of that list acts as an intermediary between Vim and an external process (a debugger of some sort), whereas that could have been implemented by an editor plugin, provided that the editor offer, guess what, support for interaction with external processes.
• Top-level. Similar to the debugging support issue, there is the interpreter issue, namely how to evaluate "phrases" of your interpreted language of choice directly from the editor. Nowadays a lot of languages have the so called top-levels which evaluate language phrases interactively (Python, Ruby, OCaml, many Lisps, ...) and show evaluation results. That ability can speed up testing considerably, but can become cumbersome to use for large code snippets without editor support (good luck with copy and paste). Yes, Vim has support for that, but the implementation choice is close to be ridicolous: only if you have linked Vim together the interpreted for a given language, then you will be able to interactive evaluate phrases of that language. What if you program in Python, Ruby, and Perl? Well, you need to link all of them. What if you also program in OCaml? Then you're screwed, because Vim currently does not support linking with it. (Yes, I know that there are other reasons for doing that linking, they are discussed below.) Do I need to tell you which feature will be enough to address top-level support? interaction with extern.. OK, I'll drop it, you know that already.

Well, I acknowledge that Emacs got this choice right, and that it's an important one: Emacs supports interaction with external processes, and plugin authors have been happy to exploit it to create really cool stuff (sticking to the UNIX philosophy). Please welcome: Flyspell, M-x grep (shameless smartass plug: same number of keystroke of :grep), Grand Unified Debugger, a sampling of language-specific major modes with top-level support. (But I recommend having a look at least once at the coolness of other stuff built-on top of external process support, like Flymake, mentioned by dancer not a long ago.)

• Its own scripting language. I've bored you enough with the external process topic, hence here is the second one: the saga of Vim and its scripting language, i.e., the programming language using which you can customize the editor. Note that the choice of scripting language impacts on 2 different targets: final users in need of fire and forget automations (when they start to get to complex to be implemented on top of, say, macros), and developer of editor extensions such as addons. The impact of scripting language on extension developer is crucial for the evolution of an editor aiming at being powerful. If the scripting language and the API towards the editor is flexible enough, then a lot of cool extensions and features will be developed by third party authors, otherwise the burden stays on editor upstream author.
  1. In its phase 1 (my term, Vim < 7.0) Vim pretended that it needed only a minimal scripting language (Vim script) and that everything else can be developed using external programming languages, linked in the editor. As a consequence Vim script was just a tiny teeny procedural language with editor interfacing capabilities, most notably lacking any "decent" data structures (no lists, no dictionaries, ...). In that phase, if you were a user it was fair enough. You just had to chose your external programming language (at compile time ...), and you were able to write your quick hacks with it. But if you were willing to develop an extension you were screwed, because you had either to impose your external programming language to the final user (that is what gave born to absurdities like vim-full, look at the deps!) or to enjoy a good deal of masochism to explode your 30 lines of Python extension into 200 lines of Vim script to make your Vim extension more "portable".
  2. In its phase 2 (Vim >= 7.0) it has been realized that something were wrong in the initial choices about Vim script, and data structures (lists, dictionaries, function references!) have been added to it, relieving a bit the pain of programming portable extensions. Nevertheless, and probably due to how it started, Vim script is far from being a programming language one can enjoy programming in. Consider for example the standard library of the language. It started as a random collection of unrelated functions being added on a "as-needed" basis, ... and continued along that path! Now the API reference lists together totally unrelated functions, organized inconsistently, and making really painful finding what you need (assuming it exists).

Even in this respect, I acknowledge that Emacs got this right. It has chosen a single, now full-fledged, programming language (Elisp), which is offered both to the final user (fact: the average Emacs user knows much better how to program her own editor than the average Vim user) and to extensions developer. The standard library of the language is organized with a consistent naming (even though not always intuitive) and documented in an organized manner. You might not like the language, but at least is has been taken seriously and managed as such. Finally, being a Lisp dialect, you might need what you learn with Elisp elsewhere. Vim script is too committing: it is useful only inside Vim. I believe that contributed significantly in its scarce diffusion among Vim users. The fact that Elisp was born together with Emacs and that it is also used internally for the editor implementation is not relevant here: I do care about what is offered to me as an user and as an extension developer. What I see on the Emacs side in this respect is much better that what I see on the Vim side. Moreover, from the point of view of the users, is the difference between what is inside the editor and what is outside (i.e., the extensions) that relevant? I think it is not, and that brings me to the final Vim choice which I consider flawed and which has the potential of seriously limit its future evolutions:

• Editor as a distribution. Emacs is managed as a intermediate software distribution layer, similarly to what happens in other softwares where many third party authors cooperates (e.g., TeXLive). The distribution architecture has advantages and contributes to the longevity and potentialities of a software project. What does this mean concretely in the Vim vs Emacs dispute? For example it means that Emacs get more feature-complete each passing release, still preserving uniformity in documentation and keybindings. On the contrary Vim sports many addons, which are very likely to step on each other feet when enabled simultaneously. Actually, that is one of the reason which brought me to the creation of vim-addons: you cannot enable all the content of vim-scripts together due to various kind of conflicts. As we know well in Debian, distributions have a fundamental role in blending together lightly-coupled software components, that's role is missing in Vim evolutionary path and I find that worrisome.

Where to from here in this blog post series? For sure some tips on how to migrate for hard-core Vim users, then we'll see ... If you did enjoy the read please let me know commenting in the discussion page (as you did in the last post, thanks!).