We added support for Marvell's Kirkwood platform to the Debian installer a few months ago and a lot of people have installed Debian testing (squeeze) on devices such as the SheevaPlug and QNAP TS-11x/TS-21x. While this works great, some people would prefer to run Debian stable (lenny). Until recently, I thought we wouldn't be able to support lenny in the installer since the kernel in stable doesn't have support for Kirkwood. However, some work by Frans Pop showed me that it would be quite trivial to change the installer so it would install Debian stable plus the kernel from an alternative repository. So from now on, it's possible to install Debian stable (lenny) on the SheevaPlug, QNAP TS-11x/TS-21x and OpenRD. This installation mechanism uses the squeeze installer to install Debian lenny (stable) plus the kernel from a repository I maintain. This repository usually contains the kernel from Debian testing (although I sometimes add the kernel from unstable if it has some important features). Since some users might be wondering which version to install, here is an overview of the benefits and downsides of each version. Debian lenny (stable): it is the current stable version of Debian (version 5.0).

• lenny has security support (note that this does not apply to the kernel, which we take from squeeze).
• lenny is stable (i.e. it doesn't change, except for security and other important updates).
• The software in lenny is older (since lenny was released in February 2009).

Debian squeeze (testing): it is currently under development and will be the next stable version of Debian (version 6.0).

• squeeze changes every day, with new software coming in.
• squeeze has fairly new software.
• While there is some security support for testing (squeeze), the security support for stable is better.

The Debian installer team today announced the alpha1 version of the installer for Debian squeeze (6.0). This release adds a lot of new features but I just wanted to highlight the ARM related enhancements. With this release, Marvell's Kirkwood platform is supported. Specifically, the installer supports the following devices: QNAP TS-110 and TS-119, QNAP TS-210, TS-219 and TS-219P, SheevaPlug and OpenRD. In addition to Kirkwood support, Wouter Verhelst added support for the Intel Storage System SS4000-E.

Linux.conf.au 2010 has come to an end and I am looking back at an intense week of conferencing. A big shout out to the organisers for their excellent work. I think LCA (as well as DebConf) just keeps getting better every year. This does not at all discredit previous organisers, because they were the best at their times and then passed on the wisdom and experience to help make it even better in the following year. The week started off with the DistroSummit, which Fabio and I organised. Slides are forthcoming, as I failed to get them off the speakers right after their talks it s interesting how stress levels and adrenaline can cause one to forget the most obvious things. This is where experience comes in. I ll be there again next year, I hope, to do things better. The theme of the day was cross-distro collaboration, and we started the day a little bit on the Debian-side with Lucas Nussbaum telling us about quality assurance in Debian, alongside an overview of available resources. We hoped to give people from other distros pointers, and solicit feedback that would enable us to tie quality assurance closer together. Next up was Bdale Garbee who talked about the status of the Linux Standard Base. While I am really interested in such standardisation efforts, I realised during his talks that I had considerable difficulties paying attention because as organiser of the conference, I had all sorts of other things occupying my thoughts. I proceeded to tell the audience the room was mostly filled throughout the day with an estimated 40 50 folks, and I d say about half of them stayed throughout, while the other half came in and left the room between talks. I could not get the projector to work with my laptop after the upgrade to Kernel Mode Setting, and thus used the whiteboard to give a brief introduction to vcs-pkg.org, talk about the current state of affairs, summarise the trends in discussions around patch management and collaboration, give an outlook of what s up next, and solicit some discussion. Sadly, just like during Bdale s talk, I found myself worrying over the organisation of the day, rather than actually taking in most of the discussion. Fortunately, others have written about the most important points, so I defer to them. Michael Homer then told us about GoboLinux s Aliens system, which is a way to integrate domain-specific packages with distro-specific package maintenance e.g. how to get APT to handle CPAN directly, or how to let YUM manage Python packages. The ensuing discussion was interesting, and we carried it over to the next slot, because Scott, the next speaker, was stuck in traffic. To summarise briefly: scripting languages have a lot of NIH-style solutions because it works for them, but these are a nightmare to distro packagers. One symptom of the status quo is that complex software packages like Zimbra are forced to distribute all required components in their installation packages, which make distro packaging, quality assurance, and security support even harder. I don t think we found a solution, other than the need for further standardisation (like the LSB), but the road seems to be a long and windy one. Laszlo Peter introduced the audience to SourceJuicer, a new build system used by OpenSolaris. The idea is that contributors submit packages via a web interface, kicking off a workflow incorporating discussion and vetting, and only after changes have been signed-off are packages forwarded to auto-builders and eventually end up in the package repository. This is very similar to upload ideas I ve had a while ago, which I ve started to (finally) implement. Unfortunately, SourceJuicer seems very specific to OpenSolaris, as well as non-modular, so that I probably won t be able to reuse e.g. the web interface on top of a Debian-specific package builder. After the break, Dustin Kirkland stepped up to tell us about his user experience of Launchpad. Unfortunately, I found his talk a bit too enthusiastic. Launchpad undoubtedly has some very cool features and ideas, but it s just one of the available solutions. The dicussion of Launchpad also dominated the next talk, in which Lucas Nussbaum told us about the Debian-Ubuntu relationship. While his presentation showed that the relationship was improving (Matt Zimmerman made the point that there are rather many relationships, rather than one relationship), I was a bit disturbed by the comments of Launchpad developers in the room, ranging from Debian is declining anyway to Just use Launchpad if you want to collaborate with others and not go down . There was a slight aura of arrogance in their comments which tainted my experience of the otherwise constructive discussions of the day. Overall I had a great time. Debian and Ubuntu made up the vast majority of attendants, with only a handful of representatives from other distros present. I wonder why that would be. One reason might be that around 70% of LCA attendants declared themselves Debian or Ubuntu users, and so there weren t many other distros around. Another might be that I still haven t spread the word enough. Let s hope to do better next year! Thanks to all the speakers. We may have organised the day, but you made it happen and interesting! Slides and recordings of the talks will be linked from the archived website when they become available (yes, the archive page does not exist yet either).

I know, preaching to the choir and all that. But this can't be said often enough (and perhaps the odd non-Debian person stumbles upon this...): Here's another example of how amazingly fast free tech support works: The newest Linux kernel (Debian package version: 2.6.32-1) wouldn't boot on my QNAP TS-419P NAS (see also my earlier posting about the device). Now, since I don't have a serial console cable, I can't really help a lot to debug this (and I am extremely happy that the people at QNAP have thought about recovery: it's trivial to just flash a working kernel or firmware image via DHCP and TFTP), bu still Martin Michlmayr immediately took the time to help me and soon could reproduce it on one of his own machines. The problem has been identified, I got a fixed kernel, and a patch is on his way to the kernel maintainers. All in the after literally just a few days.

After I had been living with mostly the same electronic toys over the last few years (ok, I got the netbook earlier this year, but that's about it... and it might even count as a serious work tool given how much I came to use it) all the toys I've agonized over buying suddenly appeared on the doorstep within these two weeks... First part was the QNAP TS-419P which I bought after reading about QNAP on Martin Michlmayr's blog. The decision was quick when I realized that at least part of the Debian related information was not hidden in some obscure web forums but was there on the QNAP wiki itself, indicating quite some level of support from the manufacturer. Beyond that, tbm does an awsome job supporting QNAP users like myself through his excellent QNAP / Kirkwood pages and also, since Debian installer support for the 419 is not quite finished, by email. Kudos to him! So, once I get past some remaining obstacles (for some reason, the initrd refused to mount a raid1 root device when I tried it at first; working on that, but it's not that easy to debug without console access...) our old storage / Internet router (very old! Pentium II 350MHz with 128M RAM and its probably third internal disk, which became ridicolous to use when I bought a (big) external USB disk and realized it only has USB 1...) can be retired. Should be good for the environment, too, I should hope, in terms of power consumption even considering that I now run 3 more disks.

For those who haven't seen it yet, LinuxDevices published an article recently looking at the roadmap of Marvell's ARM line. The new line is called Armada and for Debian the Armada 510 (known as Dove) is of particular interest. To me, it essentially looks like a Kirkwood (the current platform) but with ARMv6/v7 (instead of ARMv5), integrated VGA and some other features. According to the article, the Armada 510 is aimed at "high-end smartbooks and tablets". I'm happy to see the integration of VGA because I'd like to see more ARM based smartbooks, tablets and thin clients. At the same time, I'm worried that the VGA will be some proprietary chip without proper open source drivers and I'm surprised that the new chip only offers 1.2 GHz. After all, the current Kirkwood chip clocks 1.2 GHz already, so I'd have expected an increase to 2.0 GHz for the next generation.

When I visited Taiwan last week to talk about Debian at a conference on smartbooks, I used the opportunity to meet up with the folks from QNAP. It was really nice to meet many of my contacts at QNAP in person. We talked about their roadmap and existing products and I found out that they had just released a number of new devices that may be of interest to Debian users. I really like the hardware from QNAP but one downside of their high quality is also that the devices are fairly expensive. Last week they introduced a number of lower cost alternatives: in addition to the TS-119 and TS-219, you now have the TS-110 and TS-210. They feature a 800 MHz CPU (instead of 1.2 GHz on the TS-119/TS-219), 256 MB (instead of 512 MB) and have a plastic case (as a result of which, the TS-110 now has a fan unlike the TS-119). Similarly, in addition to the TS-419 and TS-419U, you now have a TS-410 and TS-410U. Since the TS-110/TS-210 and TS-119/TS-219 are compatible, the Debian installer will work out of the box.

You can now use the Debian installer to install Debian on the Marvell SheevaPlug. This routine will install Debian testing (squeeze), which is currently under development. The installer itself is also under development, so there may be problems from time to time but it should generally work pretty well. The Debian installer doesn't support installations to flash, but you can use it to install to a USB stick or disk as well as to an SD card. Here are installation instructions.

Marvell made a new version of U-Boot for the SheevaPlug available last month. This version, 3.4.19, integrates SD card support and improves support for FAT32 USB partitions (although USB support in general is still a bit flaky). This page describes how to install this new version of U-Boot on your SheevaPlug. First of all, download the SheevaPlug U-Boot 3.4.19. Unpack the zip file and copy the file u-boot-3.4.19/u-boot-rd88f6281Sheevaplug_400db_nand.bin to your TFTP server. I suggest you call the file u-boot.bin. Then start your SheevaPlug and type the following: The upgrade process will ask whether you want to change the environment: Make sure to answer "n" here. We also need to make a small change to the u-boot configuration because this version of u-boot no longer accepts the console parameter we have been using. Thanks to Tom Rinehart for pointing this out. Instead of console, we'll therefore add a new bootargs_console and update bootcmd to make use of it: You'll see that bootcmd contains $(console). You have to set a new bootcmd and change the console parameter to $(bootargs_console). In my case, I'd type: Make sure you don't forget the quotes! Finally, save the environment and restart your machine:

The Debian installer for QNAP TS-119, TS-219 and TS-219P devices (based on Marvell's 1.2 GHz Kirkwood chip) is now available, together with installation instructions. The installer is still under development and it will install Debian testing, which is also under development. However, I think it's working pretty well. If you try the installer, please send me feedback.

QNAP has announced a new device earlier this month, the QNAP TS-219P. The specs are very similar to the TS-219 (1.2 GHz Kirkwood CPU, 512 MB RAM and 16 MB flash) but the device is smaller and has some other advantages. What I like about the new TS-219P:

• The device is more compact and looks really nice.
• The fan is less noisy than the one in the TS-209 and TS-219.
• There are two eSATA ports.

What I don't like so much:

• The device only has one Ethernet. I'd really like to see two Ethernet ports.
• The serial connector cannot easily be accessed through the SATA slot, something that was possible with the TS-209.

Since the TS-219 and TS-219P are quite similar, Debian and the installer will work without any problems. I updated my QNAP page to document the QNAP TS-219P, including some pictures of the new device.

There are great resources to help tune PostgresQL for performance, but say you've got a Postgres database which suddenly becomes very slow. How can you figure out what's going on?
General system stateA slow database server will typically be running out of:

• cpu,
• memory or
• disk.

So the first thing you should do is to get an overview of the system state using common UNIX tools such as:

• top (press c to see the full command line): look at CPU and memory usage
• iostat -x -m 5: look at i/o wait and service time

If you notice that memory is exhausted, you could play with the following postgresql.conf settings:

shared_buffers
work_mem
effective_cache_size

If you see a lot of I/O, then try adjusting these settings:

checkpoint_segments
autovacuum_naptime
wal_buffers

Finding slow queries
On the database side, start by determining whether:

• there is one large query holding up all of the resources or
• the number of queries is the main problem.
  

To do this, make sure that the following setting is turned on in your postgresql.conf:

stats_command_string = on

Then fire up psql dbname and look at the currently executing queries:

SELECT usename, current_query, query_start
FROM pg_stat_activity;

(You can also add the procid column to the query if you want to map a query to the process ID you see in top.)

If you can't get anything useful out of pg_stat_activity, you may want to enable slow query logging by adding this to your postgresql.conf:

log_min_duration_statement = 100

(All queries which take more than 100 ms to execute will be printed to the main Postgres log file.)
Examining a particular queryOnce you have identified a slow query, you can time it by running this command before executing the query manually:

\timing

To get an idea of where Postgres spends its time when executing that query, look at the query plan:

EXPLAIN your_query;

The numbers you see there are estimates. To run the query and get actual numbers, use this instead:

EXPLAIN ANALYZE your_query;

If you still can't figure out why a certain query takes so long, have a look at the ratio between live and dead rows in the relevant tables:

ANALYZE VERBOSE tablename;

Having too many dead rows is often a sign of insufficient vacuuming. You might want to turn autovacuum on if it isn't already, or to make it a bit more aggressive by tweaking these settings:

autovacuum_vacuum_scale_factor
autovacuum_analyze_scale_factor

Troubleshooting database performance is a bit of a black art, many thanks to Mark Kirkwood for sharing his Postgres wisdom with me.

If there's anything else you have successfully used to find the cause of your performance woes, please feel free to leave a comment.