Russ Allbery has written about the hostile reactions of sci-fi fans toward the delay of a book release [1]. Russ makes some good points regarding the issue of whether paying $100 for some books entitles a reader to have the rest of the series written (in summary - no, the author owes the fans nothing). Russ also notes that when someone makes a public promise to do something (such as writing a book) they will probably be feeling bad about being unable to live up to that, and harassing them about it is not going to make them feel any better - or make them better able to do the work. Charles Stross has written a blog post about this topic, he makes some interesting points and the comments illustrate most of them [2]. One comment claims that the author of the delayed book in question flaunts his NFL watching while the same person also criticises the author for writing a blog. This level of contempt for the author amazes me. In case anyone is interested I watch The Bill and Desperate Housewives every week. Anyone who wishes to criticise me for watching soapies instead of documentaries is welcome to do so. But I will not make any apologies for spending some of my time doing things other than coding. If I watch Desperate Housewives instead of fixing a bug that inconveniences you then you just have to deal with it. I believe that taking a regular break from work is a basic human right, apart from which there is evidence to show that working excessive hours decreases the quality and quantity of work that is completed [3]. It seems to me that members of the free software community are in a better position regarding fans than most creative people, and that this is largely due to how transparent the development process is. The most influential members of the user (fan) base are software developers. While people will be disappointed when software is not released on time or has some bugs many users know what is involved and understand that no testing will find all bugs. Even some basic shell scripting can teach people the difficulties of programming and convince them to have a little more patience than they might otherwise have. So the typical user of free software will have some idea of what is required to write software while the typical reader of a novel has no idea of what is involved in writing a book. Another benefit for the free software community in this regard is the open development process. When users are unhappy about release delays (as some Debian users were before Lenny was released) they can easily look at the bug list. If a user is pushy they can be invited to fix some bugs themselves, if they investigate the matter and discover that the bugs are too difficult for them to fix then they will probably feel compelled to stop hassling the people who can fix the bugs. Complaints from users regarding the release process can be turned around with the software is late because YOU haven t fixed any bugs . Please note that submitting bug reports is NOT criticism, it is in fact helping the development process (particularly when the bug reports contain information on how to fix the problems). This exposes another significant difference between writing software and writing a book. Even if I was a talented author I would not be able to offer any assistance to an author who had trouble completing a work simply because I wouldn t have the same vision for the plot of a book. But it s not uncommon for software development projects to be forked if the users are unhappy with some aspect of how they work, and it s also not uncommon for a programmer to voluntarily give one of their projects to another developer - sometimes developers try and convince other people to take projects from them when they don t have enough time. Even without changing the ownership of a project, anyone can submit a patch. The patches that I have received from users of my software have generally been of a high quality and it has taken little work to merge them. A final difference is that any programmer can produce a private fork of a project. I have lost count of the number of times that I have produced custom versions of free software for my own personal use or for the use of my clients. Sometimes the modifications that I made were unsuitable for wider use (IE they broke things for usage cases that I didn t care about) so I just privately made the software do what I needed. This makes the upstream developer (or team) and other contributors irrelevant to fixing the problem in question. Now this is not to say that the free software community is perfect. But it seems that certain types of bad behavior that we see demonstrated in the sci-fi fandom circles are uncommon and the people who demonstrate them are significantly outnumbered by people who oppose such actions. Publicly criticising a programmer for watching the NFL in their spare time is akin to holding a sign saying kick me . As I am a fan of science fiction (and hope to write some in the near future) I wonder whether the sci-fi community could be changed to have a culture more like that of the free software community. The web site 365tomorrows.com publishes a short sci-fi story every day [4], they have been doing this for a few years now. They started out publishing work by a small team of authors with the aim of writing a story a day for a year. Now they also take submissions from users and they have a forum for budding authors to discuss their work. The site Protagonize.com is based on the idea of collaborative fiction [5]. I wonder whether the increasing number of sci-fi fans who join such forums will change the nature of the community. Maybe when a book is delayed the fans could take comfort in an increasing amount of fan-fiction based on characters from that book, I expect that the fan fiction will probably not impress all authors - and some may be offended by having their characters used in pornographic fan fiction. But that might just lead to a market shift towards authors who are not overly concerned about such things. When attending sci-fi book signings I ve noticed fans who appear to have stalking tendencies. I have been wondering how much of this is another symptom of the same problem and how much of it is due to science fiction attracting some people who don t understand the concept of personal space and who haven t considered the mathematics of interactions between famous people and fans - as one of many fans the famous person will not remember your fan letters or whether they have met you before, it s not possible. I look forward to the day when a well known author can have lunch with some fans after a book signing and not be afraid of weird fans. I wonder if Cory Doctorow [6] already has such cool fans, Cory seems to be one of the leaders in the new open way of writing and releasing sci-fi so it seems likely that he would be among the first to reap the benefits. I m sure that most authors would like to have lunch or dinner with their fans if they could be assured that the probability of getting stuck with a nutty fan was extremely low.

• [1] http://www.eyrie.org/~eagle/journal/2009-02/016.html


• [2] http://www.antipope.org/charlie/blog-static/2009/02/the_art_of_being_late.html


• [3] http://etbe.coker.com.au/2007/11/02/increasing-efficiency-through-less-work/


• [4] http://www.365tomorrows.com/


• [5] http://www.protagonize.com/


• [6] http://craphound.com/

I have just got a LG U990 Viewty mobile phone [1]. It s a 3G phone and came free on the $29 monthly cap plan from Three (minimum monthly spend is $29 - but this is free if you use $29 per month). My previous plan was the $29 cap but had a minimum spend of $20 per month, as I never happened to use less than $29 per month I am not paying more. For a modern mobile phone the actual phone functionality is a sideline. If a device was strictly designed to be a phone then I think it would be very similar to the Nokia phones that were available 5+ years ago - the Nokia I had in 1999 performed every phone function that I desired of it. Like most modern phones the LG U990 Viewty suffers in it s phone functionality from the desire to make it do non-phone tasks and from the desire to cripple it to meet the desires of the carriers (not the desires of the users). For example the home screen will always have at least two buttons for paid Three services and I have no configuration option to remove them. Replacing them with speed dial options for a couple of numbers that I regularly call would be handy. As the main screen is a touch-screen there is no excuse for this, they should allow the software to be reconfigured with more useful options. Eventually the Android will kill most of the other phones and this problem of phones being designed to suit the telephone companies instead of the users will be solved. One of the most annoying mis-features of the phone is that it doesn t properly handle address book entries with multiple phone numbers. One of my friends turns his mobile off when he is at home, so I regularly call his mobile and then immediately call his home number if the mobile is unavailable. With my previous two mobile phones I could press the dial button to bring up the list of previous calls and then use the arrow buttons to select from the other numbers that are attached to the same address-book entry, so with three button presses I would be dialing his other number. With the Viewty I have to go back to the address book. The compelling feature of the Viewty is the camera and display. It has a 5MP camera which makes it the second highest resolution camera-phone offered by Three - the LG Renoir (KC910) has 8MP but needs a $99 plan for it to be free. It also has a 240*320 resolution touch-screen display (in a quick search in January when I bought my phone the best resolution display I could find on a phone is 240*400 in the LG Renoir). While the camera is documented as being 5mp there are no specs available about the resolution of the CCD. I want to use the native resolution of the CCD for pictures (I think interpolation is a waste of space). The CCD might actually be 5mp, a picture of the 1400*1050 resolution screen on my Thinkpad allows me to read all the text even when small fonts are in use, so the CCD resolution must be significantly greater than 1400*1050. This is a really important feature as the Viewty will work well for making screen-shots for bug reports about crashed computers (several of my clients have expressed interest in getting one after seeing such a demonstration). One annoying problem is that the camera software takes a while to load, my trusty Sony digital camera starts a lot faster and the LG U890 phone I used for the past two years is also a lot faster and more convenient. This won t work well for photographing unexpected events. When I am traveling by public transport I will photograph the relevant pages of my street directory as I can zoom in to photos of the maps and read the street names, it saves some weight when traveling. The 2G micro-SD card (which incidentally cost $10 from OfficeWorks) will allow me to store a lot of maps. One interesting feature is the video recording capabilities. It can do 640*480 resolution at 30fps (which is pretty good) and 320*240 resolution at 120fps (they claim that you can film a balloon popping). In my quick tests the standard 640*480*30fps mode works well, but the 120fps mode requires much brighter light than most of my test environments, so I have not yet got it working properly. The phone has a reasonable voice recording function, it can record considerably more than 34 hours of audio and the quality is reasonably good if you use an external microphone. It is however quite poor if you use the built-in microphone for a dictaphone function, it seems that quality is poor at any distance. I had wanted to record my LCA mini-conf talks with my phone but unfortunately forgot to bring the adapter for the external microphone. It s a pity that the phone doesn t have a standard microphone socket as I have misplaced my Viewty microphone, when designing the phone they should assume that misplacing attachments is a common occurrence and design it to use common parts. I recently spoke to a journalist who uses his mobile phone to record interviews. He said that his phone supported phone calls, voice recording, and taking pictures - all the essential tasks for his work. It seems that the Viewty would be better than most phones for journalistic work apart from the issue of low quality voice recording when you have misplaced your external microphone. The text editor is unfriendly in the keyboard mode (I have not tried hand-writing recognition), one thing I don t like is the fact that the letters jump when you press them. This does allow changing a letter by moving the stylus before releasing the press (some people consider this a great feature). There are no cursor control keys (which is a serious omission), and the keyboard doesn t resemble a real keyboard. My iPaQ is far better for writing (I once wrote a full-length magazine article on an iPaQ). The stylus is quite strange and interesting. In the picture of the front of the phone the stylus is compacted with it s lid on. In the picture of the back of the phone the stylus is extended with the lid off. The end of the stylus clips in to the lid so that removing the lid drags the central part out of the body. It s an interesting design and with the string on the lid allows the stylus to be attached to the phone when it s not being used. But I have never used it. Even with an iPaQ (which had a proper stylus that attached firmly inside the body of the device) I often used a fingernail on the touch screen. I have not felt the need to ever use a stylus with my Viewty. One final noteworthy thing is the support for Google services. It seems to have client support for YouTube, Google Maps, GMail, and Blogger. This seems to be a major win for Google, the Viewty is one of the most popular phones at the moment and I expect that lots of people who buy them will now have an incentive to use the Google services. Between these sorts of deals and the Android I think that it will be necessary to have some sort of anti-trust action against Google. Google are generally doing good things for the users. I have been quite satisfied to use Google search, Google advertising on my blog, and Gmail. Also I have been moderately happy with Blogger (it was good when I started blogging) and Google Maps is useful on occasion. So generally I am happy with Google, but monopolies are bad for the users so I think that if things continue on their current trend then Google may have to be split into several little Googlets some time in the next few years.

• [1] http://en.wikipedia.org/wiki/Viewty

TurboGears 2 in debian ... soon ! After a long incubation, a few days ago TurboGears 2 has been released. Historically, I've been preferring TurboGears over Django for being closer to the open source philosophy of reusing existing components. Since the long 2.0 release was marking a gap with Django, I was eager to test the 2.0, and I was delighted to find it in Debian. Unfortunately, it doesn't seem to be in good shape yet. In particular it lacks several dependencies before it can even be used to quickstart a project, in spite of its presence in unstable. To give an idea of the needed work, after having installed it manually (via easy_install), my virgin /usr/local/lib/python2.5/site-packages/ has been polluted by 26 egg-thingies. I decided to spend some week-end time to start closing the gap, because we cannot lack such an important web framework in Debian (and ... erm, yes, also because I need it ). Here is the current status of what has been ITP-ed / packaged already (by yours truly):

• ToscaWidgets (ITP): in experimental/NEW, packaging resurrected from a never-uploaded version
• repoze.tm2 (ITP): in experimental/NEW
• transaction (no ITP) : being packaged by the Zope team (temporary unofficial package available)
• repoze.who (ITP): in experimental/NEW
• repoze.who- friendlyforms,plugins.sa,testutil (ITP): in experimental/NEW
• repoze.what (ITP): in experimental/NEW
• repoze.what- pylons,quickstart,plugins.sql : just ITP-ed

Where to Considering, foolishly, the last package as being already done, the way to go is still long:

    zack@usha:~$ ls /usr/local/lib/python2.5/site-packages/ grep -v repoze.what
    AddOns-0.6-py2.5.egg
    BytecodeAssembler-0.3-py2.5.egg
    Catwalk-2.0.2-py2.5.egg
    easy-install.pth
    Extremes-1.1-py2.5.egg
    PEAK_Rules-0.5a1.dev_r2582-py2.5.egg
    prioritized_methods-0.2.1-py2.5.egg
    sprox-0.5.5-py2.5.egg
    sqlalchemy_migrate-0.5.2-py2.5.egg
    SymbolType-1.0-py2.5.egg
    tg.devtools-2.0-py2.5.egg
    tgext.admin-0.2.4-py2.5.egg
    tgext.crud-0.2.4-py2.5.egg
    TurboGears2-2.0-py2.5.egg
    tw.forms-0.9.3-py2.5.egg
    WebFlash-0.1a9-py2.5.egg
    zope.sqlalchemy-0.4-py2.5.egg

Possibly, some of them are already in Debian hidden somewhere, but sure there is still work to be done. If you want to help, you are more then welcome. The rules are simple: all packages will be maintained under the umbrella of the Python Modules Team, but you should be willing to take responsibility as the primary maintainer. Please get in touch with if you are interested, as I'm in turn already in touch with some other very kind volunteers (thanks Enrico and Federico2!) to coordinate who-is-doing-what. Preview packages available What has already been packaged, including a temporary workaround for python-transaction, will be available in experimental after NEW processing. The idea is that nothing will go to unstable until TurboGears 2 will be (proven to be) fully functional. In the meantime, packages are available from my personal APT repository (signed by my key), here are the friendly /etc/apt/sources.list :

    deb http://people.debian.org/~zack/debian zack-unstable/
    deb-src http://people.debian.org/~zack/debian zack-unstable/

Versions are tilde-friendly and shouldn't get in your way when the official packages will hit unstable. Packaging multiple-egg / multiple-upstream packages In all this, I've faced an interesting problem with the python-repoze. who,what -plugins packages. They correspond to a handful of plugins, each of which is about 20/30 Kb. I didn't consider appropriate to prepare 5 different packages, due to archive bloat potential. Hence, you get the usual problems of multiple-upstream packages. To counter some of them, as well as some egg-specific packaging annoyances with multiple upstream, I wrote a couple of very simple helpers:

• get-orig-source.mk (live version): a debian/rules snippet which relies on several debian/*.watch files which are fed to uscan to implement (not yet completely) the get-orig-source target described in policy
• setup.py (live version): a multiplexer for sub-directories shipping distutils-enabled Python modules. That way you can use /usr/share/cdbs/1/class/python-distutils.mk as usual, and setup.py will take care of the "multiple upstream" problem at build time

I'm no Python-packaging-guru so, lazyweb, if you spot in this choice something I utterly overlooked, or if you have improvements to suggest, please let me know. Repacking eggs An interesting problem will be faced in trying to integrate the above approach with Python modules that are shipped only as .egg files (i.e., no tarballs, ... yes there are such horrifying things out there). To preserve uniformity, we would need uscan to support --repack ing of .egg files as they were simple .zip archives. Since eggs are .zip archives ... why not?

The 28th annual JP Morgan Chase Corporate Challenge race took place a couple of days ago May 21. Participation was down from the record of 23,000 runners set last year at around 17,125. With splendid weather, it is always a nice way to start the Memorial day weekend. We fielded a small but spirited team of nine runners. I finished with a decent (hand-stopped) time of 22 minutes and 27.93 seconds for the 3.5 miles -- or a 6:25 min/mile pace. That is among the fasters times but not quite the fastest compared to the other six times I have run this. Most importantly, everybody seems to have had a blast. And we did set a record for longer post-race party which sets a nice precedent for 2010.

Linux serial ports only work at standard speeds by default. MIDI runs at 31250 baud, which is not a standard speed. However there are tricks to get custom speeds, but documentation is quite fuzzy. This is a simple recipe that worked for me with an FT232 USB-Serial board.

Check the baud base of the device:

$ setserial -g -a /dev/ttyUSB0
/dev/ttyUSB0, Line 0, UART: unknown, Port: 0x0000, IRQ: 0
Baud_base: 24000000, close_delay: 0, divisor: 0
closing_wait: infinite
Flags: spd_normal low_latency

As you can see the baud base is 24000000 here.
Next calculate the divisor by dividing the baud_base you see here by the speed you want.
In my case 24000000/31250=768.

Apply the new setting:

$ setserial -v /dev/ttyUSB0 spd_cust divisor 768

Next start your serial application, you might want to make sure it is already set to the correct speed before you do the above changes else it might destroy your settings. The correct speed is 38400 baud, which is now aliased to 31250.

With minicom just use the menu (CTRL-A Z) to change the settings.

Screen can be used like this:

screen /dev/ttyUSB0 38400

Exit screen by pressing CTRL-A CTRL-\

Sweet.

There are great resources to help tune PostgresQL for performance, but say you've got a Postgres database which suddenly becomes very slow. How can you figure out what's going on?
General system stateA slow database server will typically be running out of:

• cpu,
• memory or
• disk.

So the first thing you should do is to get an overview of the system state using common UNIX tools such as:

• top (press c to see the full command line): look at CPU and memory usage
• iostat -x -m 5: look at i/o wait and service time

If you notice that memory is exhausted, you could play with the following postgresql.conf settings:

shared_buffers
work_mem
effective_cache_size

If you see a lot of I/O, then try adjusting these settings:

checkpoint_segments
autovacuum_naptime
wal_buffers

Finding slow queries
On the database side, start by determining whether:

• there is one large query holding up all of the resources or
• the number of queries is the main problem.
  

To do this, make sure that the following setting is turned on in your postgresql.conf:

stats_command_string = on

Then fire up psql dbname and look at the currently executing queries:

SELECT usename, current_query, query_start
FROM pg_stat_activity;

(You can also add the procid column to the query if you want to map a query to the process ID you see in top.)

If you can't get anything useful out of pg_stat_activity, you may want to enable slow query logging by adding this to your postgresql.conf:

log_min_duration_statement = 100

(All queries which take more than 100 ms to execute will be printed to the main Postgres log file.)
Examining a particular queryOnce you have identified a slow query, you can time it by running this command before executing the query manually:

\timing

To get an idea of where Postgres spends its time when executing that query, look at the query plan:

EXPLAIN your_query;

The numbers you see there are estimates. To run the query and get actual numbers, use this instead:

EXPLAIN ANALYZE your_query;

If you still can't figure out why a certain query takes so long, have a look at the ratio between live and dead rows in the relevant tables:

ANALYZE VERBOSE tablename;

Having too many dead rows is often a sign of insufficient vacuuming. You might want to turn autovacuum on if it isn't already, or to make it a bit more aggressive by tweaking these settings:

autovacuum_vacuum_scale_factor
autovacuum_analyze_scale_factor

Troubleshooting database performance is a bit of a black art, many thanks to Mark Kirkwood for sharing his Postgres wisdom with me.

If there's anything else you have successfully used to find the cause of your performance woes, please feel free to leave a comment.