The QNAP TS-XXX NAS boxes are nice little systems but one thing I really miss is a console. Martin Michlmayr has some instructions (TS-119, TS-219, TS-41x) on his excellent Debian On QNAP pages on how to build a suitable adaptor but sadly even though I've managed to get all the parts I've been too lazy to actually solder the thing together (it doesn't help that I nearly always burn myself when I use a soldering iron!). This became more pressing when someone reported that Debian bug #693263 in qcontrol was not fixed in Wheezy since the issue appeared to be in the initramfs hook. Worse it seemed like something in my proposed fix was causing the system to not boot at all! Having finally found the hardware needed to reproduce the issue my first thought was to try netconsole. To do this I edited /etc/initramfs-tools/modules to add:

      mv643xx_eth
      netconsole netconsole=@<IP>/eth0,@<DST-IP>/<DST-MAC>

The option syntax is described in netconsole.txt but briefly: <IP> is the address of the TS-419P II I'm debugging on, and <DST-IP> and <DST-MAC> are the IP and MAC address of another machine on the network. Having done that, running update-initramfs -u and rebooting I can use netcat -u -l -p 6666 on the other machine to see all the kernel messages. So far so good but this doesn't get me any debugging from the userspace portions of the initramfs. To get those we have to get a bit hacky by editing /usr/share/initramfs-tools/init, first to change:

     # Parse command line options
    -for x in $(cat /proc/cmdline); do
    +for x in $(cat /proc/cmdline) debug; do
            case $x in

and secondly:

             debug)
                     debug=y
                     quiet=n
    -                exec >/run/initramfs/initramfs.debug 2>&1
    +                exec >/dev/kmsg 2>&1
                     set -x
                     ;;

The first of these simulates adding debug to the kernel command line (which can't otherwise easily be edited on these systems) and the second redirects the initramfs process's output to the kernel log. The overall effect is that the output of the initramfs processes appears over netcat.

I started several coding projects today, but none of them reached an announcable conclusion, so this seems like a good day to write up my current discoveries in automated Perl testing modules. This post isn't about test frameworks like Test::More and friends, or even about helper modules (although I do recommend Test::Warn if you want to test warning code in your modules). It's not about writing functionality tests. Rather, it's about various CPAN modules that can be used almost identically in just about every Perl module distribution (and in some other places, like Perl scripts inside a larger package written in another language). You can drop a small driver script into your t directory and (mostly) run make test with no further customization. If you want to see the driver scripts for all of the below that I'm currently using, see the t directory of the Term::ANSIColor distribution. They'll slowly make their way into my other modules, and probably, despite the name of the package, rra-c-util, since that's my equivalent of gnulib and I have tools to synchronize shared code from that package. The driver script generally tests whether the module is installed, skips the test if not, and otherwise dispatches to the "test everything" routine in the module. Test::MinimumVersion: This tests that you're not accidentally using a newer Perl feature than you were expecting. It's very handy if you want to support backward compatibility to a particular version of Perl. It knows about features as far back as 5.004_05. It won't be able to catch everything, but it will catch some of the most obvious things. Test::Perl::Critic: This is sort of the thousand-pound gorilla. Test::Perl::Critic runs all of your code through perlcritic and reports any errors. You will probably need to combine this with a perlcriticrc file in your distribution that you configure it to use, since otherwise the person running the tests may have customized perlcritic. If you let it run perltidy to check for formatting problems, you'll also want a perltidyrc. See the driver script mentioned above for how I set that up. I usually configure this test to only run for me (by checking an environment variable) since the results vary a lot depending on what version of perlcritic is installed, and it's likely new versions will cause the test to start failing. If you use this, expect to need to add ## no critic annotations to your code. Test::Pod: Checks for syntax errors in your POD documentation so that you won't have those embarassing POD ERRORS sections. Note that the latest version of podlators turns syntax errors into fatal errors, so soon you won't have to worry as much about this. But if you'd been using this module all along, you wouldn't have to worry about the fact that pod2man may start faliing due to syntax errors you never fixed! Test::Pod::Coverage: Only of interest to module distributions, not ramdom scripts, this checks that every sub provided by your module has POD documentation. It understands various subs that don't need to be documented (such as the ones you provide to implement a tied class) and lets you mark some subs as not requiring documentation. Test::Pod::Spelling: Checks your POD documentation for spelling mistakes with a spell checker. (Note that perlcritic, and hence Test::Perl::Critic, can also do this, but I use this module separately since I think the output is more readable.) You will almost certainly want to make this test maintainer-only, since spell checkers and spelling dictionaries vary a lot. But it prevents a lot of embarassment when you read your documentation on search.cpan.org and find that obvious spelling error you never saw before. Test::Strict: Checks all Perl code to ensure that it compiles, checks whether it has use strict, and optionally checks whether it has use warnings. This is great; I use this even with packages that just have a few Perl utility scripts in a package otherwise written in C, since I previously kept releasing packages with helper scripts with obvious syntax errors since I didn't run them much. Another place it's useful is if you have internal Debian packages that wrap up Perl utility scripts; just create a small test suite with this module and run it from override_dh_auto_check and you won't ever release a package with scripts that won't run. You can also do test suite coverage testing with this module (and Devel::Cover), but that you'll want to mark maintainer-only since it takes a while. Test::Synopsis: Checks the code in the SYNOPSIS section of any POD documentation to ensure that it compiles. No more syntax errors in your example code in the documentation! All of these modules are, of course, packaged for Debian (thanks largely to the wonderful Perl packaging team!), and otherwise available on CPAN. There are a variety of others, of course; these are just the main ones that I've found personally useful. One final whine, however: all this wonderful Perl testing infrastructure desperately needs a common module and a common interface for providing a list of files to check. Every one of these modules reinvents that wheel, and most of them do it a slightly different way than all the others. Most of them check in either blib or lib (or sometimes both). Most of them also add t. Most of them don't include Makefile.PL, but some of them do. Almost none of them look in examples. Some have a simple way to add more files or directories; some don't. Some can be run from the t directory and still find things, but most of them can't. And of course the syntax is different in every case! Most of the work in my driver scripts is finding the right way to add a few additional directories to the scope of the testing, and in one annoying case (Test::Synopsis), I actually had to use Perl::Critic::Utils (for lack of a better module) to build a list of files to check instead of using the build-in file list, since the build-in file list required all module code be in lib. A module to abstract out all of this and provide a common interface would be easy to write. Most of the work would involve talking to all the authors of all these different modules and asking them to use it. If someone in the Perl community felt like doing that, it would make life so much easier for everyone.

Let s start a convention: bare-metal machines have the linguistic male gender ( der Computer , he needs to be rebooted), whereas VMs have the linguistic female gender ( die virtuelle Maschine , she runs better since the last upgrade of Linux-KVM), and neutral linguistic gender is used when you cannot or do not want or need to make such distinction.
This is, of course, entirely unrelated to human gender, but not unrelated to #debian-68k (on OFTC) discussions ;-) ObRant: DO NOT USE xz COMPRESSION LEVELS ABOVE 6! (For -7 we can make exceptions, for example in Debian *-dbg or *-source packages.) You may use -e if you absolutely need the better compression, but please think of the poor sods who have to create the archives. You must not use the highest compression levels -8 or -9 since they have absolutely insane memory requirements on compression and will still hinder machines with less RAM on decompression. (Using -e only affects CPU usage at compression time; decompression is exactly as fast and memory-consuming as without.) Furthermore, DO NOT CHOOSE A COMPRESSION LEVEL WITH A DICTIONARY SIZE MUCH LARGER THAN THE DATA TO COMPRESS, as that makes absolutely no sense and will rather worsen than improve compression. As a reminder, xz uses the following dictionary sizes:

• 256 KiB at -0 (compresses better than gzip(1) and faster than either gzip(1) or bzip2)
• 1 MiB at -1
• 2 MiB at -2 (compresses better than gzip(1) and bzip2 without losing much speed)
• 4 MiB at -3 and -4 (the difference is in the match finder between these two levels)
• 8 MiB at -5 and -6
• 16 MiB at -7 (186 MiB RAM used to compress a file)
• 32 MiB at -8 (370 MiB RAM used to compress a file)
• 64 MiB at -9 (674 MiB RAM used to compress a file)

Decompression uses less than 1 MiB more than the dictionary size, but the dictionary must always be allocated wholly. (You re fine to use custom presets, but mind the RAM usage!) As a general rule, if you have something of up to 20 MiB to compress, -4 is fine, and -5 will only be better if you have similar data spread across the whole of the file instead of close to each other. When I make mksh distfiles, I instead put files close to each other that have related content, which improves compression much more nicely without penalising low-memory systems; for example, you could put documentation, Makefiles, scripts, m4(1) files, and C source code into groups before archiving, instead of doing it alphabetically. Another note on bzip2: its decompression is slow. I see no reason to use it any more, at all. Use gzip(1) if you care for compatibility or have an issue with xz not having a free copyright licence, and xz otherwise.

Recently, I've started programming in Common Lisp. My idea here is to use cl-irc, an IRC library into an event loop. This can be really useful, for example to trigger action based on time, using timers. Creating a connection The first step is to create a basic cl-irc:connection object on our own. This can be achieved easily with this:
This will return a cl-irc:connection object, logging to stdout (:client-stream t) and having the server name server. Note that the server name could be any string. You probably noticed the ? I used as :network-stream value. This is not a real and working value: this should be a stream established to the IRC server you want to chat with. This is where we'll need to use cl-async:tcp-connect to establish a TCP connection. As you can read in this function's documentation, all we need to pass is the server address, two callbacks for read and general events, and the :stream option to get a stream rather than a socket. So you would do something like:
If you run this program, it will connect to the OFTC IRC server, and then notice you each time the server is sending you a message. Therefore our problem here is how we you treat the message read from the stream in connection-socket-read and handle them in the name of our connection object you used? We can't link both together at this point. We can't build a closure, because as the time we use as:tcp-connect we don't have the cl-irc:connection instance. Also we can't change easily the read-cb parameter of our network-stream established by as:tcp-connect, simply because cl-async doesn't use to do allow that. Building a closure So one solution here is to hack cl-irc:make-connection so we can build an cl-irc:connection instance without providing in advance the network-stream, allowing us to build a closure including the cl-irc:connection to read event for. This is what we're going to do in the connect function.
And here we are! If we run this, we're now using an event loop to run cl-irc. Each time the socket has something to read, the function connection-socket-read will be called on the non-blocking mode socket. If there's no message to be read, then the function will exit and the loop will continue to run. Using timers You can now modify the last line with this:
This will connect to the IRC server, join a channel and then say the same sentence every minute. Challenge accomplished! And I'd like to thank Andrew Lyon, the author of cl-async, who has been incredibly helpful with my recent experimentations in this area.

AreWomenHuman has an interesting article about ViolentAcrez and the wide support for trolling (including by media corporations) [1]. Chrys Stevenson wrote an important article for the ABC about the fundamentalist Christians who are trying to take over the Australian education system [2]. Tavi Gevinson gave an interesting TED talk titled A teen just trying to figure it out about her work starting Rookie magazine and her ideas about feminism [3]. Burt Rutan gave an interesting and inspiring TED talk about the future of space expploration [4]. One of his interesting points is that fun really is defendable in regard to tourism paying for the development of other space industries. Stephen Petranek gave an interesting TED talk about how to prepare for some disasters that could kill a significant portion of the world s population [5]. Some of these are risks of human extinction, we really need to spend some money on it. John Wilbanks gave an intresting TED talk about the way that current informed consent laws prevent large-scale medical research [6]. He says I live in a web world where when you share things beautiful stuff happens, not bad stuff . Joey Hess was interviewed for The Setup and the interview sparked a very interesting Hacker News discussion about workflow for software development [7]. Like most developers I prefer large screens with high resolution, I have an EeePC 701 which works reasonably well for an ultra-portable system but I largely don t use it now I have an Android phone (extremely portable and totally awful input usually beats moderately portable and mostly awful input for me). But Joey s methods are interesting and it seems that for some people different systems give the best result. Jeff Masters gave an insightful TED talk about the weather disasters that may seriously impact the US in the next 30 years [8]. Governments really need to start preparing for such things, some of them are really cheap to mitigate if work is started early. Bryan Stevenson gave an inspiring TED talk about the lack of justice in the US justice system [9]. Wouter Verhelst wrote an insightful article about some of the criticisms of Linux from Windows users [10]. He references a slightly satirical post he previously wrote about why Windows isn t ready for desktop use. Paul Carr wrote an interesting article comparing disruptive business practices of dot-com companies to the more extreme aspects of Ayn Rand s doctrine [11]. In reading some of the links from that article I discovered that Ayn Rand was even more of a sociopath than I had previously realised. Lindy West gave an amazing Back Fence PDX talk about dealing with nasty blog comments from the PUA/MRA communities [12]. After investigating them she just feels sorry for the trolls who s lives suck. Hang from the Vlogbrothers explains gender, sex, sexual orientation, etc [13]. Rick Falkvinge wrote an interesting article about recent political news from Brazil, they had a proposed law that was very positive for liberty on the Internet but it was sabotaged by the media and telcos [14]. We should try to avoid paying any money to the media industry so that they can go away sooner. Amy Cuddy gave an interesting TED talk about body language, power, and the imposter syndrome [15]. Caleb Chung gave an interesting TED talk about toy design which focussed on Pleo a robotic dinosaur with a SD card and USB socket to allow easy reprogramming by the user [16].

• [1] http://tinyurl.com/bswjwkf
• [2] http://www.abc.net.au/religion/articles/2012/10/22/3615647.htm
• [3] http://tinyurl.com/8x3map2
• [4] http://www.ted.com/talks/burt_rutan_sees_the_future_of_space.html
• [5] http://www.ted.com/talks/stephen_petranek_counts_down_to_armageddon.html
• [6] http://www.ted.com/talks/john_wilbanks_let_s_pool_our_medical_data.html
• [7] http://joeyh.name/blog/entry/out_and_about/
• [8] http://tinyurl.com/ctnunyd
• [9] http://www.ted.com/talks/bryan_stevenson_we_need_to_talk_about_an_injustice.html
• [10] http://grep.be/blog/en/computer/ui/windows_desktop_followup
• [11] http://pandodaily.com/2012/10/24/travis-shrugged/
• [12] http://vimeo.com/50006487
• [13] http://www.youtube.com/watch?v=xXAoG8vAyzI
• [14] http://tinyurl.com/a62ruf3
• [15] http://tinyurl.com/8lyyqmm
• [16] http://www.ted.com/talks/caleb_chung_plays_with_pleo.html

If you rip your DVD's with a unix-y name like longish-show-name-season-01-01.m4v then the iPad and iPhone video player only shows the prefix of the file-name to it's string limit, so you only ever see longish-show-na and can never tell what episode is what in the overview list. You can edit the files in iTunes to add meta-data information, but that's annoying and non-unix-y. It is of course easy, but I went down several dead-ends with tools that didn't recognise file-extensions, corrupted files and plain didn't work. The package I had most success with is mp4v2-utils and the mp4tags utility. The help is fairly self-explanatory, but something like

mp4tags -show "Show Name" -season <season> -episode <episode> -type "tvshow"  file.m4v

will mean that when you import your file into iTunes it displays correctly as a TV show (and not a movie), is grouped correctly into series and has a name you can see. Unfortunately, it doesn't seem my Nexus 7 tablet picks up these tags. And both players interfaces remain terrible at indicating which episodes have been watched and which have not. But this has allowed me to re-visit my old box-set of Yes Minister which, despite a few odd references to Quango's still seems as relevant as ever!

In 2011, I wrote about how small files could consume a lot of space. I meant to do a follow-up on the savings but I forgot about it until now. In 2.5.7, we started compressing some of the collected data files. Some of these are ridiculously compressable (#664794). Even better, compressing them is sometimes faster than writing them directly to the disk, so in some cases it is a pure win/win. For lintian.d.o, we also see a vast size reduction in overall size of the laboratory. I have taken a few samples occasionally. The samples were done with du(1):
$ du -csh [--apparent-size] laboratory/*


Version/date	du -csh	apparent-size
N/A around 20 Mar 2012 (#664794)	16G	13G
2.5.6 (Fri Apr 27 2012)	14GB	N/A
2.5.6 (Mon Jun 04 2012))	N/A	12G
2.5.10.2 (Fri Sep 21 2012)	12G	8.3G
2.5.11 (Wed Jan 2 2013)	10G	6.1G

And the most awesome part of this? The comparison is quite biased against the 2.5.11 entry, which is the only entry to also process experimental (approx. 10% extra packages). Some of the early entries (2.5.6 and older ) might also have suffered from the too many links issue[1]. I only wish I had been better at collecting data points, so I could have made a proper graph of it. It sounds almost too good to be true, but if you look at the size of one of the linux-image packages[1], the space usage dropped from 27M to 15M between 2.5.5 to 2.5.9. Currently it is squeezed down to 14M (tested with head of the git master branch). [1] I believe is about 5-10% less binary packages processed for those runs. [2] linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb

podlators is the package that contains Pod::Man and Pod::Text and the pod2man and pod2text driver scripts, which convert Perl's POD documentation format into text and *roff (for man pages) output. This release is a "clearing the decks" release that rolls up various smallish feature requests and bug fixes that have been pending for a while. I want to do the same reworking to podlators that I did to Term::ANSIColor, but it's going to take longer since it's quite a bit more code. (And Term::ANSIColor took me about two days.) I didn't want to block the various pending issues on that work, so this tries to clean up all the open bugs I was aware of other than sorting out the defaults for Unicode output. (That's a much larger problem and will probably be fixed in a major release.) The primary change is that pod2man and pod2text now die by default if the POD contains syntax errors. This will hopefully flush out numerous POD documents around the world that have been carrying POD ERRORS sections around forever. I considered just warning by default but not failing, since this will also cause CPAN builds to fail, and I may fall back to that if there are too many complaints. But usually fixing the POD syntax is straightforward, and normally parsing programs will fail on syntax errors. Note that the defaults for the modules, Pod::Man and Pod::Text, continue to be to add a POD ERRORS section, since aborting inside a module isn't very friendly to the caller. There is a new option for both the modules and the driver scripts to set the error handling behavior, which can be set to die, report to standard error but continue, add a POD ERRORS section, or ignore errors entirely. Also in this release is a new nourls option that suppresses the URLs in text and *roff output for L<> codes that have anchor text, which can declutter some POD texts, and fixes for X<> formatting codes containing whitespace and small-caps handling of paragraphs consisting entirely of all-caps text. You can get the latest release from the podlators distribution page.

For the year of 2012, I finished and reviewed 60 books, the same number as 2011. Given how stressful and chaotic much of the year was, I count this a major triumph. The hardest part was not the reading but the review writing; the end of the year required a significant push to finish writing reviews of all the books I'd read that year, and at one point I was reviewing things I'd read more than two months earlier. But I can enter 2013 entirely caught-up. This continues to feel like about the right pace, striking a balance between reading enough that I can pursue multiple reading goals at the same time, while leaving enough time for other projects and video games. I gave three novels a 10 out of 10 this year, but the stand-out even among that group was Elizabeth Wein's Code Name Verity. Not only was it my favorite book of the year, but it was one of the best books I've ever read. The other two 10-rated books are also highly recommended, of course: Matt Ruff's fascinating novel of multiple personalities, Set This House in Order; and C.J. Cherryh's SF classic, Cyteen. The latter was a re-read in advance of reading the sequel, Regenesis, which I also recommend if you've read and liked the last third of Cyteen. Other fiction highlights of the year were China Mi ville's Embassytown and Suzanne Collins's The Hunger Games triology, particularly the third book, Mockingjay. Embassytown continues the trend from Mi ville's The City & The City of tighter, faster-moving novels while moving into space opera and first contact territory. It keeps Mi ville in the top rank of current SFF writers, despite having some suspension of disbelief problems. Collins's world-building in The Hunger Games also has suspension of disbelief problems, but I understood why the series was so popular. I was caught by surprise by its look at violence and its after-effects and was particularly impressed by Mockingjay and Collins's chosen ending. I think this is a popular series that lives up to the hype. There were no 10 ratings in non-fiction this year, but several books nonetheless stood out. John Kenneth Galbraith's The Affluent Society is a thought-provoking book that questions the foundation of a consumer-oriented economy while perceptively pointing out how it distorts choices away from public goods. Susan Cain's Quiet is a passionate defense of introversion in an extroverted world. And, finally, Joshua Bloch's Effective Java and Damian Conway's Perl Best Practices are both insightful looks at the good and bad of their respective languages and taught me a great deal as a practicing programmer. One final highlight to mention: Eclipse Phase by Posthuman Studios is an excellent RPG sourcebook, at least from the perspective of interesting world-building. I haven't played it, but I thoroughly enjoyed reading it (and have acquired nearly all of the supplements). The full analysis includes some additional personal reading statistics, probably only of interest to me.

Choices, options, selections if you rather. These are what set us all apart from our fellow man, perhaps it is only the appearance of free will and individuality but our world is full of choice.

In software it might be argued programming is nothing more than making thousands perhaps millions of choices which result in a list of instructions for a machine incapable of making decisions of its own.

On a more mundane level sometimes a programmer cannot make a choice suitable for all expected use cases and a user option is created. These options have become something of a cause for arguments within certain sections of the open source software community especially amongst the groups that influence the graphical user experience.

The discussion should be nuanced and varied (perhaps that is my age, or maybe I am more diplomatic than I thought?) but there seems to be little compromise on this discussion which (from an outsiders point of view) splits into two viewpoints:

• On one side of the argument, which I shall label reductionist, is that all options should be removed with the software just doing the correct thing
• On the other, whom I shall refer to as maximalist, is that users should be presented with options to customise everything.

The reductionist group is currently winning the argument in the popular graphical environments and seem to be removing functionality which requires user choice wherever they can find it. This results in the absurd "joke" that the UI will become a single button, which they are trying to remove.

Personally my view is that an option should be present only when an option cannot be satisfactorily selected by the computer and even then a default suitable for as many users as possible should be picked.

You may ask why I have raised this topic at all? well over the last few days I have been trying to fix the preferences selection for the GTK port of NetSurf. The NetSurf project follows my view on this subject pretty closely but being a browser its very difficult to do the right thing for everyone

NetSurf has numerous frontends for the core browser, I use the term frontend because in some places the toolkit and OS are conflated (windows, cocoa) and not in others (gtk). For each frontend NetSurf is a native application, this is an important distinction, the windows and widgets a user interfaces with are produced by that platforms toolkit.

Old NetSurf Preferences Dialog

This is a deliberate choice unlike other browsers which render their UI themselves as web content, this is a beguiling solution when the authors wish there to be a single browsing "experience" with a common look and feel. However NetSurf looks and feels like a native browser for each frontend, which is what the project decided it wanted to achieve.

Given this the gtk frontend (is it a Gnome application? not sure of the distinction TBH) has had no dedicated maintainer for some time it has suffered from a bad case of bitrot from both GTK library changes and general neglect.

I have slowly been improving this situation, the browser can now be compiled with GTK version 2 (2.12 onwards) or 3, the menus and other UI elements have been translated for the supported languages and now the turn has come for the options dialog.

Oh, right, its "Preferences" not options, fair enough a common semantic throughout all applications does give a degree of homogeneousness but the word choice does seem to indicate the users control has been reduced. At this point I gained an education in just how unfriendly GTK and its tools have become towards the casual programmer.

The dialog I wanted to construct was a pretty standard tabbed layout which reflected the current options and allowed the user to change them in an obvious way. Given that I have constructed and equivalent interface in idiomatic manner for cocoa and windows I thought this would be straightforward, I was very wrong.

The interface construction tool is called glade which used to be the name of the UI "resource" file and interfaces. However the tool is still called glade but the interface is now GtkBuilder which has a different (but similar) XML based file format. Then we discover that despite it being an extensible file format the UI files are specifically versioned against the GTK library. Also why on earth can these resources not be compiled into my program? OK make them over-ridable perhaps. Generally it is yet another file to distribute and update in step with the executable.

New NetSurf Preferences

So because I want to support back to version 2.12 of GTK I do not get to use any of the features from 2.16 in my UI unless I load a different UI builder file...oh and GTK3? requires a completely different version of glade and the UI files which are incompatible with gtk 2. Once this was worked round with having multiple UI files I then moved on to the next issue.

The GTK model uses function callbacks on events, these are known as signals. Perfectly reasonable but because the ui files are loaded at runtime and not compiled in there must be a way for the GTKBuilder to map the textural signal names to function pointers.

The way the GTK developers have chosen to do this is to search the global function table which means any signal handler function symbol has also to be global which means unnecessary overhead for the ELF loader increasing load times.

This lookup should have been confined to a single object or even placed in an alternative section to avoid these issues, this would not have seemed especially challenging to implement as all callback handlers have to use a preprocessor to define already (G_MODULE_EXPORT).

Another thing that makes developing GTK applications worse is the documentation, this seems to be a continuous complaint about may open source projects, which is taciturn or simply missing in many cases. GTK seems to suffer dreadfully from having multiple API versions all subtly different resulting in a lot of work on the developers part to simply find what they want.

A specific example of this is the signals and under what circumstance they occur I wanted to update all the widgets with the current configuration state whenever the options window is shown (using gtk_widget_show()) so I figured that would be the "show" signal...right? nope, never did find the right one to use, ended up with "realize" which occurs once when the dialog is created, not what I wanted, but is at least consistent and works.

NetSurf with Preferences Dialog

Overall my impression of developing just one small dialog (60 widgets total exclusing labels and containers) for a GTK program is that the toolkit and its tooling is missing the facilities to make developers life easier when doing the drudge work that a good proportion of graphical program development.

It is not the case that one cannot do things, just that everything has to be done manually instead of having the tools do the work, and because of the documentation that tedium is magnified.

I did eventually reach the stage that the thought of writing the boilerplate to add another check button to enable "do not track" had me thinking "do they really need this or can I avoid the work?" perhaps that is why they are all reductionist?

Review: City of Diamond, by Jane Emerson

Publisher:	DAW
Copyright:	March 1996
ISBN:	0-88677-704-6
Format:	Mass market
Pages:	624

The setting is deep into the star-faring future of humanity. Humans have settled many other worlds, thanks largely to technology from a race named the Curosa. There is a thriving network of settlement based around a sort of warp conduit network. But, standing apart from most human settlements, are three huge ships: the City of Opal, the City of Pearl, and the City of Diamond. Their inhabitants are devout members of the Redemptionist faith, a merging of Christian (primarily Catholic) belief and Curosa religion. Their ships are gifts from the now-departed Curosa, featuring drives unknown to the rest of humanity that allows them to ignore the normal transport network. They are meant to witness and spread their religion, but now, six centuries later, they primarily trade. Much of this is not apparent at the start. City of Diamond opens with Adrian Mercati, chosen heir of the current dying Protector, dodging people in what seems to be an oddly-organized city. Impudent, occasionally flippant, utterly charming, and in love with danger, he inherits the position of Protector shortly after we meet him. Emerson slowly reveals more of the nature of the ship while Adrian is shocking (and charming) the nobility and the church, and consorting with demons. Or, at least, one demon: Tal, an entirely human-appearing creature except for his eyes, who the Redemptionists claim is born without a soul. Slowly, it becomes clear that he's actually the offspring of an alien and a human, a mating of species whose children cannot socialize as human and behave in ways that seem like human sociopaths. Tal is no exception, but his interests align with Adrian's for the time. To those two viewpoint characters, Emerson adds four more: a low-rank administrator named Spider who works for Tal after Tal saved him from death by recycling, Adrian's politically-arranged bride-to-be from the rival (and even more religious) ship City of Opal, a guard from a French-descended slum on City of Opal, and a dangerously honorable and literal mercenary from an off-shoot culture of humanity that's now only a legend. This is a book that tends to sprawl. There's quite a lot going on, starting with the negotiations between City of Diamond and City of Opal to put to rest the remnants of a civil war and leading to the search for a legendary Curosa artifact on the planet they're both visiting. All of the characters have their own agendas, their own history, and their own goals, and are pursuing them in parallel and occasionally in opposition. Emerson takes the time to let all of this build. Despite that, there is very little in this book that drags or bores, and that's due primarily to the strength of the characters. Every one of the viewpoint characters is unique, bringing a different perspective to the story, and every one of them changes and grow over the course of the story. They also play off of each other in delightful ways. This is a book that makes full use of the fact that three characters make seven character interactions. For example, Tal by himself is logical but just slightly askew, Tal seen by Spider is uncomfortable and worrisome, Tal and Adrian is a dance of mutual amusement and respect, and Tal and Keylinn finding points of commonality is just delightful. I did wish that there was a bit more Keylinn and Tal and a bit less of Spider, but once Emerson built her world, introduced her characters, and got the story moving, there aren't many weak pieces. Part of the fun is figuring out the world background. There are almost no infodumps here; Emerson just starts telling stories of characters in the surroundings in which they find themselves, and the reader is left to piece things together from background description, the occasional explanation from one character to another, and parallels with long-standing SF tropes. It's not a puzzle book, but it is a book of where the reader is dropped as an alien into a fully-developed culture. In both that way and in the use of nobility, religion, and formal hierarchy against a starfaring backdrop, City of Diamond reminds me of the sort of classic space opera that led up to Star Wars but isn't seen as much these days. It's also funny. I was unsurprised to discover that Emerson is a pseudonym for Doris Egan, who (in addition to writing some other SF) is an occasional writer and producer for House. The humor isn't entirely obvious at the start, although Adrian frequently provokes a grin from the start of the book, but Tal's perceptive but obliviously blunt comments become more and more fun as the book progresses, and both Iolanthe (Adrian's bride) and Keylinn (the mercenary, sort of) are a delight. I want to see more of both of them, but sadly the book ends just as they (particularly Iolanthe) are hitting their strides. And that's the biggest problem with City of Diamond: it's a failed (or at least postponed) trilogy. As with Walter Jon Williams's Metropolitan and City on Fire, there were supposed to be more books, but apparently either the publisher lost interest or other projects took precedence. The story does come to something of a conclusion, but most of the major story arcs aren't resolved. Emerson is still bringing more guns onto the set late in the book. None of the villains truly get their just desserts, nor do most of the character arcs fully resolve, although friendships and more have formed satisfyingly by the end of the book. There is obviously far more to both these characters and this universe, including an entire ship that was never visited and an alien whose presence should prove startling. This is a great book: warm, funny, full of interesting characters, and with a nicely complex background universe. If it were complete in itself, or if the sequels had materialized, I would recommend it unreservedly. As is, you will have to weigh your dislike of unfinished stories against its stand-alone merits. But I enjoyed it a great deal. Rating: 7 out of 10

• Facebook Like
• Google Plus One
• Linkedin Share Button
• Tweet Widget

A while ago I wrote a blog post debunking the idea that swap should be twice as large as RAM [1]. The issue of swap size has come up for discussion in a mailing list so it seems that it s worth another blog post. Swap Size In my previous post I suggested making swap space equal to RAM size for systems with less than 1G of RAM and half RAM size for systems with 2-4G of RAM with a maximum of 2G for any system. Generally it s better to have the kernel Oom handler kill a memory hungry process than to have the entire system go so slow that a hardware reset is needed. I wrote memlockd to lock the important system programs and the shared objects and data files they use into RAM to make it easier to recover from some bad paging situations [2], but even that won t always make it possible to login to a thrashing system in a reasonable amount of time. But it s not always good to reduce swap size. Sometimes if you don t have enough swap then performance sucks in situations where it would be OK if there was more swap. One factor in this regard is that pages mapped read-only from files are discarded when there is great memory pressure and no swap space to page-out the read-write pages. This can mean that you get thrashing among memory for executables and shared objects while there s lots of unused data pages in RAM that can t be paged out. One noteworthy corner case in this regard is Chromium which seems to take a lot of RAM if you have many tabs open for a long time. Another factor is that there is a reasonable amount of memory allocated which will almost never get used (EG all the X based workstations which have 6 gettys running on virtual consoles which will almost never be used). So a system which has a large amount of RAM relative to it s use (EG a single-user workstation with 8G of RAM) can still benefit from having a small amount of swap to allow larger caches. One workstation I run has 8G of RAM for a single user and typically has about 200M of the 1G swap space in use. It doesn t have enough swap to make much difference if really memory hungry programs are run, but having 4G of memory for cache instead of 3.8G might make a difference to system performance. So even systems which can run without using any swap can still be expected to give better performance if there is some swap space for programs that are very inactive. I have some systems with as much 4G of swap, but they are for corner cases where swap is allocated but there isn t a lot of paging going on. For example I have a workstation with 3G of RAM and 4G of swap for the benefit of Chromium. Hardware Developments Disk IO performance hasn t increased much when compared to increases in RAM size over the last ~15 years. A low-end 1988 era hard drive could sustain 512KB/s contiguous transfer rates and had random access times of about 28ms. A modern SATA disk will have contiguous transfer rates getting close to 200MB/s and random access times less than 4ms. So we are looking at about 400* the contiguous performance and about 10* the random access since 1988. In 1988 2MB was a lot of RAM for a PC, now no-one would buy a new PC with less than 4G so we are looking at something like 2000* the RAM size in the same period. When comparing with 1993 (when I first ran a Linux server 24*7) it was 4M of RAM, maybe 15ms random access, and maybe 1MB/s contiguous IO approximately double everything I had in 1988. But I ll use the numbers from 1988 because I m more certain of them even though I never ran an OS capable of proper memory management on the 1988 PC. In the most unrealistically ideal situation paging IO will be a factor of 5* more expensive now relatively than it was in 1988 (RAM size increase by a factor of 2000 divided by contiguous IO performance increase by a factor of 400). But in a more realistic situation (random IO) it will be 200* more expensive relatively than it was. In the early 90s a Linux system with swap use equal or greater than RAM could perform well (my recollection is that a system with 4M of RAM and 4M of active swap performed well and the same system with 8M of active swap was usable). But nowadays there s no chance of a system that has 4G of RAM and 4G of swap in active use being usable in any way unless you have some corner case of an application allocating RAM and not using it much. Note that 4G of RAM doesn t make a big system by today s standards, so it might be reasonable to compare 2M in 1988 to 8G or 16G today. But I think that assuming 4G of RAM for the purpose of comparison makes my point. Using Lots of Swap It is probably possible to have some background Chromium tabs using 4G of paged out RAM with good system performance. I ve had a system use 2G of swap for Chromium and perform well overall. There are surely many other ways that a user can launch processes that use a lot of memory and not use them actively. But this will probably require that the user know how the system works and alter their usage patterns. If you have a lot of swap in use and one application starts to run slowly you really don t want to flip to another application which would only make things worse. If you use tmpfs for /tmp then you need enough swap to store everything that is written to /tmp. There is no real upper limit to how much data that could be apart from the fact that applications and users generally don t want to write big data files that disappear on reboot. I generally only use a tmpfs for /tmp on servers. Programs that run on servers tend not to store a large amount of data in /tmp and that data is usually very temporary, unlike workstation users who store large video files in /tmp and leave them there until the next reboot. Are there any other common corner cases that allow using a lot of swap space without serious performance problems? My Swap Space Allocation When I setup Linux workstations for other people I generally use 1G of swap. For a system that often has multiple users (via the user-switch feature) I use 2G of swap if it only has 3G of RAM (I run many systems which are limited to 3G of RAM by their chipset). For systems that I personally use I generally allocate 4G of swap, this is to cater to excessive RAM use by Chromium and also because I test out random memory hungry programs and can t easily predict how much swap space will be needed a typical end-user desktop system is a lot more predictable than a workstation used for software development. For servers I generally allocate 512M of swap space. That s enough to page out things that aren t used and make space for cache memory. If that turns out to be inadequate then it s probably best to just let the watchdog daemon reboot the system.

• [1] http://etbe.coker.com.au/2007/09/28/swap-space/
• [2] http://doc.coker.com.au/projects/memlockd/

For some time, I have been concerned about the amount of data I store on Google. This post aims to show alternatives to the service. This is primarily for two reasons. The first is simple. I would like to avoid a single point of failure, and the risk that I could be suspended for a number of months with no recourse. The second if more political. I'm worried about the way that certain governments can subpoena information which they may not be able to collect directly. Now, don't get me wrong. Google is good at transparency. They also provide a very useful service that works incredibly well. In fact, the quality of the integration and services provided is the primary reason I have not changed previously. Calendaring is something that I naturally feel shouldn't be hard, but somehow is. Google Calendaring service is rather good, and far outstrips anything open source I've found so far. However, I've managed to migrate to Zoho which offers a very similar level of functionality. It's also very easy to extract your entire calendar from Google, which makes the transition easier. Reader is a service provided to subscribe and read RSS feeds. In the past, I've used Gregarius as a self hosted solution, but this seems to have disappeared recently. I briefly flirted with NetVibes, which works well once you turn off the frustrating 'widget' mode, but has completely failed to address the need to have native tablet apps. In the end, I settled for NewsBlur which also has the clever use of hotkeys for navigation which I found essential in Google's products. Picasa has stood me well for hosting photos for some time. And photo hosting is something I have quite a large amount of professional experience with :) Despite having previously been in competition with them, I can heartily recommend Flickr, even with it's dogmatic reliance to creating a Yahoo! email address for each account. Search still has some way to go to find a viable replacement. I've tried DuckDuckGo, but found it lacking. Much like the mistake that Apple seem to have made with the launch of Apple Maps, there seems to be a misunderstanding that search is actually HARD.
There are two camps when it comes to searching. You can be dumb, or you can be smart. By being dumb, the user can provide enough information to help narrow the results down to a single (or at least manageable) result set. This works fine, and is a traditional model. The opposite is to be smart. Simply get the results right, using a variety of metrics to work out what the user wanted in the first place. This is obviously more difficult, but infinitely more valuable, and is something that Google excels at. The problem lies when you fall in between these stalls. Unfortunately, it seems that there is no viable alternative to the big G at the moment.

Courtesy of a recent Google+ post/Stack overflow answer, I stumbled upon ghc-gc-tune, a simple but nice tool which generates interesting graphs for Haskell programs. What is does is quite trivial: iterate over a range of arena and heap sizes, and run a specified program with those RTS options, then generate a graph comparing the performance (by default cpu time) across the combinations of the values. Note that for newer GHC versions, you'll need to link with -rtsopts to allow for the -A/-H custom sizes. The reason I mention it is that the graphs it generates can be quite interesting. For one of the Ganeti programs, hspace, run with the command line ./hspace --simu p,20,1t,96g,16,1 --disk-template drbd, it generates this graph: This picture, if I read it correctly, says that this program is actually well behaved (well, +RTS -s says "3 MB total memory in use" with default options), and that the optimum sizes actually relate to the (L1? L2?) cache size. But note that the maximum difference is only about ~1.6 . By changing the parameters to hspace and make it allocate more memory (./hspace --simu p,20,1t,128g,256,32 --disk-template plain --tiered=1g,128m,1, ~52MB reported by +RTS -s), the graph changes significantly: Now we have somewhat the opposite situation: very small arena sizes are detrimental (and by a big factor, 4.5 ), large arena/heap sizes are OK-ish, and the sweet spot is around 2-4MB arena size with heap sizes up to 4MB. Now maybe these particular examples were not very elightening (and they were definitely not well-conducted tests, etc.), but they should allow some intuition into how the program behaves. Plus, the tool can also generate other plots, for example peak memory usage.

Term::ANSIColor is a Perl core module that provides a few different interfaces to get at the ANSI color and text attribute escape sequences. It can be used to, for example, print out bold text or colorize output. This is a major new feature release, incorporating patches sent to me by Kurt Starsinic and Stephen Thirlwall. Kurt Starsinic contributed support for 256-color emulators. These hang special extended sequences off the otherwise-unused 38 and 48 sequences, providing an alternative naming of the base 16 colors, 216 colors arranged in an RGB namespace, and 24 shades of grey. Those colors are now available via all the normal Term::ANSIColor interfaces. He also contributed a program to print out test data for 256-color emulators that I enhanced to print out test data for the basic and 16-color sets as well, which allowed me to get rid of the static test files in the distribution. Stephen Thirlwall contributed support for creating aliases for the standard colors via an environment variable. I extended his patch to also provide a coloralias() function interface. This feature probably won't be that widely used, but it allows a user to set up custom color names for applications that take color names from user configuration (which might be handy for doing things like using the Solarized color scheme), or to define an alias like "alert" or "warning" in one place and then use it throughout the code. With the new 256-color support, it may be useful to set up more human-readable aliases for some of the RGB colors. Since I was working on this project anyway, I also redid the whole module in the new Perl coding style that I've been experimenting, based on Damian Conway's Perl Best Practices, and added a ton of new tests. There are a lot of great CPAN modules out there to do automated tests of various aspects of Perl code, and most of them don't require much effort to use. I'll probably write a separate post about that later. That sort of comprehensive review of course uncovered a bunch of microbugs and lack of clarity in the code, which is now hopefully much improved. I'm pleased to report that the test suite achieves 100% code coverage as reported by Devel::Cover (and in fact there's a maintainer-only test to ensure that it stays that way). You can get the latest version from the Term::ANSIColor distribution page.

I don t like watching videos sitting in front of a computer, I usually store them into a USB key to plug into my Samsung LE37 TV set. It s not an easy task, though, as I have to fight with other components of my family who usually like watching something completely different than DebConf videos I could have invested 200 to buy a new one, but I tried a different approach first. Recycling a netbook I received a Tweety device as Christmas gift, and I tried to connect it to my netbook to see how it worked. I was quite impressed by the good quality of the sound (if compared to the really poor quality of the netbook speakers), and I realized I could have recycled the netbook as a brand new TV set with some little adjustments. I plugged my SyncMaster 920n PC monitor into the netbook VGA port, switched Totem to fullscreen mode, and here it is my TV set! Remote control OK, that was quite straightforward to accomplish. A real TV set has a remote control too, but my netbook is not equipped with an IrDA port, so I had to think about a different solution. My BlackBerry is equipped with a cool app called BBSSH, which allows you to connect to any SSH server, this could be a good starting point to implement a remote control device. Then, I had to write an interface to Totem using Python and DBus binding to allow my SSH remote control to issue commands to start and pause the player. I published a first working implementation: at the moment the only supported operations are play, pause, stop and volume adjustments, but there are a lot of opportunities by implementing full MPRIS specification. Just to avoid typing a long command each time, I finally created a set of bash aliases, one for each command, so I just have to type one letter to have the desired function, as much as a real remote control

I recently had a chance to briefly play with the new Samsung Galaxy Camera [1]. The Galaxy Camera is an Android device with a 4.8 display (the same size as the Samsung Galaxy S3) that has a fairly capable camera (IE nothing like a typical phone camera). It runs Android 4.1 (Jelly Bean) and the camera has 21* zoom with a 16 megapixel sensor. Camera Features It seems that professional photographers are often annoyed when they see someone with a DSLR set in auto mode. It s widely regarded that auto mode is a waste of a good camera, although the better lenses used with DSLRs will usually give a better result than any compact camera even when it s in auto mode. The problem is that photography is quite complex, in an earlier post about digital cameras I summarised some of the technical issues related to cameras and even without any great detail it became a complex issue [2]. The Galaxy Camera has a reasonably friendly GUI for changing camera settings which even includes help on some of the terms, I expect that most people who use it will end up using most of the features which could make it a good training camera for someone who is going to move to a DSLR. A DSLR version of the Galaxy Camera could also be an interesting product. The camera also has modes such as Waterfall and Panorama , hopefully the settings for those would be exposed to the user so they could devise their own similar groups of settings. I ve seen the phone criticised for the lack of physical controls as the expert mode in software is inherently slower than manually turning dials on a DSLR. But it seems obvious to me that anyone who knows how to use the controls manually should be using a DSLR or bridge camera and anyone who doesn t already know how to do such things will be better suited by the software controls. It supports 120fps video at 720*480 resolution (with a file format stating that it s 30fps to give 1/4 speed) which could be useful. I used to have a LG Viewty smart-phone that did 120fps video but the resolution was too low to be useful. 720*480 is enough resolution to see some detail and has the potential for some interesting video, one use that I ve heard of is filming athletes to allow them to analyse their performance in slow motion. It also does 60fps video at 720p (1280*720) resolution. One down-side to the device is that the lens cover doesn t seem particularly sturdy. It s quite OK for a device that will be stored in a camera case but not so good for a device that will be used as a tablet. I didn t get to poke at the lens cover (people don t like it if you mess with their Christmas presents) but it s design is a couple of thin flaps that automatically retract when the camera is enabled which looks quite weak. I d like to see something solid which doesn t look like it will slide back if the device is treated as roughly as a phone. I think that the lack of a solid lens cover could be the one factor that prevents it from being used as a replacement for a smart phone. Apart from that a Galaxy Camera and a cheap GSM phone could perform all the functions of a high end phone such as the Galaxy S3 while also producing great pictures. It would probably make sense for retailers to bundle a cheap phone with a Galaxy Camera for this purpose. Tablet Features The device boasts WiFi Direct to allow multiple cameras and phones to talk to each other without a regular WiFi access point [3]. I didn t test this and I don t think it would be particularly useful to me, but it seems like a handy feature for less technical users. It can connect to the Internet via Wifi or 3G, supports automatic upload of pictures (it comes with Dropbox support by default like the Galaxy S3), and has a suite of photo and video editing software. I don t expect that any photo editing software that runs on an Android device would be much good (I think that you really need fine cursor control with a mouse and a high resolution screen), but it would probably be handy for sending out a first draft of photos. Most Android apps should just work, the exceptions being apps that rely on a camera that faces the user or full phone functionality. So the Galaxy Camera can do almost anything that an Android phone or tablet can do. Value The RRP for the Galaxy Camera is $599, that puts it in the same price range as a DSLR with a single lens. While that s not a bad price when compared to smart-phones (it s cheaper than the LTE version of the Galaxy S3 phone) it s still quite expensive for a camera that s not a DSLR. Fortunately Kogan is selling it for $469 and has free shipping at the moment [4]. This still makes it more expensive than some of Bridge Cameras which probably have significantly better optical features, but in terms of what the typical user can do with a camera the Galaxy Camera will probably give a much better result. The sensor in the Galaxy Camera is smaller than that in the Nokia 808 PureView [5] (1/2.3 vs 1/1.2 ) so the Nokia PureView should be able to take better pictures in some situations. Unfortunately the Nokia 808 doesn t run Android, I d probably own one if it did. Some of the reviews are rather harsh, the Verge has a harsh but fair review by Aaron Souppouris which makes a number of negative comparisons to cheaper cameras [6]. I really recommend reading Aaron s review as there s a lot of good information there. But I think that Aaron is missing some things, for example he criticises the inclusion of ebook software by saying that he wouldn t read a book on a camera. But the device is a small tablet computer which also has a compact camera included. I can easily imagine someone reading a book or playing Angry Birds on their camera/tablet while in transit to where they are will photograph something. I can also imagine a Galaxy Camera being a valuable tool for a journalist who wants to be able to write short articles and upload pictures and video when out of the office. Aaron concludes by suggesting that the Galaxy Camera is a $200 camera with $300 of editing features. I think of it as $200 in camera hardware with software that allows less skilled users to take full advantage of the hardware and the ability to do all the software/Internet things that you would do on a $450+ smart-phone. Would I Buy One? No. The Galaxy Camera is among other things a great toy, I d love to have one to play with but I can t spare $469 on one. Part of the reason for this is that my wife just bought a DSLR and is getting lessons from a professional photographer, so I really won t get better pictures from a Galaxy Camera. The DSLR on auto mode will allow me to take pictures that will usually be better than a Galaxy Camera can achieve (sometimes you just can t beat a good lens). For more demanding pictures my wife can tweak the DSLR. The 120fps video is a really nice feature, I don t know if my wife s DSLR can do that, but it s a toy feature not something I really need. I ve just bought a Galaxy S3 which is a great little tablet computer (most of the time it won t be used for phone calls). I don t need another 4.8 tablet so a significant part of the use of the Galaxy Camera doesn t apply to me. I recommend the Galaxy Camera to anyone who wants to take good photos but can t get a DSLR and lessons on how to use it properly. But if you would rather get a 35mm camera with interchangeable lenses that runs Android then it might be worth waiting. I expect that the Galaxy Camera will be a great success in the market (it s something you will love when you see it). That will drive the development of similar products, if Samsung doesn t release a 35mm Android camera soon then someone else will (for example Sony develops both high end cameras and Android phones). If my wife didn t have a DSLR then I d probably have bought a Galaxy Camera already. I will recommend it to my parents and many other people I know who want an OK camera and can benefit from a tablet, but don t know how to use a DSLR properly (or don t want to carry a bulky camera).

• [1] http://en.wikipedia.org/wiki/Galaxy_Camera
• [2] http://etbe.coker.com.au/2011/08/24/digital-cameras/
• [3] http://en.wikipedia.org/wiki/WiFi_Direct
• [4] http://www.kogan.com/au/buy/samsung-galaxy-camera-ek-gc100-white/
• [5] http://en.wikipedia.org/wiki/Nokia_808
• [6] http://www.theverge.com/2012/12/3/3706206/samsung-galaxy-camera-review