Review: Advanced Perl Programming, by Sriram Srinivasan

Publisher:	O'Reilly
Copyright:	August 1997
ISBN:	1-56592-220-4
Format:	Trade paperback
Pages:	392

The problem with buying lots of books is that one frequently doesn't read them soon after purchase. And while nearly all fiction and most non-fiction doesn't have a time limit, technical books are more of a problem. As you might have guessed from the publication date and subject matter of this book, it's one I waited too long to read. I picked up Advanced Perl Programming not too long after it was first published in 1997. I bought it primarily for the chapters on XS programming, since I was just starting to develop and help maintain code that built Perl interfaces for C libraries. And then it sat on my bookshelf in the office, and then at home, until it was 16 years old. That's problematic for a general Perl programming guide; it's particularly troublesome for a book that dives deep into internals. That said, I still got some interesting information from it. Advanced Perl Programming is something of a grab bag of different and largely unrelated topics. It starts with a data structures overview (which is mostly about references), covers typeglobs and the Perl symbol table, briefly discusses closures, and then dives deeper into eval. That's followed by an overview of modules and object-oriented programming in Perl. The general Perl topics close with a solid (if outdated) discussion of tied objects and a somewhat rambling and remarkably general discussion of persistance. Srinivasan then covers some recommended module APIs for networking and graphics (Tk, specifically) while working through some complex examples, takes a deeper dive into eval and code generation, and concludes with three chapters on XS and embedded Perl interpreters. This is all solid material, but a lot of it is at least partly obsolete. The chapters on modules and object-oriented programming fare the worst; you'll find much better guides to those topics elsewhere. Data structures also fare poorly, partly because Srinivasan is still happy to use symbolic references, now considered bad style due to lack of compile-time checking. (That does mean, though, that this is one of the best introductions to symbolic references that you'll find, since few other Perl programmers discuss them.) But the rest of the book is fairly solid, if limited by age. If I'd read it when I first started with XS, it would have indeed helped. Advanced Perl Programming contains a succinct and clear overview of Perl internals and data representation (assuming, of course, it's not changed drastically, but I didn't notice any significant issues from what I know of internals from quite a lot of XS programming), particularly when combined with the earlier chapter on typeglobs and the symbol table. It is interesting, though, to see how much accepted Perl style has changed. Srinivasan is quite fond of eval with a scalar argument to construct code at runtime, something that most current authors recommend against when it can be avoided. The loss of static syntax checking means the technique is often hard to debug and can cause surprising runtime errors. Several of the more advanced tricks in this book are built on that framework, including a moderately interesting templating system that he constructs from scratch in this book. The examples here are also missing the focus on use strict, warnings, and avoiding special variables and unusual constructs that you see in more recent Perl books. This is a very thoughtful book, with several structural techniques that I appreciated and would like to see more technical writers use. One is the comparisons to the approaches of other languages at the end of most chapters. It puts Perl's approach in a broader context and encourages thought about alternative ways to view a particular feature and implementation tradeoffs in Perl's choices. Another is the careful discussion of design approaches and possibilities during the construction of larger examples. I think there's a tendency in technical books to spend a lot of time developing the best possible implementation of something and then show that completed example. Even with a detailed analysis of its construction, this can be less useful than a broader discussion of possible approaches, including dead ends. The problem, of course, is that 16 years is a long time, even for a relatively mature programming language, and Perl has changed. There are significant improvements not only to the standard library but to Perl core functionality that have happened since this book was written. The chapter on tying, which is otherwise excellent, is badly out of date on what functionality and methods are available and is no longer a reliable guide to using that feature except for simple cases. Similar problems plague the chapter on persistence, which develops the concept from basics that most programmers reading an advanced book no longer need, and which has been rendered mostly obsolete by further developments in the Perl module ecosystem. I think the best parts of this book, the ones that cover material that I think is still relatively accurate and that are not well-covered by other resources, are the chapters on typeglobs and on XS. The typeglob discussion is both clear and informative, covering material and techniques that are used in a few core Perl modules and that I've never seen adequately addressed elsewhere. I'd subsequently picked up XS and Perl embedding the hard way: lots of trial and error and reading difficult POD documentation. I wish I'd read this book earlier, since Srinivasan does a great job of basic orientation. He doesn't go far enough for me (the book I probably wanted was Extending and Embedding Perl by Jenness and Cozens, published five years later), but I would have been up and running with basic XS much faster by starting here. If this were 1997, or even 2000, I would probably recommend this book, albeit with a few caveats. Higher Order Perl is the place you should go for discussion of closures, rather than here (although of course it didn't exist then). I wasn't fond of string eval even in 1997, and I still think it produces code that, while efficient and quite powerful, is horribly difficult to understand. And the introduction to modules is adequate but not compelling; most other introductions to object-oriented programming and module construction in Perl are better. But even discarding those chapters (and the mostly unnecessary Tk reference), there's a lot of great material here that would have been quite helpful in 2000. Unfortunately, this is not the sort of technical book that ages well, and while I did pick up some useful background knowledge and a much better understanding of typeglobs, I mostly read this book because I wanted to salvage some knowledge from a book that sat for far too long on my shelves. There's really no reason to buy it today. If you run across it in a library, there are a few chapters that are still worth reading, but by and large if you know enough Perl to know where to be careful of landmines due to changes in Perl since the book was written, you also know enough Perl to not really need this book. Rating: 5 out of 10

Whenever I want to work on some package, I usually clone its git repository, make my changes, then push and upload the Debian package. I don t keep those repositories around in order to avoid cruft and also to have a 100% clean, up-to-date setup whenever I start working on something. Everytime I clone such a repository, I struggle with the setup. For example, I usually forget the --pristine-tar flag for gbp-clone. Also, I usually forget to push other branches (working on debian , forgetting upstream ) and, even more often, I forget pushing tags. I spent some time on this and figured out that one can use the following to make --pristine-tar the default:

cat >> ~/.gbp.conf <EOF
[DEFAULT]
pristine-tar = True
EOF

Avoiding my other pain points is not so easy apparently, so I wrote a little shell function (only tested with zsh!) which uses debcheckout to get the git URL, gbp-clone to actually clone it and a few git config calls to make me able to just git push when I am done and not worry about anything:

# Clones the git sources of a Debian package
# needs debcheckout from devscripts and gbp-clone from git-buildpackage
function d-clone()  
    local package=$1
    if debcheckout --print $package >/dev/null
    then
        set -- $(debcheckout --print $package)
        if [ "$1" != "git" ]
        then
            echo "$package does not use git, but $1 instead."
            return
        fi
        echo "cloning $2"
        gbp-clone $2   return
        # Change to the newest git repository
        cd $(dirname $(ls -1td */.git   head -1))   return
        # This tells git to push all branches at once,
        # i.e. if you changed upstream and debian (after git-import-orig),
        # both upstream and debian will be pushed when running  git push .
        git config push.default matching   return
        # This tells git to push tags automatically,
        # so you don t have to use  git push; git push --tags .
        git config --add remote.origin.push "+refs/heads/*:refs/remotes/origin/*"   return
        git config --add remote.origin.push "+refs/tags/*:refs/tags/*"   return
        echo "d-clone set up everything successfully."
    else
        echo "debcheckout $package failed. Is $package missing Vcs tags?"
    fi
 

With that function, starting work on a package becomes as easy as d-clone golang-doc .

I ve been hacking on some static analyses stuff for debuild.me, and i ve been involved in a multi-year long yak shaving exercise. As today s fun part, I wrote python-schroot to help run commands in a schroot chroot (say that 10 times fast!) After a while, I got some neat stuff working. Here s an honest example:

from schroot import schroot
with schroot('unstable-amd64') as chroot:
    chroot.copy('/etc/issue', '/etc/issue', user='root')

This will copy a file (/etc/issue) from the host system into the schroot chroot. Neat! Now, to run something:

with schroot('unstable-amd64') as chroot:
    out, err, ret = chroot.run("whoami")
    print(out)    

Then, in an effort to make a DSL, I set out to create the following syntax:

with schroot('unstable-amd64') as chroot:
    "apt-get update" in chroot

but, hit some issues with implementing it, and got the following to work:

with schroot('unstable-amd64') as chroot:
    "apt-get update" > chroot // "root"  # apt-get update as root

More later!

In the past, we have had multiple heated discussions involving systemd. We (the pkg-systemd-maintainers team) would like to better understand why some people dislike systemd. Therefore, we have created a survey, which you can find at http://survey.zekjur.net/index.php/391182 Please only submit your feedback to the survey and not this thread, we are not particularly interested in yet another systemd discussion at this point. The deadline for participating in that survey is 7 days from now, that is 2013-05-26 23:59:00 UTC. Please participate only if you consider yourself an active member of the Debian community (for example participating in the debian-devel mailing list, maintaining packages, etc.). Of course, we will publish the results after the survey ends. Thanks! Best regards,
the Debian systemd maintainers

Apparently (hi Zhenech, found on Pl net Debian), a Man does not only need to fork a child, plant a tree, etc. in their life but also write a DynDNS service. Perfect for opening a new tag in the wlog called arch ology (pagetable.com Some Assembly Required is also a nice example for these). Once upon a time, I used SixXS heartbeat protocol client for updating the Legacy IP (known as IPv4 earlier) endpoint address of my tunnel at home (My ISP offers static v4 for some payment now, luckily). Their client sucked, so I wrote on in ksh, naturally. And because mksh(1) is such nice a language to program in (although, I only really begun becoming proficient in Korn Shell in 2005-2006 or so, thus please take those scripts with a grain of salt, I d do them much differently nowadays) I also wrote a heartbeat server implementation. In Shell. The heartbeat server supports different backends (per client), and to date I ve run backends providing DynDNS (automatically disabling the RR if the client goes offline), an IP (IPv6) tunnel of my own (basically the same setup SixXS has, without knowing theirs), rdate(8) based time offset monitoring for ntpd(8), and an eMail forwarding service (as one must not run an MTA on dynamic IP) with it; some of these even in parallel. Not all of it is documented, but I ve written up most things in CVS. There also were some issues (mostly to do with killing sleep(1)ing subprocesses not working right), so it occasionally hung, but very rarely. Running it under the supervise of DJB d montools was nice, as I was already using djbdns, since I do not understand the BIND zone file format and do not consider MySQL a database (and did not even like databases at all, back then). For DynDNS, the heartbeat server s backend simply updated the zone file (by either adding or updating or deleting the line for the client) then running tinydns-data, then rsync ing it to the djbdns server primary and secondaries, then running zonenotify so the BIND secondaries get a NOTIFY to update their zones (so I never had to bother much with the SOA values, only allow AXFR). That s a really KISS setup Anyway. This is arch ology. The scripts are there, feel free to use them, hack on them, take them as examples even submit back patches if you want. I ll even answer questions, to some degree, in IRC. But that s it. I urge people to go use a decent ISP, even if the bandwidth is smaller. To paraphrase a coworker after he cancelled his cable based internet access (I think at Un*tym*dia) before the 2-week trial period was even over: rather have slow but reliable internet at Netc*logne than that . People, vote with your purse!

I usually try to avoid administering printers whenever possible. As a result I end of flailing around the CUPS web interface before I figure out how to re-enable a printer. And, when I get a call to help debug a printer, I can't easily tell people what to do. When I try to do what I need via the command line, I end up spending at least 10 or 15 minutes re-reading man pages before I piece together the steps. Here's my attempt to document the steps so I don't have to re-read man pages. Setup In these examples, the printer name in question is: stability and it is a network printer, with local DNS that properly resolves the hostname stability to an IP address. The cups commands in these examples can be run as a non-root user if that user is in the lpadmin group. Type:

groups

To see if lpadmin is listed. If not:

sudo adduser <your-user-name> lpadmin

Then, to gain access to the new group without logging out and logging in again:

newgrp lpadmin

Network access First, try to ping the printer:

ping stability

If this fails, restart the printer and/or check network cables. No point in doing anything else until it responds to pings. Can't submit new jobs to the printer Next, if the problem is that the printer is greyed out when you try to print a document or your application tells you that the printer is rejecting jobs, confirm this status with:

lpstat -a stability

It will either output:

stability accepting requests since Mon 20 May 2013 10:28:57 AM EDT

Or

stability not accepting requests since Mon 20 May 2013 10:28:57 AM EDT -
  Rejecting Jobs

If it is rejecting jobs, try:

/usr/sbin/cupsaccept stability

Accepts new jobs, but just doesn't print On the other hand, if the printer is accepting jobs, but the jobs are not printing, find out if the printer is enabled with:

lpstat -p stability

You should get either:

printer stability is idle.  enabled since Mon 20 May 2013 10:28:57 AM EDT

Or:

printer stability disabled since Mon 20 May 2013 10:35:10 AM EDT -
  Paused

If it is disabled, you should first see what queued jobs there are:

lpq

If you have a list of duplicate pending jobs, be sure to delete the duplicates to avoid having your print job come out multiple times. To delete a queued job, type the following (n should be the number in the Job column of the lpq output):

cancel <n>

After you have deleted duplicate jobs, try "enabling" it:

/usr/sbin/cupsenable stability

Then, re-rerun the lpq command and see if it's now "ready." At this point, the jobs should start printing. Review of concepts For review... a few important concepts:

• cupsaccept/cupsreject: controls whether a printer will accept or reject new jobs. It doesn't matter whether the printer is enabled or disabled.
• cupsenable/cupsdisable: controls whether a printer will print existing jobs. It doesn't matter whether the print is accepting or rejecting new jobs.

PyBit is a distributed build system able to build packages in response to VCS commits or other triggers, across multiple architectures, multiple clients and multiple build environments with automated uploads to a nominated repository.

Support is included in 1.0.0 for building Debian packages using sbuild in response to subversion commits or changes in debian-devel-changes@lists.debian.org (by using apt as a version control handler) for any architecture and build environment which sbuild can support. There is also an example git commit template. Pybit has been designed to be fully extensible, so support for RPM or other package formats can be added as well as other version control handlers, other build environments and other architectures. Pybit is also scalable, when one type of client is struggling with the workload, another machine of the same architecture can be added to the pool to share the load. Pybit can also build a package for any number of architectures and build environments at the same time. The Pybit web interface provides an at-a-glance summary of all current builds as well as options to blacklist certain combinations, cancel and retry specific jobs and add monitor each pybit client. Current use cases include:

• Rapidly changing VCS - one or more subversion repositories with lots of Debian packages, built automatically for any number of build environments and architectures every time the debian/changelog is modified. Clean chroot builds provide continuous integration testing of the every package.


• Rebuilding the archive with different compilers or flags - a dedicated email account subscribed to debian-devel-changes@lists.debian.org feeding messages through procmail to the changes-debian hook, passing build requests to the apt handler to rebuild each package in your own sbuild chroots, using whatever environments, suites and build options can be configured within those chroots.


• something else we haven't thought of yet ... there is scope for a lot more hooks, package formats, chroot tools and handler plugins.




Pybit 1.0.0 has arrived in Debian unstable as a direct result of the efforts put in by the pybit team during a sprint on 18th May 2013. Thanks to everyone involved in Pybit.

https://freecode.com/projects/pybit

http://nicholasdavidson.github.io/pybit/

https://github.com/nicholasdavidson/pybit

Most UNIX users have heard of the nice utility used to run a command with a lower priority to make sure that it only runs when nothing more important is trying to get a hold of the CPU:

nice long_running_script.sh

That's only dealing with part of the problem though because the CPU is not all there is. A low priority command could still be interfering with other tasks by stealing valuable I/O cycles (e.g. accessing the hard drive).

Prioritizing I/O Another Linux command, ionice, allows users to set the I/O priority to be lower than all other processes. Here's how to make sure that a script doesn't get to do any I/O unless the resource it wants to use is idle:

sudo ionice -c3 hammer_disk.sh

The above only works as root, but the following is a pretty good approximation that works for non-root users as well:

ionice -n7 hammer_disk.sh

You may think that running a command with both nice and ionice would have absolutely no impact on other tasks running on the same machine, but there is one more aspect to consider, at least on machines with limited memory: the disk cache.

Polluting the disk cache If you run a command (for example a program that goes through the entire file system checking various things, you will find that the kernel will start pulling more files into its cache and expunge cache entries used by other processes. This can have a very significant impact on a system as useful portions of memory are swapped out. For example, on my laptop, the nightly debsums, rkhunter and tiger cron jobs essentially clear my disk cache of useful entries and force the system to slowly page everything back into memory as I unlock my screen saver in the morning. Thankfully, there is now a solution for this in Debian: the nocache package. This is what my long-running cron jobs now look like:

nocache ionice -c3 nice long_running.sh

Turning off disk syncs Another relatively unknown tool, which I would certainly not recommend for all cron jobs but is nevertheless related to I/O, is eatmydata. If you wrap it around a command, it will run without bothering to periodically make sure that it flushes any changes to disk. This can speed things up significantly but it should obviously not be used for anything that has important side effects or that cannot be re-run in case of failure. After all, its name is very appropriate. It will eat your data!

If I did everything right, this post will not appear on any RSS feed yet still make it to my blog to maintain history. The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 1088
  • Affecting Jessie: 214 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting Jessie and unstable: 183 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 43 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 15 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 125 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting Jessie only: 31 Those are already fixed in unstable, but the fix still needs to migrate to Jessie. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 0 bugs are in packages that are unblocked by the release team.
      • 31 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Diff
43	284 (213+71)	468 (332+136)	+184 (+119/+65)
44	261 (201+60)	408 (265+143)	+147 (+64/+83)
45	261 (205+56)	425 (291+134)	+164 (+86/+78)
46	271 (200+71)	401 (258+143)	+130 (+58/+72)
47	283 (209+74)	366 (221+145)	+83 (+12/+71)
48	256 (177+79)	378 (230+148)	+122 (+53/+69)
49	256 (180+76)	360 (216+155)	+104 (+36/+79)
50	204 (148+56)	339 (195+144)	+135 (+47/+90)
51	178 (124+54)	323 (190+133)	+145 (+66/+79)
52	115 (78+37)	289 (190+99)	+174 (+112/+62)
1	93 (60+33)	287 (171+116)	+194 (+111/+83)
2	82 (46+36)	271 (162+109)	+189 (+116/+73)
3	25 (15+10)	249 (165+84)	+224 (+150/+74)
4	14 (8+6)	244 (176+68)	+230 (+168/+62)
5	2 (0+2)	224 (132+92)	+222 (+132/+90)
6	release!	212 (129+83)	+212 (+129/+83)
7	release+1	194 (128+66)	+194 (+128/+66)
8	release+2	206 (144+62)	+206 (+144/+62)
9	release+3	174 (105+69)	+174 (+105/+69)
10	release+4	120 (72+48)	+120 (+72/+48)
11	release+5	115 (74+41)	+115 (+74/+41)
12	release+6	93 (47+46)	+93 (+47/+46)
13	release+7	50 (24+26)	+50 (+24/+26)
14	release+8	51 (32+19)	+51 (+32/+19)
15	release+9	39 (32+7)	+39 (+32/+7)
16	release+10	20 (12+8)	+20 (+12/+8)
17	release+11	24 (19+5)	+24 (+19/+5)
18	release+12	2 (2+0)	+2 (+2/+0)

Graphical overview of bug stats thanks to azhag:

Debian Edu / Skolelinux is an operating system based on Debian intended for use in schools. It contain a turn-key solution for the computer network provided to pupils in the primary schools. It provide both the central server, network boot servers and desktop environments with heaps of educational software. The project was founded almost 12 years ago, 2001-07-02. If you want to support the project, which is in need for cash to fund developer gatherings and other project related activity, please donate some money. A topic that come up again and again on the Debian Edu mailing lists and elsewhere, is the question on how to transform a Debian or Ubuntu installation into a Debian Edu installation. It isn't very hard, and last week I wrote a script to replicate the steps done by the Debian Edu installer. The script, debian-edu-bless in the debian-edu-config package, will go through these six steps and transform an existing Debian Wheezy or Ubuntu (untested) installation into a Debian Edu Workstation:

1. Add skolelinux related APT sources.
2. Create /etc/debian-edu/config with the wanted configuration.
3. Install debian-edu-install to load preseeding values and pull in our configuration.
4. Preseed debconf database with profile setup in /etc/debian-edu/config, and run tasksel to install packages according to the profile specified in the config above, overriding some of the Debian automation machinery.
5. Run debian-edu-cfengine-D installation to configure everything that could not be done using preseeding.
6. Ask for a reboot to enable all the configuration changes.

There are some steps in the Debian Edu installation that can not be replicated like this. Disk partitioning and LVM setup, for example. So this script just assume there is enough disk space to install all the needed packages. The script was created to help a Debian Edu student working on setting up Raspberry Pi as a Debian Edu client, and using it he can take the existing Raspbian installation and transform it into a fully functioning Debian Edu Workstation (or Roaming Workstation, or whatever :). The default setting in the script is to create a KDE Workstation. If a LXDE based Roaming workstation is wanted instead, modify the PROFILE and DESKTOP values at the top to look like this instead:

PROFILE="Roaming-Workstation"
DESKTOP="lxde"

The script could even become useful to set up Debian Edu servers in the cloud, by starting with a virtual Debian installation at some virtual hosting service and setting up all the services on first boot.

In my professional life I am sometimes required to provide technical support to one of our salesmen. I find this an interesting change in pace though sometimes challenging.

Occasionally I fail to clearly convey the solution we are trying to sell because of my tendency to focus on detail the customer probably does not need to understand but I think is the interesting part of the problem.

Conversely sometimes the sales people gloss over important technology choices which have a deeper impact on the overall solution. I was recently in such a situation where as part of a larger project the subject of internationalisation (you can see why it gets abbreviated to i18n) was raised.

I had little direct personal experience with handling this within a project workflow so could not give any guidance but the salesman recommended the Transifex service as he had seen it used before, indicated integration was simple and we moved onto the next topic.

Unfortunately previous experience tells me that sometime in the near future someone is going to ask me hard technical questions about i18n and possibly how to integrate Transifex into their workflow (or at least give a good estimate on the work required).
LearningBeing an engineer I have few coping strategies available for situations when I do not know how something works. The approach I best know how to employ is to give myself a practical crash course and write up what I learned...so I did.

I proceeded to do all the usual things you do when approaching something unfamiliar (wikipedia, google, colleagues etc.) and got a basic understanding of internationalisation and localisation and how they fit together.

This enabled me to understand that the Transifex workflow proposed only covered the translation part of the problem and that, as Aldrich observed in my title quote, there is an awful lot more to translation than I suspected.
PlatformsMy research indicated that there are numerous translation platforms available for both open source and commercial projects and Transifex is one of many solutions.

Although the specific platform used was Transifex most of these observations apply to all these other platforms. The main lesson though is that all platforms are special snowflakes and once a project invests effort and time into one platform it will result in the dreaded lock in. The effort to move to another platform afterwards is at least as great as the initial implementation.

It became apparent to me that all of these services, regardless of their type, boil down to a very simple data structure. They appear to be a trivial table of Key:Language:Value wrapped in a selection of tools to perform format conversions and interfaces to manipulate the data.

There may be facilities to attach additional metadata to the table such as groupings for specific sets of keys (often referred to as resources) or translator hints to provide context but the fundamental operation is common.

The pseudo workflow is:

• Import a set of keys
• Provide a resource grouping for the keys.
• Import any existing translations for these keys.
• Use the services platform to provide additional translations
• Export the resources in the desired languages.

The Transifex service does at least allow for the Key values to be arbitrary although the resources are separated by language.

Even assuming a project uses gettext PO files and UTF-8 character encoding (and please can we kill every other character encoding and move the whole world to UTF-8) the tools to integrate the import/export into the project must be written.

A project must decide some pretty important policies, including:

• Will they use a single service to provide all their translations.
• Will they allow updates to the files in their revision control system and how those will be integrated.
• Will there be a verification step and if so who and how will that be performed. Especially important is the question of a reviewer understanding the translated language being integrated and how that is controlled.
• Will the project be paying for translations
• Will the project allow machine translations, if not can they be used as an initial hint (sometimes useful if the translators are weak in the "default" language

These are project policy decisions and, as I discovered, just as difficult to answer as the technical challenges.

Armed with my basic understanding it was time to move on and see how the transifex platform could be integrated into a real project workflow.
ImplementingProof of conceptMy first exercise was to take a trivial command line tool, use xgettext to generate a PO file and add the relevant libintl calls to produce gettext internationalised tool.

A transifex project was created and the english po file uploaded as the initial resource. A french language was added and the online editor used to provide translations for some strings. The PO resource file for french was exported and the tool executed with LANGUAGE=fr and the french translation seen.

This proved the trivial workflow was straightforward to implement. It also provided insight into the need to automate the process as the manual website operation would soon become exceptionally tedious and error prone.
Something more usefulTo get a better understanding of a real world workflow I needed a project that:

• Already internationalised but had limited language localisation
• Did not directly use gettext
• Had a code base I understood
• Could be modified reasonably easily.
• Might find the result useful rather than it being a purely academic exercise.

I selected the NetSurf web browser as it best fit this list.

Internally NetSurf keeps all the translated messages in a simple associative array this is serialised to an equally straightforward file named FatMessages. The file is UTF-8 encoded with keys separated from values by a colon. The Key is constrained to be ASCII characters with no colons and is structured as language.toolkit.identifier and is unique on identifier part alone.

This file is processed at build time into a simple identifier:value dictionary for each language and toolkit.

Transifex can import several resource formats similar to this, after experimenting with YAML and Android Resource format I immediately discovered a problem, the services import and export routines were somewhat buggy.

These routines coped ok with simple use cases but having more complex characters such as angle brackets and quotation marks in the translated strings would completely defeat the escaping mechanisms employed by both these formats (through entity escaping in android resource format XML is problematic anyway)

Finally the Java property file format was used (with UTF-8 encoding) which while having bugs in the import and export escaping these could at least be worked around. The existing tool that was used to process the FatMessages file was rewritten to cope with generating different output formats and a second tool to merge the java property format resources.

To create these two tools I enlisted the assistance of my colleague Vivek Dasmohapatra as his Perl language skills exceeded my own. He eventually managed to overcome the format translation issues and produce correct input and output.

I used the Transifex platforms free open source product, created a new project and configured it for free machine translation from the Microsoft service, all of which is pretty clearly documented by Transifex.

Once this was done the messages file was split up tinto resources for the supported languages and uploaded to the transifex system.

I manually marked all the uploaded translations as "verified" and then added a few machine translations to a couple of languages. I also created spanish as a new language and machine translated most of the keys.

The resources for each language were then downloaded and merged and the resulting FatMessages file checked for differences and verified only the changes I expected appeared.

I quickly determined that manually downloading the language resources every time was not going to work with any form of automation, so I wrote a perl script to retrieve the resources automatically (might be useful for other projects too).

Once these tools were written and integrated into the build system I could finally make an evaluation as to how successful this exercise had been.
ConclusionsThe main things I learned from this investigation were:

• Internationalisation has a number of complex areas
• Localisation to a specific locale is more than a mechanical process.
• The majority of platforms and services are oriented around textural language translation
• There is a concentration on the gettext mode of operation in many platforms
• Integration to any of these platforms requires both workflow and technical changes.
• At best tools to integrate existing resources into the selected platform need to be created
• Many project will require format conversion tools, necessitating additional developer time to create.
• The social issues within an open source project may require compromise on the workflow.
• The external platform may offer little benefits beyond a pretty user interface.
• External platforms introduce an external dependency unless the project is prepared and able to run its own platform instance.
• Real people are still required to do the translations and verify them.

Review: Asimov's Science Fiction, July 2011

Editor:	Sheila Williams
Issue:	Volume 35, No. 7
ISSN:	1065-2698
Pages:	112

Williams's editorial is a mildly interesting piece about story titles. Silverberg's column is a more interesting (and rather convincing) rebuttal of the joke that fiction authors are "professional liars," combined with an examination of a fake and fantastic 14th travelogue that (at least in Silverberg's telling) was widely believed at the time. The precis of Silverberg's argument is that lying requires an intent to deceive, which is a property of deceptive memoir writers but not of fiction authors. Di Filippo's review column, as usual, is devoted almost entirely to esoterica, although I was moderately interested to hear of Stableford's continued work on translating early French SF. None of it seems compelling enough to go buy, but good translations of early works seem like a good thing to have in the world. "Day 29" by Chris Beckett: The conceit of this novelette is an interstellar travel system akin to a transporter that allows near-instantaneous travel between worlds. The drawback is that all memories from somewhere between 40 and 29 days before transit up until transit are wiped. The progatonist is a data analyst who is about to travel, and therefore by agency rule is required to stop doing work on day 40 before transmission since he can't be held legally liable for anything he has no recollection of doing. (I would like to say that I find this implausible, since one could always keep records, but it's exactly the sort of ass-covering regulation that a human resources department would come up with.) The premise is quite interesting: what do you do during that period that you're going to forget? Beckett wisely mixes Stephen's current waiting period on the colony world with his diary of his original waiting period on Earth the first time he went through the transmission process, and the latter adds greatly to the reader's appreciation of the weirdness of the forgotten interval. Unfortunately, this is a story more about psychological exploration than about plot, and Stephen just isn't very interesting. The telepathic but possibly nonsentient aliens add weirdness but not much else, and the ending of the story provided little sense of closure or conclusion for me. A good idea, but not the execution I wanted. (5) "Pug" by Theodora Goss: Since I grew up with a pug, I have a soft spot for a story featuring one; sadly, though, this story has insufficient pug in it. This is a quiet fantasy (Asimov's calls it SF, presumably on the basis of parallel worlds and a hypothesized scientific explanation, but it reads like fantasy to me) featuring Victorian girls, including one with a bad heart. They discover a hidden door to other versions of their world and do some minor exploration. There's little or nothing in the way of plot; the story is more of an attempt to capture a mood. It's mildly diverting, but I wish it had gone somewhere more substantial. (5) "Dunyon" by Kristine Kathryn Rusch: A Rusch story is often the highlight of an issue, and this is no exception. The protagonist is the owner of a bar in a space station that's become a combination of a refugee camp and a slum. War and chaos have created desperate people, most of whom are attempting to find some way to resources and get out of the bottom of society. The story is about a rumor: a mythical system named Dunyon that's safe and far away. And it's about how people react to that rumor. There's nothing particularly surprising about the direction the story goes (it's fairly short), but Rusch is always a good storyteller. (7) "The Music of the Sphere" by Norman Spinrad: I've had mixed feelings about Spinrad's fiction (and some of his essays), but I liked this story, despite its implausibility. It's set in the near future, featuring an expert in cetaceans and dolphin perception and a composer obsessed with both loud music and classical musical style. Just from that description, you can probably predict much of the story, but I thought it had some neat ideas about dolphins, whales, and alternate perception and aesthetics. (Note: neat, not necessarily biologically plausible.) Enjoyable. (6) "Bring on the Rain" by Josh Roseman: In a change of pace from the rest of the issue, this is a post-apocalyptic story of caravans of wheeled ships traversing a scorched and ruined landscape in search of weather systems and rain. The feel is of an inverted Waterworld, but with more emphasis on military tactics and cooperating fleets. The transposition of fleet maneuvers to huge ground vehicles adds some extra fun. The plot has little to do with the background and is a fairly stock military adventure scenario, but it's reasonably well-told. The story feels like an excerpt from a larger military-SF-inspired adventure, but the length keeps the quantity of tactics and maneuvering below the threshold where I would get bored. (6) "Twelvers" by Leah Cypess: This is a sharp and occasionally mean story of adolescent cruelty and alienation. Darla is a "twelver," a child who was carried an extra three months in the womb using newly-invented medical technology because of a belief in the advantages this would bring in later life. Unfortunately for all those who used this technique, what it also brought was a preternatural calm and an unusual reaction to emotions. Darla finds it almost impossible to get upset at anything, and that, of course, prompts the cruelty and abuse of other children. Most of the story is a description of that abuse, leading up to Darla stumbling into a nasty solution to her immediate problem. It's all very believable (well, apart from the motivating biology), but I didn't enjoy reading about it, and I'm certainly not convinced that the ending will lead to anything good. (5) "The Messenger" by Bruce McAllister: This is a very short time travel story, where time travel is used to try to unwind old family pain. This world follows the unalterable history model: no changes to the past are possible, and anything you do in the past has already happened. The mechanics are mostly avoided. Instead, McAllister concentrates on his mother, his father, and their complex relationship. I would have needed a bit more background on the characters to care enough about them for the story to be fully effective, but while the heartstring-pulling is kind of obvious, it's still a solid story. (6) "The Copenhagen Interpretation" by Paul Cornell: This is the most ingenious of the stories in this issue. It's set in a future world that extends what seemed to me to be pre-World-War-I great power politics, although there may be a hint of the Cold War. Great nations have reached a careful balance of power, and spies and secret services work to sustain that balance. The progatonist is one of those agents, making use of advanced technology like space folds in the service of a cause that he doesn't entirely believe in. Cornell mixes in mental conditioning, artificial people, space travel, and even aliens (maybe) in a taut thriller plot that, for me, gained a great deal from the unexplained strangeness of its background. If you like diving into the deep end and following a fast-moving plot against a background of strangeness, this is the sort of SF you'll enjoy. (7) Rating: 6 out of 10

Let s consider the following scenario: I have two Python processes receiving the same events and I have to measure the delay between when process A received the event and when process B received it, as precisely as possible (i.e. less than 1ms). Using Python 2.7 and a Unix system you can use the time.time method which provides the time in seconds since Epoch and has a typical resolution of a fraction of a ms on Unix. You can use it on different processes and still compare the results, since both processes receive the time since Epoch, a defined and fixed time in the past. On Windows time.time also provides the time since Epoch, but the resolution is in the range of 10ms, which is not suitable for my application. There is also time.clock which is super precise on Windows, and much less precise on Unix. The mayor drawback is that it returns the time since the process started or since the first call of time.clock within that processes. This means you cannot compare the results of time.clock between two processes as they are not calibrated to a common t-zero. I had high hopes for Python 3.3 where the time module was revamped and I was reading about time.monotonic and time.perf_counter. Especially time.perf_counter looked like it would suit my needs as the documentation said it provides the highest available resolution for the system and was system-wide , in contrast to for example the new time.process_time which was process_wide . Unfortunately it turned out that time.perf_counter acts similar to time.clock on Python 2.7 as it provides you with the time since the process started or the first time the method was called within the process. The results of time.monotonic are comparable between processes, but again not precise enough on Windows. Here is a small script which demonstrates how the times provided by time.clock and time.perf_counter are not comparable between processes. It starts two processes and lets both of them print out the output of the timer to stdout. In the output the times should be monotonically increasing. Since I let process 2 sleep for one second before calling the timer method for the first time, the output of this process is usually one second smaller when using time.clock or time.perf_counter.

#!/usr/bin/env python
from multiprocessing import Process
import time
timers = ['clock', 'time', 'monotonic', 'perf_counter']
def proc(timer):
    timer = getattr(time, timer)
    time.sleep(1)
    for i in range(3):
        print('P2  time '.format(time=timer()))
        time.sleep(1)
if __name__ == '__main__':
    for t in timers:
        print("Using  timer ".format(timer=t))
        p = Process(target=proc, args=(t,))
        timer = getattr(time, t)
        p.start()
        for i in range(3):
            print('P1  time '.format(time=timer()))
            time.sleep(1)
        p.join()

The result when running on Windows with Python 3.3:

$ python timertest.py
Using clock
P1 6.146032526480321e-06
P1 0.9926582847820045
P2 2.9612702173041547e-05
P1 1.9941743992602412
P2 1.0008579302676737
P2 2.0022709590185346
Using time
P1 1368614235.509732
P1 1368614236.511172
P2 1368614236.601301
P1 1368614237.512612
P2 1368614237.602741
P2 1368614238.604181
Using monotonic
P1 484.636
P1 485.63800000000003
P2 485.738
P1 486.639
P2 486.73900000000003
P2 487.741
Using perf_counter
P1 12.390910576623565
P1 13.39050745276285
P2 7.542858100680394e-06
P1 14.39190763071843
P2 1.0014012954160376
P2 2.0041399116368144

So as far as I see it, there is no way of getting comparable times between two processes on Windows with more precision than 10ms. Is that correct or am I missing something?

Most OpenPGP User IDs look like this:

Jane Q. Public <jane@example.org>

This is clean, clear, and unambiguous. However, some tools (gpg, enigmail among others) ask the user to provide a "Comment:" field when they are choosing a new User ID (e.g. when making a new key). These UI prompts are evil. The savvy user knows to avoid entering anything in this field, so that they can end up with a User ID like the one above. The user who provides something here (perhaps even something inconsequential like "I like strawberries", due to not being sure what should go in this little box) will instead end up with a User ID like:

Jane Q. Public (I like strawberries) <jane@example.org>

This is bad. This means that Jane is asking the people who certify her key+userid to certify whether she actually likes strawberries (how could they know? what if she changes her mind? should they revoke their certifications?) and anywhere that she is referred to by name will include this mention of strawberries. This is not Jane's identity, and it doesn't belong in an OpenPGP User ID packet. Furthermore, since User IDs are atomic, if Jane wants to change the comment field (but leave her name and e-mail address the same), she will instead need to create a new User ID, publish it, get everyone who has certified her old key+userid to certify the key+newuserid, and then revoke the old one. It is difficult already to help people understand and participate in the certification network that forms that backbone of OpenPGP's so-called "web of trust". These bogus comment fields make an already-difficult task harder. And all because of strawberries! Tools like enigmail and gpg should not expose the "Comment:" field to users who are generating keys or choosing new User IDs. If they feel it absolutely must be present for some weird corner case that 0.1% of their users will have, they could require that the user enters some sort of "expert mode" before prompting the user to do something that is likely to be a mistake. There is almost no legitimate reason for anyone to use this field. Let's go through some examples of this people use, taken from some examples i have lying around (identifying marks have been changed to protect the innocent who were duped by this bad UI choice, but you can probably find them on the public keyserver network if you want to hunt around):

domain repetition

John Q. Public (Debian) <johnqpublic@debian.org>

We know you're with debian already from the @debian.org address. If this is in contrast to your other address (johnqpublic@example.org) so that people know where to send you debian-related e-mail, this is still not necessary. Lest you think i'm just calling out debian developers, people with @ubuntu.com addresses and (Ubuntu) comments (as well as @example.edu addresses and (Example University) comments and @example.com addresses and (Example Corp) comments) are out there too.

nicknames already evident

John Q. Public (Johnny) <johnqpublic@example.net>
John Q. Public (wackydude) <wackydude@example.net>

Again, the information these comments are providing offers no clear disambiguation from the info already contained in the name and e-mail address, and just muddies the water about what the people who certify this identity should actually be trying to verify before they make their certification.

"Work"

John Q. Public (Work) <johnqpublic@example.com>

if John's correspondents know that he works for Example Corp, then "Work" isn't helpful to them, because they already know this as the address that they're writing to him with. If they don't know that, then they probably aren't writing to him at work, so they don't need this comment either. The same problem appears (for example) with literal comments of (School) next to their @example.edu address.

This is my nth try at this crazy system!

John Q. Public (This is my second key) <johnqpublic@example.com>
John Q. Public (This is my primary key) <johnqpublic@example.com>
John Q. Public (No wait really use this one) <johnqpublic@example.com>

OpenPGP is confusing, and it can be tricky to get it right. We all know :) This is still not part of John's identity. If you want to designate a key as your preferred key, keep it up-to-date, get people to certify it, and revoke or expire your old keys. People who care can look at the timestamps on your keys and tell which ones are the most recent ones. You do have a revocation certificate for your key handy just in case you lose it, right?

Don't use this key

John Q. Public (Old key, do not use) <johnqpublic@example.com>
John Q. Public (Please only use this through September 2004) <johnqpublic@example.com>

This kind of sentiment is better expressed by revoking the key in question or setting an expiration time on the key or User ID self-sig directly. This sentiment is not part of John's identity, and shouldn't be included as though it were.

"none"

John Q. Public (none) <johnqpublic@example.com>

sigh. This is clearly someone getting mixed up by the UI.

I use strong crypto!

John Q. Public (3092 bits of RSA) <johnqpublic@example.com>

This comment refers to the strength of the key material, or the algorithms preferred by the user. Since the User ID is associated with the key material already, people who care about this information can get it from the key directly. This is also not part of the user's actual identity.

"no comment"

John Q. Public (no comment) <johnqpublic@example.com>

This is actually not uncommon (some keyservers reply "too many matches!"). It shows that the user is witty and can think on their feet (at least once), but it is still not part of the user's identity.

But wait (i hear you say)! I have a special case that actually is a legitimate use of the comment field that cannot be expressed in OpenPGP in any other way! I'm sure that such cases exist. I've even seen one or two of them. The fact that one or two cases exist does not excuse the fact that that overwhelming majority of these comments in OpenPGP User IDs are a mistake, caused only by bad UI design that prompts people to put something (anything!) in the empty box (or on the command prompt, depending on your preference). And this mistake is one of the thousand papercuts that inhibits the robust growth of the OpenPGP certification network that some people call the "web of trust". Let's avoid them so we can focus on the other 999 papercuts. Please don't use comments in your OpenPGP User ID. And if you make a user interface for OpenPGP that prompts the user to decide on a new User ID, please don't include a prompt for "Comment" unless the user has already certified that they are really and truly a special special snowflake. Thanks!Tags: openpgp, ui

I would like to suggest that the word unprofessional be struck from the dictionary and anyone who uses it struck with a dictionary. It is a word which conveys no useful information or proposal for action, and is thus nothing but meaningless noise. The purpose of communication is to adjust another person s process of cognition. I ve heard it said that all communication is persuasion , which is quite true you re trying to persuade someone to change what they think. We can consider the intention and effectiveness of an attempt to communicate in this light. What is someone trying to achieve when they label a person or behaviour unprofessional ? If we re being charitable, we would probably say that they re trying to highlight that something is bad, or could be better. However, just stamping our foot and saying bad! isn t enough it s also important to provide some information that the recipient can act upon. The problem with the word unprofessional is that it really isn t specific enough on the subject of what is wrong . Have you ever had someone say something like, your behaviour yesterday was really unprofessional ? They re assuming you know what they re talking about and you might well have a reasonable guess but what if you guess wrong? Should you never do anything you did yesterday, just in case that particular thing was unprofessional? When I ve caught myself thinking, that was unprofessional , of my own behaviour, or someone else s, I think about what caused me to think that. Once I drill down into it, I usually come to the conclusion that what I really meant was, I don t like that . Since I m not paid to like things, that s pretty much irrelevant as a reason to tell someone not to do something. On the occasions when I come up with something more concrete, it is invariably a more useful expression than unprofessional . Things like, it frustrates the customer , or it pisses off the person sitting in the next cube are a much better expression of why something is bad than unprofessional . I d encourage everyone to keep a careful watch over themselves and those around them for use of the word. When you catch yourself saying it (or thinking it), examine your motives more closely. Whatever the more specific adjective is, use that instead. If it just comes down to I don t like that , at the very least say that to the person you re talking to. Don t try and hang anything grandiose on your personal prejudices. You might come off as being petty, but at least you ll be honest.

A couple of days ago, I had an opportunity to give a guest lecture on our Rcpp package for R and C++ integration. This was in CMSC 12300 Computer Science with Applications-3 in the Department of Computer Science at University of Chicago. The course is the final part of a three term sequence introducing students to data-centric work in R, Python, Java and C++. I tried to keep it brief and engaging in order to motivate the why or R/C++ integration while providing plenry of useful examples. And yesterday I got to spend a day giving an invited day-long workshop at the Medical College of Wisconsin as part of a two-day R workshop sponsored by the Milwaukee Chapter of the American Statistical Assocation as well as the CTSI and PCOR centers at the Medical College of Wisconsin. In the workshop, I followed the previously-used setup of four parts on introduction, Rcpp details, advanced topics and last-but-not-least applications, but also updated and extended to more recent topics. Pdf slides from both events are now on my presentations page.

I've just released qcontrol 0.5.1. Changes since the last release:

• Add build targets to enable static linking (Original patch by Frans Pop).
• Wake-on-Lan and EUP control (by Michael Stapelberg, Debian bug #703888).
• Updated example configurations (based on Debian package).
• Support loading configuration snippets from a directory (Ian Campbell, Debian bug #697574).
• Various other bug fixes.

I also put together a very basic homepage. Get it from gitorious or http://www.hellion.org.uk/qcontrol/releases/0.5.1/. The Debian package will be uploaded shortly.