In my professional life I am sometimes required to provide technical support to one of our salesmen. I find this an interesting change in pace though sometimes challenging.
Occasionally I fail to clearly convey the solution we are trying to sell because of my tendency to focus on detail the customer probably does not need to understand but I think is the interesting part of the problem.
Conversely sometimes the sales people gloss over important technology choices which have a deeper impact on the overall solution. I was recently in such a situation where as part of a larger project the subject of
internationalisation (you can see why it gets abbreviated to i18n) was raised.
I had little direct personal experience with handling this within a project workflow so could not give any guidance but the salesman recommended the
Transifex service as he had seen it used before, indicated integration was simple and we moved onto the next topic.
Unfortunately previous experience tells me that sometime in the near future someone is going to ask me hard technical questions about i18n and possibly how to integrate Transifex into their workflow (or at least give a good estimate on the work required).
LearningBeing an engineer I have few coping strategies available for situations when I do not know how something works. The approach I best know how to employ is to give myself a practical crash course and write up what I learned...so I did.
I proceeded to do all the usual things you do when approaching something unfamiliar (wikipedia, google, colleagues etc.) and got a basic understanding of internationalisation and localisation and how they fit together.
This enabled me to understand that the Transifex workflow proposed only covered the translation part of the problem and that, as
Aldrich observed in my title quote, there is an awful lot more to translation than I suspected.
PlatformsMy research indicated that there are numerous translation platforms available for both open source and commercial projects and Transifex is one of many solutions.
Although the specific platform used was Transifex most of these observations apply to all these other platforms. The main lesson though is that all platforms are special snowflakes and once a project invests effort and time into one platform it will result in the dreaded lock in. The effort to move to another platform afterwards is at least as great as the initial implementation.
It became apparent to me that all of these services, regardless of their type, boil down to a very simple data structure. They appear to be a trivial table of Key:Language:Value wrapped in a selection of tools to perform format conversions and interfaces to manipulate the data.
There may be facilities to attach additional metadata to the table such as groupings for specific sets of keys (often referred to as resources) or translator hints to provide context but the fundamental operation is common.
The pseudo workflow is:
- Import a set of keys
- Provide a resource grouping for the keys.
- Import any existing translations for these keys.
- Use the services platform to provide additional translations
- Export the resources in the desired languages.
The first three steps are almost always performed together by the uploading of a resource file containing an initial set of translations in the "default" language and
due to the world being the way it is this is almost always english (some services are so poorly tested with other defaults they fail if this is not the case!)
The platforms I looked at generally follow this pattern with a greater or lesser degree of freedom in what the keys are, how the groupings into resources are made and the languages that can be used. The most common issue with these platforms (especially open source ones) is that the input convertors will only accept a very limited number of formats and often restricted to just
GNU gettext PO files. This means that to use those platforms a project would have to be able to convert any internal resources into gettext translatable format.
The prevalence of the PO format pushes assumptions into almost every platform I examined, mainly that a resource is for a single language translation and that the Key (msgid in gettext terms) is the untranslated default language string in the C locale.
The Transifex service does at least allow for the Key values to be arbitrary although the resources are separated by language.
Even assuming a project uses gettext PO files and
UTF-8 character encoding (and please can we kill every other character encoding and move the whole world to UTF-8) the tools to integrate the import/export into the project must be written.
A project must decide some pretty important policies, including:
- Will they use a single service to provide all their translations.
- Will they allow updates to the files in their revision control system and how those will be integrated.
- Will there be a verification step and if so who and how will that be performed. Especially important is the question of a reviewer understanding the translated language being integrated and how that is controlled.
- Will the project be paying for translations
- Will the project allow machine translations, if not can they be used as an initial hint (sometimes useful if the translators are weak in the "default" language
These are project policy decisions and, as I discovered, just as difficult to answer as the technical challenges.
Armed with my basic understanding it was time to move on and see how the transifex platform could be integrated into a real project workflow.
ImplementingProof of conceptMy first exercise was to take a trivial command line tool, use xgettext to generate a PO file and add the relevant libintl calls to produce gettext internationalised tool.
A transifex project was created and the english po file uploaded as the initial resource. A french language was added and the online editor used to provide translations for some strings. The PO resource file for french was exported and the tool executed with LANGUAGE=fr and the french translation seen.
This proved the trivial workflow was straightforward to implement. It also provided insight into the need to automate the process as the manual website operation would soon become exceptionally tedious and error prone.
Something more usefulTo get a better understanding of a real world workflow I needed a project that:
- Already internationalised but had limited language localisation
- Did not directly use gettext
- Had a code base I understood
- Could be modified reasonably easily.
- Might find the result useful rather than it being a purely academic exercise.
I selected the NetSurf web browser as it best fit this list.
Internally NetSurf keeps all the translated messages in a simple associative array this is serialised to an equally straightforward file named
FatMessages. The file is UTF-8 encoded with keys separated from values by a colon. The Key is constrained to be ASCII characters with no colons and is structured as language.toolkit.identifier and is unique on identifier part alone.
This file is processed at build time into a simple identifier:value dictionary for each language and toolkit.
Transifex can import
several resource formats similar to this, after experimenting with YAML and Android Resource format I immediately discovered a problem, the services import and export routines were somewhat buggy.
These routines coped ok with simple use cases but having more complex characters such as angle brackets and quotation marks in the translated strings would completely defeat the escaping mechanisms employed by both these formats (through entity escaping in android resource format XML is problematic anyway)
Finally the
Java property file format was used (with UTF-8 encoding) which while having bugs in the import and export escaping these could at least be worked around. The
existing tool that was used to process the FatMessages file was rewritten to cope with generating different output formats and a
second tool to merge the java property format resources.
To create these two tools I enlisted the assistance of my colleague Vivek Dasmohapatra as his Perl language skills exceeded my own. He eventually managed to overcome the format translation issues and produce correct input and output.
I used the Transifex platforms free open source product, created a new project and configured it for free machine translation from the Microsoft service, all of which is pretty clearly documented by Transifex.
Once this was done the messages file was split up tinto resources for the supported languages and uploaded to the transifex system.
I manually marked all the uploaded translations as "verified" and then added a few machine translations to a couple of languages. I also created spanish as a new language and machine translated most of the keys.
The resources for each language were then downloaded and merged and the resulting FatMessages file checked for differences and verified only the changes I expected appeared.
I quickly determined that manually downloading the language resources every time was not going to work with any form of automation, so I wrote a
perl script to retrieve the resources automatically (might be useful for other projects too).
Once these tools were written and integrated into the build system I could finally make an evaluation as to how successful this exercise had been.
ConclusionsThe main things I learned from this investigation were:
- Internationalisation has a number of complex areas
- Localisation to a specific locale is more than a mechanical process.
- The majority of platforms and services are oriented around textural language translation
- There is a concentration on the gettext mode of operation in many platforms
- Integration to any of these platforms requires both workflow and technical changes.
- At best tools to integrate existing resources into the selected platform need to be created
- Many project will require format conversion tools, necessitating additional developer time to create.
- The social issues within an open source project may require compromise on the workflow.
- The external platform may offer little benefits beyond a pretty user interface.
- External platforms introduce an external dependency unless the project is prepared and able to run its own platform instance.
- Real people are still required to do the translations and verify them.
Overall I think the final observation has to be that integrating translation services is not a straightforward operation and each project has unique challenges and requirements which reduce the existing platforms to compromise solutions.
Occasionally, someone asks me whether we should encourage use of the --ask-cert-level option when certifying OpenPGP keys with gpg. I see no good reason to use this option, and i think we should discourage people from trying to use it. I don't think there is a satisfactory answer to the question "how will specifying the level of identity certification concretely benefit anyone involved?", and i don't see why we should want one. gpg gets it absolutely right by not asking users this question by default. People should not be enabling this option. Some background: gpg's --ask-cert-level option allows the user who is making an OpenPGP identity certification to indicate just how sure they are of the identity they are certifying. The user's choice is then mapped into four levels of OpenPGP certification of a User ID and Public-Key packet, which i'll refer to by their signature type identifiers in the OpenPGP spec: 
Or rather, hello Planet!
Here s a somewhat traditional introductory post.
I m Nicolas Dandrimont, I m French, I m sysadmin in a 
Unrelatedly, 
Hopefully I ll be able to make regular updates on the work I do in Debian and free software, so stay tuned!
besides working on the preparation of the 
The 
One of the most exciting places to view the Matterhorn is from the nearby observatory at 
The scenic train to Zermatt is included in any of the Swiss rail passes, but the train up to Gornergrat is a private railway and a special ticket must be purchased. Discounts are sometimes offered at rail stations in Swiss cities or 
Dear all,
The Linux init systems are a bit in flux at the moment. That is,
they're in flux in Debian; outside Debian, most other
distributions have stepped away from sysvinit and towards something else
(systemd, openrc, or upstart). I've not been a proponent of any switch,
though I understand the reasoning, and it probably makes sense for us to
switch at some point. But yesterday, the fact that this
customer's system was running sysvinit and not systemd or upstart saved
me quite a bit.
There's a server. It has one quadcore processor. For reasons that I
won't go into here, the customer wants an extra quadcore processor to be
added to the system.
After having done so, I power on the system... only to see it power
itself off at some point during boot. I did notice some kernel
messages fly by just moments before the system would power itself off,
but it was impossible for me to read them. So what did I do?
There was an interesting question by Fool recently on the StackExchange site for Boardgames: 
One of my recent innocent tweet about Gerrit vs Github triggered much more
reponses and debate that I expected it to. I realize that it might be worth
explaining a bit what I meant, in a text longer than 140 characters.
I always looked at Github from a distant eye, mainly because I always
disliked their pull-request handling, and saw no value in the social hype it
brings. Why?
One click away isn't one click effort
The pull-request system looks like an incredible easy way to contribute to
any project hosted on Github. You're a click away to send your contribution
to any software. But the problem is that any worthy contribution isn't an
effort of a single click.
Doing any proper and useful contribution to a software is never done right
the first time. There's a dance you will have to play. A slowly rhythmed
back and forth between you and the software maintainer or team. You'll have
to dance it until your contribution is correct and can be merged.
But as a software maintainer, not everybody is going to follow you on this
choregraphy, and you'll end up with pull-request you'll never get finished
unless you wrap things up yourself. So the gain in pull-requests here, isn't
really bigger than a good bug report in most cases.
This is where the social argument of Github isn't anymore. As soon as you're
talking about projects bigger than a color theme for your favorite text
editor, this feature is overrated.
Contribution rework
If you're lucky enough, your contributor will play along and follow you on
this pull-request review process. You'll make suggestions, he will listen
and will modify his pull-request to follow your advice.
At this point, there's two technics he can use to please you.
Technic #1: the Topping
Github's pull-requests invite you to send an entire branch, eclipsing the
fact that it is composed of several commits. The problem is that a lot of
one-click-away contributors do not masterize Git and/or do not make efforts
to build a logical patchset, and nothing warns them that their branch
history is wrong. So they tend to change stuff around, commit, make a
mistake, commit, fix this mistake, commit, etc. This kind of branch is
composed of the whole brain's construction process of your contributor, and
is a real pain to review. To the point I quite often give up.
<figure>
<figcaption>
A typical case: 3 commits to build a 4 lines long file.
</figcaption>
</figure>
Without Github, the old method that all software used, and that many
software still use (e.g. Linux), is to send a patch set over e-mail (or any
other medium like Gerrit). This method has one positive effect, that it
forces the contributor to acknowledge the list of commits he is going to
publish. So, if the contributor he has fixup commits in his history, they
are going to be seen as first class citizen. And nobody is going to want to
see that, neither your contributor, nor the software maintainers. Therefore,
such a system tend to push contributors to write atomic, logical and
self-contained patchset that can be more easily reviewed.
Technic #2: the History Rewriter
This is actually the good way to build a working and logical patchset using
Git. Rewriting history and amending problematic patches using the famous
git rebase --interactive trick.
The problem is that if your contributor does this and then repush the branch
composing your pull-request to Github, you will both lose the previous
review done, each time. There's no history on the different versions of the
branch that has been pushed.
In the old alternative system like e-mail, no information is lost when
reworked patches are resent, obviously. This is far better because it eases
the following of the iterative discussions that the patch triggered.
Of course, it would be possible for Github to enhance this and fix it, but
currently it doesn't handle this use case correctly..
<figure>
It's not a secret for anyone that I've been contributing to OpenStack as a
daily routine for the last 18 months. The more I contribute, the more I like
the contribution workflow and process. It's already
Some weeks ago, I contacted Rosa Mart nez, a tech journalist with some questions regarding what I regarded as 
Congrats to the Debian release team on the