The open-source chess program Stockfish recently climbed to the top of the TCEC rating lists. Even though it does not seem likely to win the actual TCEC tournament (it is 15.5 11.5 down in the 48-game superfinal against Komodo, a closed-source commercial program, after winning 4, losing 8 and drawing 15), this is a tremendeous achievement. This is not only because it is highly unusual that the strongest chess program is open-source (it's naturally hard to keep a competitive edge when everybody can see your secrets!), but also because it is a mere 9000 lines of C++ code. (This is without a GUI; it interfaces through a standardized protocol known as UCI.) In comparison, Crafty, a far weaker open-source program, is about four times that, GNU Chess is above 40000, and I seem to recall that the Deep Blue program was something like 65000 lines (in addition to, of course, all the code for the ASIC chess chips it contained, and stood for the majority of its chess strength!). So, the development team is really to be commended. It should also be said that TCEC is testing the latest beta versions of most of these engines and while Stockfish is developed in the open, on GitHub, the latest versions of e.g. Komodo and Houdini are private. (This is probably one of the reasons why Stockfish is slightly lower on e.g. the CCRL list, which also computes chess program rating lists by playing games between them.) Now, why does anyone care? Only a very small handful of players would be able to score even a draw against any of these programs in a 10-game match, so does it really matter which one we choose? Or is this some obscure pissing contest? The answer, of course, is that these engines are not primarily useful for competitive play; they are useful for analysis. Almost any player, right down from beginners to the very top players, can benefit from having someone stronger than them look at their games (or others' games!) and say where things went wrong. (Somehow analyzing losses always seem more instructive than analyzing wins. Nobody ever got better by winning, as the saying goes.) And the better a program is at playing chess, the more likely is its analysis to be correct. Of course, for the weaker players, a chess coach is likely to produce much more understandable guidance ( this creates a weak square on c7 is much more palatable than after these 18 moves here, white is 0.48 pawns stronger than you, so this was a bad move , and engines produce the latter sort of output), but they are always available, incredibly strong, and cost much less than a coach. :-) They also add a lot to the otherwise quite hard-to-comprehend activity of watching high-level human players; while the computers do not understand every kind of position equally well, they at least provide some sort of objective verdict on which player has the upper hand, something which can be really hard to discern unless you have very strong players around for commentary. So, in trying to improve my skills a bit, I've been analyzing my games with engines for a while, and recently I've switched from Rybka to Stockfish. (Rybka, a closed-source program, used to be the clear leader of the pack, but after a huge amount of drama involving accusations of who has copied code from who, it seems no longer to be actively developed, and Stockfish is now somewhat stronger.) This, unfortunately, brings me to the things I don't like too much about Stockfish; it seems to behave more erratically than Rybka did in interactive analysis mode. Its score jumps a lot more around when I move something (worse at remember the analysis from the previous position?), and sometimes, it changes its mind a lot when I move back and forth between moves maybe the huge selectivity that makes chess engines so strong these days also make them behave less usefully when only looking at a move for a few seconds. So, even though Stockfish is a fantastic chess engine, and will only get better in the future, maybe I'll go back to Rybka at some point. But then again, maybe I should then check out Komodo or any of the other engines out there or maybe talk to someone who actually knows chess, that would help a lot, too =)

Background I upgraded from Linux 3.8 to 3.11 among with newer Mesa, X.Org and Intel driver recently and I found a small workaround was needed because of upstream changes.

The upstream change was the Add "Automatic" mode for "Broadcast RGB" property, and defaulting to the Automatic. This is a sensible default, since many (most?) TVs default to the more limited 16-235, and continuing to default to Full from the driver side would mean wrong colors on the TV. I've set my screen to support the full 0-255 range available to not cut the amount of available shades of colors down.

Unfortunately it seems the Automatic setting does not work for my HDMI input, ie blacks become grey since the driver still outputs the more limited range. Maybe there could be something to improve on the driver side, but I'd guess it's more about my 2008 Sony TV actually having a mode that the standard suggests limited range for. I remember the TV did default to limited range, so maybe the EDID data from TV does not change when setting the RGB range to Full.

I hope the Automatic setting works to offer full range on newer screens and the modes they have, but that's probably up to the manufacturers and standards.

Below is an illustration of the correct setting on my Haswell CPU. When the Broadcast RGB is left to its default Automatic setting, the above image is displayed. When set to Full, the image below with deeper blacks is seen instead. I used manual settings on my camera so it's the same exposure.


Workaround For me the workaround has evolved to the following so far. Create a /etc/X11/Xsession.d/95fullrgb file:

if [ "$(/usr/bin/xrandr -q --prop   grep 'Broadcast RGB: Full'   wc -l)" = "0" ] ; then
/usr/bin/xrandr --output HDMI3 --set "Broadcast RGB" "Full"
fi

And since I'm using lightdm, adding the following to /etc/lightdm/lightdm.conf means the flicker only happens once during bootup:

display-setup-script=/etc/X11/Xsession.d/95fullrgb

Important: when using the LightDM setting, enable executable bits (chmod +x) to /etc/X11/Xsession.d/95fullrgb for it to work. Obviously also check your output, for me it was HDMI3.

If there is no situation where it'd set back to "Limited 16:235" setting on its own, the display manager script should be enough and having it in /etc/X11/Xsession.d is redundant and slows login time down. I think for me it maybe went from 2 seconds to 3 seconds since executing xrandr query is not cheap.
Misc Note that unrelated to Full range usage, the Limited range at the moment behaves incorrectly on Haswell until the patch in bug #71769 is accepted. That means, the blacks are grey in Limited mode even if the screen is also set to Limited.

I'd prefer there would be a kernel parameter for the Broadcast RGB setting, although my Haswell machine does boot so fast I don't get to see too many seconds of wrong colors...

I happens quite often that I install a manually built, newer version of some package on a machine. Occassionally I forget to remove it or to downgrade it to the version in the APT repo.

$ apt-show-versions   fgrep newer

easily finds those packages. But usually when doing such a check, I want this list of packages in my aptitude TUI to have a look at the other versions of that package and to take actions. And I don t want to manually search for each of the package manually. This can be done with the following one-liner :

# aptitude -o "Aptitude::Pkg-Display-Limit=(  apt-show-versions   fgrep newer   awk -F '[ :]' ' printf "~n ^"$1"$   " '   sed -e 's/  *$//'  )"

It uses apt-show-version s output, searches for the right packages, takes the first column and transforms it into an aptitude search pattern matching all packages whose name is exactly one of the listed packages. But this solution is quite ugly and slow. So I wondered if this is also doable with pure aptitude search patterns which likely would also be faster. And after some playing around I found the following working aptitude search term:

~i ?any-version(!~O.) !~U !~o

This matches all packages which which are installed and which have a version which has no origin, i.e. no associated APT repository. Since this also matches all hold packages as well as all packages not available in any archive, I use !~U !~o to exclude those packages from that list again. Since nobody can remember that nor wants to type that everytime needed, I added the following alias to my setup:

alias aptitude-newer-than-in-archive='aptitude -o "Aptitude::Pkg-Display-Limit=~i ?any-version(!~O.) !~U !~o"'

Only caveat so far: It seems to also match packages from APT repos which haven t set an Origin . This should not happen with any Debian or Ubuntu APT repository, but seems to happen occasionally with privately run APT repositories. And using ~A instead of ~O, i.e. ~i ?any-version(!~A.), does not work for this case either, despite it matches installed packages of which versions not in any available archive exist. But unfortunately aptitude seems to remember in some way if a package was in some archive in the past, so this only shows packages installed with dpkg -i, but not packages removed from e.g. unstable but with older versions still being available in stable.

Yesterday I asked myself, how to setup a crossbuild environment on a Debian wheezy/amd64 to build binary packages compatible for Raspbian. After digging around it seemed to be the easiest way to use mk-sbuild to setup such a build environment.
We just need to install sbuild (>= 0.64.0-1) and ubuntu-dev-tools (>= 0.146), both packages are available since jessie:

aptitude install sbuild ubuntu-dev-tools

Some more packages are needed for crossbuilding:

aptitude install qemu-user-static binfmt-support linux-image-amd64

Setting up the chroot is quite easy with:

mk-sbuild --arch=armhf --debootstrap-mirror=http://archive.raspbian.org/raspbian jessie

Unfortunately you get thrown an error about bad signing key:

Release signed by unknown key (key id 9165938D90FDDD2E)

This happens cause debootstrap is using per default /usr/share/keyrings/$ DISTRO -archive-keyring.gpg, which doesn't ship the Raspbian signing key indeed. After looking how to solve that problem, I decided to use a quik&dirty fix:

echo 'DEBOOTSTRAP_KEYRING="--keyring=/usr/share/keyrings/raspbian-archive-keyring.gpg"' >> \
    /root/.mk-sbuild.rc
cp -a /usr/bin/mk-sbuild /tmp/mk-sbuild
patch -p0<./mk-sbuild_raspbian.diff /tmp/mk-sbuild

mk-sbuild_raspbian.diff Now you should be able to setup your Raspbian sbuild chroot via:

/tmp/mk-sbuild --arch=armhf --debootstrap-mirror=http://archive.raspbian.org/raspbian wheezy

Earlier this year the blog had its tenth anniversary. I had meant to celebrate the occassion by revamping the site and blog a little. Having set up the updated R/Finance site, the Rcpp Gallery and Rcpp sites as well as the much-needed overhaul of the html side of the CRANberries RSS feed (which also integrates it with the static blog compiler I use), I figured I just needed a little time to get this done. Well, eight months later we're there. It still uses Twitter Bootstrap for layout, and a slightly modified Bootswatch theme. Comments welcome, and please let me know if links are missing or going nowhere in places.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

After many years break from the package and a vain hope that development would be continued by someone else, I finally pulled my acts together this morning and wrapped up a new release of chrpath, the command line tool to modify the rpath and runpath of already compiled ELF programs. The update was triggered by the persistence of Isha Vishnoi at IBM, which needed a new config.guess file to get support for the ppc64le architecture (powerpc 64-bit Little Endian) he is working on. I checked the Debian, Ubuntu and Fedora packages for interesting patches (failed to find the source from OpenSUSE and Mandriva packages), and found quite a few nice fixes. These are the release notes: New in 0.15 released 2013-11-24:

• Updated config.sub and config.guess from the GNU project to work with newer architectures. Thanks to isha vishnoi for the heads up.
• Updated README with current URLs.
• Added byteswap fix found in Ubuntu, credited Jeremy Kerr and Matthias Klose.
• Added missing help for -k --keepgoing option, using patch by Petr Machata found in Fedora.
• Rewrite removal of RPATH/RUNPATH to make sure the entry in .dynamic is a NULL terminated string. Based on patch found in Fedora credited Axel Thimm and Christian Krause.

You can download the new version 0.15 from alioth. Please let us know via the Alioth project if something is wrong with the new release. The test suite did not discover any old errors, so if you find a new one, please also include a testsuite check.

Although Don Norman was discussing designers attitude to user errors I assert the same is true for programmers when we use static program analysis tools.

The errors, or rather defects in the jargon, that a static analysis tools produce can be considered low cost well formed bug reports available very early in the development process.

When I say low cost it is because they can be found by a machine without a user or fellow developer wasting their time finding them. Well formed comes because the machine can describe exactly how it came to the logical deduction leading to the defect.
IntroductionStatic analysis is in general terms using the computer to examine a program for logic errors beyond those of pure syntax before it is executed. Examining a running program for defects is known as dynamic program analysis and while a powerful tool in its own right is not the topic of discussion.

This analysis has historically been confined to compiled languages as their compilers already had the Abstract Syntax Tree (AST) of the code available for analysis. As an example the C language (released in 1972) had the lint tool (released in 1979) based on the PCC compiler.

Practical early compilers (I am generalising here as the 19070s were a time of white hot innovation in computing and examples of just about any innovation in the field could probably be found) were pretty primitive and produced executables which were less good than hand written assembler output. Due to practical constraints the progress of optimising compilers was not as rapid as might be desired so static analysis was largely used as an external process.

Before progressing I ought to explain why I just mixed the concept of an optimising compiler and static analysis. The act of optimisation within those compilers requires program analysis, from which they can generate defect reports which we all know and love as compiler warnings, also explaining why many warnings only appear at higher optimisation levels where deeper analysis is required.

The attentive reader may now enquire as to why we would need external analysis tools when our compilers already perform the task. The answer stems from the issue that a compiler is trying to reconcile many desirable traits including:

• Produce correct (expected) output from the source code for the target processor
• Produce output which will execute using the smallest amount of resources possible (not just CPU time but memory access and cache usage)
• Generate output in a reasonable amount of time.
• Have a reasonable cost (both developer time and research into new methods) to implement the compiler itself.
• Produce useful diagnostics

Using static analysisOnce the analysis has been performed and a list of defects generated the main problem with this technology rears its ugly head, that of so called "false positives". The analysis is fundamentally an undecidable problem (it is a variation of the halting problem) and relies on algorithms to generate approximate solutions. Because of this some of the identified defects are erroneous.

The level of erroneous defect reports varies depending on the codebase being analysed and how good the analysis tool being used is. It is not uncommon to see false positive rates, even with the best tools, in excess of 10%

Good tools allow for this and provide ways to supply additional context through model files or hints in the source code to suppress the incorrect defect reports. This is analogous to using asserts to explicitly constrain variable values or a type cast to suppress a type warning.

Even once the false positives have been dealt with there comes the problem of defects which while they may be theoretically possible take so many steps to achieve that their probability is remote at best. These defects are often better categorized as a missing constraint and the better analysis tools generate fewer than the more naive implementations.

An issue with some defect reports is that often defects will appear in a small number of modules within programs, generally where the developers already know the code is of poor quality, thus not adding useful knowledge about a project.

As with all code quality tools static analysis can be helpful but is not a panacea code may be completely defect free but still fail to function correctly.
Defect DensityA term that is often used as a metric for code quality is the defect density. This is nothing more than the ratio of defect to thousands of lines of code e.g. a defect density of 0.9 means that there is approximately one defect found in every 1100 lines of code.

The often quoted industry average defect density value is 1, as with all software metrics this can be a useful indicator but should not be used without understanding.

The value will be affected by improvements in the tool as well as how lines of code are counted so is exceptionally susceptible to gaming and long term trends must be treated with scepticism.
Practical examplesI have integrated two distinct static analysis tools into the development workflow for the NetSurf project which I shall present as case studies. These examples show a good open source solution and a commercial offering highlighting the issues with each.

Several other solutions, both open source and commercial, exist many of which have been examined and discarded as either impractical or proving less useful than those selected. However the investigation was not comprehensive and only considered what was practical for the project at the time.
clangThe clang project is a frontend to the LLVM project providing an optimising compiler for the C, C++ and objective C languages. As part of this project the compiler has been enhanced to run a collection of "checkers" which implement various methods of analysis on the code being compiled.

The "scan-build" tool is provided to make the using these features straightforward. This tool generates defect reports as a series of html files which show the analysis results.


Because the scan-build takes in excess of eight minutes on powerful hardware the NetSurf developers are not going to run this tool themselves as a matter of course. To get the useful output without the downsides it was decided to integrate the scan into the CI system code quality checks.

Whenever a git commit happens to the mainline branch and the standard check build completes successfully on all target architectures the scan is performed and the results are published as a list of defects.

The list is accessible directly through the CI interface and also incorporates a trend graph showing how many defects were detected in each build.

Each defect listed has a detail link which reveals the full analysis and logic necessary to cause the defect to occur.

Unfortunately even NetSurf which is a relatively small piece of software (around 200,000 lines of code at time of writing) causes 107 defects to be emitted by scan-build.

All but 25 of the defects are however "Dead Store" where the code has a value assigned but is never checked. These errors are simply not interesting to the developers and are occurring in code generated by a tool.

Of the remaining defects identified the majority are false positives and several (like the example in the image above) are simply improbable requiring a large number of steps to reach.

This shows up the main problem with the scan-build tool in that there is no way to suppress certain checks, mark defects as erroneous or avoid false positives using a model file. This reduces the usefulness of these builds because the developers all need to remember that this list of defects is not relevant.

Most of the NetSurf developers know that the project currently has 107 outstanding issues and if a code change or tool improvement were to change that value we have to manually work through the defect list one by one to check what had changed.
CoverityThe coverity SAVE tool is a commercial offering from a company founded in the Computer Systems Laboratory at Stanford University in Palo Alto, California. The results of the original novel research has produced a good solution which improved on analysis tools previously available.

The company hosts a gratis service for open source projects, they even provide scans for the Linux kernel so project size does not appear to be an issue.

The challenges faced integrating the coverity tool into the build process differed from clang however the issue of execution time remained and the CI service was used.

The coverity scanning tool is a binary executable which collects data on the build which is then submitted to the coverity service to be analysed. This tool obviously relies upon the developer running the executable to trust coverity to some degree.

A basic examination of the binary was performed and determined the executable was not establishing network connections or performing and observably undesirable behaviour. From this investigation the decision was made that running the tool inside a sandbox environment on a CI build slave was safe. The CI system also submits the collected results in a compressed form directly to the coverity scan service.

Care must be taken to only submit builds according to the services Acceptable Use Policy which limits the submission frequency of NetSurf scans to every other day. To ensure the project stays within the rules the build performed by the CI system is manually controlled and confined to a subset of NetSurf developers.

The results are presented using the coverity connect web technology based defect management tool. Access to the coverity connect interface is controlled by a user management system which precludes publicly publishing the results within the CI system.

Unfortunately NetSurf itself does not currently have good enough JavaScript DOM bindings to support this interface so another browser must be used to view it.

Despite the drawbacks the quality of the analysis results is greatly superior to the clang solution. The false positive rate is very low while finding many real issues which had not been previously detected.

The analysis can be enhanced by use of collection configuration and modelling files which remove intended constructions from consideration reducing the false positive rate to very low levels. The ability to easily and persistently suppress false positives through the web interface is also available.

The false positive management capabilities coupled with a user interface that makes understanding the defect path simple make this solution very practical and indeed the NetSurf developers have removed over 50 actual issues within a relatively short period since the introduction of the tool.

Not all of those defects could be considered serious but they had the effect of encouraging deeper inspection of some very dubious smelling source.
ConclusionsThe principle conclusions of implementing and using static analysis have been:

• It is a powerful tool which aids programmers in improving their software.
• It is not a panacea and bad code can have no defects.
• It can suggest possible defects early in the development cycle.
• It can highlight possibly problematic areas well before they affect a programs users.
• The tool and the infrastructure around it have a large impact on the usefulness of the results.
• The way results are presented has disproportionately significant impact on the usability of the defect reports.
• The open source tools are good, and improving, but coverity currently provides a superior experience.
• Integration into a projects CI system is beneficial.

Review: The Pathway to Awesomeness, by Mark Forster

Publisher:	Hyperink
Copyright:	October 2013
ASIN:	B00FXGXMPQ
Format:	Kindle
Pages:	97

Mark Forster is probably my favorite time management (or, if you prefer, attention management) writer. The place to start with Forster if you're not familiar with him is Do It Tomorrow, not here. Unlike the other books by him I've reviewed, this is just an edited collection of blog posts presented in a convenient ebook form, without even an ISBN number. But if you've read his books (at least the ones that aren't out of print and cost a fortune), this is a convenient way to not have to trawl back through his blog and read the articles in a more convenient fashion. For me, it was also an easy way to give him a bit of money for his blog. Unless you have a similar motive, I don't think the content to price ratio is worth it. But I think everything is still up at Get Everything Done in case anything here strikes your fancy. As with most blog compilations, The Pathway to Awesomeness (I really hate that title) is composed of a whole bunch of fairly short articles, rarely more than three pages each, vaguely collected by topic. The first few are rather general and focused on the normal self-help platitudes that try to build up your sense that you're in control of your own life. But Forster doesn't take too long to get to what he's best at: short ideas, mental tools, interesting tricks, and simple techniques. Even more so than his books, due to the nature of this collection, there's a huge variety here, which means you can probably find something that strikes a chord no matter what attention management problem you've been running into. That's one of the main reasons why I prefer his writing: he tries to provide a giant toolbox instead of a single, streamlined System For Everyone. The other thing I like about Forster is that he is much more blunt than many authors about the biggest problem I have with attention management: trying to do more than anyone could possibly do given the available time. There are some more essays here about that, and several of them are quite good. I think my favorite is the short essay in which he points out how often people decide to do some task for a particular reason, and then get so engrossed in the performance of that task that they forget the original reason, risking sinking far more energy into that task than is actually warranted. Like a lot of Forster's observations, it's obvious in retrospect, but he has a way of phrasing those observation so that they shake the right thing loose in my head and let me re-evaluate something I was doing. Regular readers of Forster's blog know that he loves playing around with different simple to-do list management systems. Unfortunately, none of those are here, so don't pick this up expecting a collection of articles about Superfocus, Autofocus, etc. It makes sense given the topic of this collection, which is more general advice rather than systems, but this collection is so short that I wish he'd added a section on system development. I would have enjoyed reading a collection of all of his miniature systems together in one place and showing how he tweaked and changed them over time, particularly if he'd added some discussion of why he made each change and what goals he was trying to meet with each system. Ah well. If you aren't already familiar with Forster, there's no point in this, but if you are, I got a few new tidbits out of it and will probably skim through it a few more times. Buying it is mostly an exercise in putting money in the tip jar, and his writing is much less polished here (and more prone to cliches), but I still enjoyed reading through it. Rating: 7 out of 10

A few things for which I'd been waiting for quite some time were released, so there was another book order. Allie Brosh Hyperbole and a Half (graphic novel)
Jacqueline Carey Dark Currents (sff)
Debra Dunbar A Demon Bound (sff)
Nicola Griffith Hild (historical)
Ursula K. Le Guin The Unreal and the Real, Volume 1 (sff)
Ursula K. Le Guin The Unreal and the Real, Volume 2 (sff)
Ken Stern With Charity for All (non-fiction) I'm trying to buy fewer books given how many I have and haven't read, and also because I've been on a gaming tear recently and there are only so many hours in the day. Unfortunately, playing a lot of video games means reading fewer books. So I'm trying for quality rather than quantity, and this particular order has one of the best densities of hopefully good books as anything I've ordered in a while. For those who aren't familiar with Allie Brosh, Hyperbole and a Half is one of the most brilliant blogs that I've ever read, and this is a collection plus new stories and new illustrations. I specifically got the hardcover, which is beautiful and very high-quality. I've been waiting for Hild for years since Griffith started talking about writing it on her blog, and now I have the lovely hardcover in my hands. That's probably going to be Christmas reading. The Le Guin is a two-volume hardcover collection of Le Guin's personal favorites from her large lifetime oeuvre of short fiction.

This is part 6 in a series on organizational design and growth.

The change from a business that the owner-entrepreneur can run with helpers to a business that requires management is a sweeping change. [...] One can compare the two kinds of business to two different kinds of organism: the insect, which is held together by a tough, hard skin, and the vertebrate animal, which has a skeleton. Land animals that are supported by a hard skin cannot grow beyond a few inches in size. To be larger, animals must have a skeleton. Yet the skeleton has not evolved out of the hard skin of the insect; for it is a different organ with different antecedents. Similarly, management becomes necessary when an organization reaches a certain size and complexity. But management, while it replaces the hard-skin structure of the owner-entrepreneur, is not its successor. It is, rather, its replacement. Peter Drucker

What it means Management is the art of enabling people to cooperate in achieving shared goals. I ve written elsewhere about what management is not. Management is a multifaceted discipline which is centered on people and the environment in which they work. Why it s important In very small organizations, management can be comparatively easy, and happen somewhat automatically, especially between people who have worked together before. But as organizations grow, management becomes a first-class concern, requiring dedicated practice and a higher degree of skill. Without due attention to management, coordination becomes excessively difficult, working systems are outgrown and become strained, and much of the important work described in this series just won t happen. Management is part of the infrastructure of the organization, and specifically the part which enables it to adapt and change as it grows. Old Status Quo People generally just do stuff , meaning there is little conscious understanding of the system in which people are working. If explicit managers exist, their jobs are poorly understood. Managers themselves may be confused or uncertain about what their purpose is, particularly if they are in such a role for the first time. The organization itself has probably developed more through accretion than deliberate design. New Status Quo People work within systems which help coordinate their work. These systems are consciously designed, explicitly communicated, and changed as often as necessary. Managers guide and coordinate the development and continuous improvement of these systems. The role of managers in the organization is broadly understood, and managers receive the training, support and coaching they need to be successful. Behaviors that help

• It can be helpful to bring more experienced managers into the organization at this stage, especially if there isn t much management experience in house.
• Show everyone in the organization (including managers themselves) what managers do and why it matters.
• Consider very carefully whether someone should become a manager.
• If someone does take on a management role, treat this as a completely new job, which requires handing off their existing responsibilities and learning a new discipline. Don t treat it as just an extension of their work. Write a new job description and discuss it up front.

Obstacles that stand in our way

• Management misbeliefs
• Granting promotions to management roles as rewards for performance
• Many people, when they experience what management work is like, don t enjoy it and aren t motivated by it. It can be hard to predict when this will be the case, and people can feel trapped in a management role that they don t want. Make sure there are mechanisms to gracefully transition out of roles that don t fit for the people holding them.

Conrad released a bug fix version 3.920.3 for his Armadillo library. So I prepared a new release 0.3.920.3 of RcppArmadillo which is nor on on CRAN and also already in Debian. Note that we skipped interim version .2 which addressed an Atlas linking issue which doesn't affect us in the R context as we get LAPACK and BLAS via the R process we get used from. The complete list of changes is below.

Changes in RcppArmadillo version 0.3.920.3 (2013-11-20)

• Upgraded to Armadillo release Version 3.920.3
  • fix for handling of tiny matrices by .swap()

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Tor is a proxy server which allows its users to hide their IP address from the websites they connect to. In order to provide this level of anonymity however, it introduces latency into these connections, an unfortunate performance-privacy trade-off which means that few users choose to do all of their browsing through Tor. Here are a few things that I have found work quite well through Tor. If there are any other interesting use cases I've missed, please leave a comment!

Tor setup There are already great docs on how to install and configure the Tor server and the only thing I would add is that I've found that having a Polipo proxy around is quite useful for those applications that support HTTP proxies but not SOCKS proxies. On Debian, it's just a matter of installing the polipo package and then configuring it as it used to be recommended by the Tor project.

RSS feeds The whole idea behind RSS feeds is that articles are downloaded in batch ahead of time. In other words, latency doesn't matter. I use akregator to read blogs and the way to make it fetch articles over Tor is to change the KDE-wide proxy server using systemsettings and setting a manual proxy of localhost on port 8008 (i.e. the local instance of Polipo). Similarly, I use podget to automatically fetch podcasts through this cron job in /etc/cron.d/podget-francois:

0 12 * * 1-5 francois   http_proxy=http://localhost:8008/ https_proxy=http://localhost:8008/ nocache nice ionice -n7 /usr/bin/podget -s

Prior to that, I was using hpodder and had the following in ~/.hpodder/curlrc:

proxy=socks4a://localhost:9050

GnuPG For those of us using the GNU Privacy Guard to exchange encrypted emails, keeping our public keyring up to date is important since it's the only way to ensure that revoked keys are taken into account. The script I use for this runs once a day and has the unfortunate side effect of revealing the contents of my address book to the keyserver I use. Therefore, I figured that I should at least hide my IP address by putting the following in ~/.gnupg/gpg.conf:

keyserver-options http-proxy=http://127.0.0.1:8008

However, that tends to makes key submission fail and so I created a key submission alias in my ~/.bashrc which avoids sending keys through Tor:

alias gpgsendkeys='gpg --send-keys --keyserver-options http-proxy=""'

Instant messaging Communication via XMPP is another use case that's not affected much by a bit of extra latency. To get Pidgin to talk to an XMPP server over Tor, simply open "Tools Preferences" and set a SOCKS5 (not Tor/Privacy) proxy of localhost on port 9050.

GMail Finally, I found that since I am running GMail in a separate browser profile, I can take advantage of GMail's excellent caching and preloading and run the whole thing over Tor by setting that entire browser profile to run its traffic through the Tor SOCKS proxy on port 9050.

Review: Wanderlust, by Ann Aguirre

Series:	Sirantha Jax #2
Publisher:	Ace
Copyright:	September 2008
ISBN:	0-441-01627-8
Format:	Mass market
Pages:	312

Wanderlust is the second book in the Sirantha Jax series and a direct sequel to Grimspace. There's no point in reading them out of order. The new shape of Sirantha's universe is still coming together after the climactic events of Grimspace, but they seem to involve a new job. In politics, even, but one for which she's uniquely qualified. Before she can even accept it, she finds out that multiple parties are interested in her taking this position, for widely varying reasons. But she starts her new position by taking a detour to visit some old friends (and the planet I liked least in Grimspace's settings), which turns into a war and a comprehensive distraction that lasts the rest of the book. The biggest problem with Wanderlust (and unfortunately there are several) is that it doesn't feel like a complete book. The first few chapters set up what appears to be the next step of Sirantha's adventures, as well as establishing a new villain, and then the rest of the book is an extended detour. She does learn more about the new villain, but almost everything that's set up at the start of this book is still unresolved at the end of it. The detours are moderately entertaining and (at least after an initial decision based on unfinished business from the first book) are at least related to the plot, but this feels like the setup and middle-book distractions of a book twice as long. I came away wondering why all of that was really necessary and why I had to wait to the next book to read the actual plot. The second problem with this book is the whining. And the angst. And the self-esteem issues. And did I mention the whining? Okay, I get that one of the distinguishing elements of this series is that it's space opera with an urban fantasy first-person perspective, and the latter usually comes with extensive angst and insecurity about one's love life. But, really, I am not interested in Sirantha complaining about her appearance yet again, and I am particularly not interested in the obnoxiously tedious "I love you and therefore I must push you away" dance that both she and the male romance lead do through half this book, and only stop doing because they can't any more. I realize that self-destructiveness is a thing, and that people do react this way, but I don't want to read about it, and I particularly don't want to listen to the narrator whine about how hard it all is for much of the book. Some of the appeal of Sirantha is that she's fragile, paranoid, and all sorts of messed up, but she muddles through and manages to do the right thing anyway. However, for me, the key word there is "do." When she's twitchy and angry and sarcastic and scared but doing something (which does also happen in this book), I really enjoy this series. When she's living in her own head and building elaborate plans for how she's going to destroy her life this time, it starts getting tiresome. It doesn't help that I don't see what she sees in her romantic partner (at least most of the time; he has his moments), and therefore I have even less sympathy for her grand self-sacrificing gestures. Thankfully, there are good parts too. Aguirre introduces a few new characters, all of whom I liked. Sirantha is assembling quite the ragtag band of misfits, and they're misfits in all the right sorts of ways. They play off each other well and they're full of plot hooks. We also get an amusing twist on Sirantha's relationship with her personal assistant AI, which, while bizarrely tangential to the plot, looks like it will hold a lot of promise for the future. And I love Vel, who made the end of Grimspace and who continues to be a highlight of these books, partly because he doesn't throw emotional temper tantrums or indulge in bizarre silent treatments. I always love the quietly competent characters in stories like this. This is not up to Grimspace, but hopefully it's an aberration. It's not bad, exactly, but it went off the rails. Thankfully, the next book promises quite a lot of Vel, which will hopefully be an improvement. Followed by Doubleblind. Rating: 6 out of 10

We ve been trying to build Mono on MIPS in Debian for a long time. Just under a decade, in fact. Mono 0.29.99.20040114-3 was the first attempt, back when the Mono source was 10 meg, not today s ~80. It never worked though. Not once. At the end of 2004, 10 upstream versions later, we gave up, and turned off mipsel as a target architecture in the package. When I became a Debian Developer in 2009, one of the first things I did was try again with Mono on various previously unsupported architectures, by running a build on the Debian porterboxes. Supposedly MIPS support was there, as evidenced by all the MIPS-related files in the source tree. But I d try it, and it d fail, and upstream would say works for me . I repeat the process every major release or so, to see what s changed. The breakthrough Spurred by the removal of two bitrotted architectures (IA64 and SPARC), I decided to try again, this time investigating more deeply into the reason for the failures, with the help of upstream developer Alex R nne Petersen. A couple of hours of IRC-driven gdb and test program disassembly later, a seemingly innocent comment flagged something in my brain: 09-07-2013 15:10:11 < alexrp!~zor@baldr.rfw.name: TIL that mul and mult are not the same thing on mips Why is this notable? Well, MIPS processors lack a whole bunch of instructions which are commonly used in assembler. MUL is one of them it s valid MIPS assembler, which is expanded to MULT/MFLO when compiling. Call it a macro, or a mnemonic, or shorthand the preferred term is pseudoinstructions . So what s the issue? 09-07-2013 15:18:27 > directhex: mono isn't trying to use a mul instruction, right? i mean, that instruction doesn't exist as far as the cpu is concerned, it's a macro the compiler does things with See where this is headed? 09-07-2013 15:23:59 > directhex: mini/mini-mips.c:#define USE_MUL 1 /* use mul instead of mult/mflo for multiply */ Argh. See, the thing about MIPS pseudoinstructions is they may be real instructions on a given CPU implementation. Strictly speaking MUL isn t a standard instruction, but a given CPU might have it anyway, to make multiplication a little faster (by using only one instruction, not two, for multiplication). In this case, the Debian MIPS infrastructure is based around ICT Loongson-2E processors which don t have that extension but the upstream Mono developers were building and testing on an extended CPU, never seeing the issue themselves. Flipping that define to 0 (and amending the instruction length setting in another file) fixed the build. Mono was running on MIPS for me for the first time ever. Digging through the history in git showed just how annoying this implementation quirk was. USE_MUL was added in late 2008 replacing a previously used #if 1 . The mult/mflo version of the code existed in the Mono source since the first time the full MIPS port was committed in 2006, but we never saw it. The breakage So, with that patched to work, I added mipsel to the Experimental build of Mono which still failed. The runtime would build fine, but the class library build would fail at random times, with random meaningless stack traces. Unrepeatable. Some kind of race condition. The build would eventually succeed if I hammered make a few times, but that s no good for the Debian build daemons. Back to square one except I had an epiphany yesterday. I have heard more than once that Loongson processors are missing a few instructions. What if one of those was being hit, intermittently? I started doing a search in places that might need to work around that kind of issue, and found this. A patch to binutils in 2009, replacing one no-op instruction with another, when /usr/bin/as is fed the -mfix-loongson2f-nop flag. Turns out NOP is another pseudoinstruction on MIPS. Well, more of an alias. The opcode 0 000000 is Shift Left Logical with 0 registers and 0 data, which is a no-op. But on all but the latest generation of Loongson-2F chips, that opcode can, under heavy load, fail causing inconsistent state in the CPU registers. The flag to as replaces sll 0,0,0 with or $at,$at,0 , which is also a no-op instruction, but doesn t trigger the failure on Loongson-2F chips (and 2E chips, although that s not stated in the documentation). As long as ALL your programs get fed through as , you don t have a problem, since it uses the replacement opcode but what if you use a JITter to generate your own opcodes? Oh fuck, it couldn t be diff --git a/mono/arch/mips/mips-codegen.h b/mono/arch/mips/mips-codegen.h
index dc4df7d..1dbd1c6 100644
--- a/mono/arch/mips/mips-codegen.h
+++ b/mono/arch/mips/mips-codegen.h
@@ -334,7 +334,7 @@ enum
/* misc and coprocessor ops */
#define mips_move(c,dest,src) mips_addu(c,dest,src,mips_zero)
#define mips_dmove(c,dest,src) mips_daddu(c,dest,src,mips_zero)
-#define mips_nop(c) mips_sll(c,0,0,0)
+#define mips_nop(c) mips_or(c,mips_at,mips_at,0)
#define mips_break(c,code) mips_emit32(c, ((code)<<6) 13)
#define mips_mfhi(c,dest) mips_format_r(c,0,0,0,dest,0,16)
#define mips_mflo(c,dest) mips_format_r(c,0,0,0,dest,0,18) Oh yes it could! Mono was using sll 0,0,0 (the recommended no-op instruction from the MIPS instruction reference manual), causing failures in my tests, because Debian s build and test infrastructure just happens to use defective silicon. And, again, upstream were unable to reproduce a problem because they use better silicon than we do. So what now? Well, last night I uploaded mono_3.2.3+dfsg-3, which includes the above patch to force the replacement no-op instruction. It test built fine on the porterbox, and it should (when the damn experimental buildd gets around to it), just work. Finally. After just under a decade, Mono packages will be available on MIPS in Debian. And after all this time, all we had to change was 4 lines to work around 7 year old Chinese knock-off processors. The edit So, things are finally built. It turns out that despite everything, the replacement NOP opcode is not enough. If you re-read the post to the binutils list, pay close attention to: In theory this is still not enough to fully eliminate possible hangs, but the possiblity is extremely low now and hard to be hit in real code. It s a filthy lie. It s easy to hit the issue in real code: just do a from-source build of the whole Mono class library. With the replacement instruction it builds .NET 2.0, 3.0, 3.5, 4.0, and most of 4.5, before dying in the same way as before an improvement on failing early in the 2.0 build, but not enough. Thankfully, 2 out of the 5 Debian mipsel build servers are not Loongson 2 they re 11 year old Broadcom SWARM developer boards. Not fast but also not broken. Luck smiled on me, and caused my build to go to one of these Broadcom machines. As a result (experimental_mipsel-dchroot)directhex@eder:~$ mono --version head -1
Mono JIT compiler version 3.2.3 (Debian 3.2.3+dfsg-3) It s been a long time coming.

Review: Water Logic, by Laurie J. Marks

Series:	Elemental Logic #3
Publisher:	Small Beer Press
Copyright:	2007
ISBN:	1-931520-23-2
Format:	Trade paperback
Pages:	332

Water Logic is the third book in the Elemental Logic series, starting with Fire Logic. As with Earth Logic, it's best to read all three books in close temporal proximity, since Water Logic offers little recap of what happened in the previous books and relies on a large cast of characters who are not reintroduced. It's also hard to review without spoilers for the previous books. Earth Logic ended with a monumental shift in Shaftal politics, and half of Water Logic is deeply concerned with the implications of that shift and where Shaftal should go from here. Karis and Zanja are, of course, deeply involved, but much of the political story is told from the perspective of the character introduced in Earth Logic, and from the perspective of another earth blood, Meriseth. Water bloods haven't been seen much so far in the series. Water magic is about repeated patterns and the malleability of time, and the plot of Water Logic eventually entangles Zanja in events far in the past of Shaftal. This proceeds in parallel with more of Marks's excellent analysis of the meaning of war and peace, diplomacy, cooperation, multiculturalism, adaptation, and empathy. The two sides of the book do to some extent reinforce each other, but the connections are tricky and less obvious. Parts of this book are excellent. Wars are complicated and powerful, and don't end easily. The mutual incomprehension and mutual (if intermittant) struggle for understanding between the natives of Shaftal and the Sainnites is beautifully presented. I particularly loved Mariseth's approach throughout the book. Earth bloods are perhaps Marks's best invention, and seeing another (besides Karis) at work, with the same stubborn patience but with different tactics and different levels of power, was a delight. I feel like reading, and thinking about, Seth's decisions makes me a better person. As with the previous books, though, parts of Water Logic are a bit boring, or at least don't seem to coherently move in a conclusive direction. Zanja has long journeys across the landscape as well as several adventures after she's grabbed by water logic that seem only distantly connected to the rest of the story. Marks does pull this mostly together in the end, and I could detect other thematic linkages while mulling the book over after finishing it, but the book feels a bit disjointed while reading it. And there are some elements, such as the cowdogs, that are neat bits of world-building but that seem tossed in just because they're neat and don't feel strongly connected to the plot. The other complaint I have, which would probably have been lessened by reading the books closer together, is that the cast is a bit too large. All of Karis's extended family are, of course, still around, which I found more confusing than helpful. Named and important characters keep popping in and then disappearing, with no introduction and no help for the reader who doesn't remember the exact details of their roles from previous books. It's a bit frustrating, and it makes the book feel a bit cluttered, although it does a great job of showing the dynamics of a large and comfortable family. (Marks shows the dynamics of family webs with multiple adults better than any other fantasy author I know.) Karis is a bit more sidelined in this book, casting her in the role of force of nature, and I think that works well. It took me a bit to warm to Zanja again, but by the end of the book I was greatly enjoying her sections. But the star, for me, is Seth. Even though some of her segments should have been the boring parts of the book, since they were mostly concerned with quitodian issues, they were the parts I wanted to read about. I was very touched by the ending, even if I think a bit too much authorial hand-waving was required. There is material left for a sequel at the end of Water Logic, including a villain who has yet to fully appear, but it's currently not clear whether Air Logic will ever materialize. But, as with Earth Logic, Water Logic provides a satisfying ending and leaves the world in a stable enough place that I didn't feel cheated out of the end of the story. I think Fire Logic remains the strongest book of this series, but Water Logic is very much worth reading if you've read the previous books. Unfortunately, the books apparently didn't sell well, so this book is from a small press instead of Tor. You may have to order it on-line instead of finding it a local bookstore. But the whole series is very much worth reading. It's a refreshingly original take on fantasy and is much more thoughtful about human relationships and politics than most fantasy works. Rating: 7 out of 10

In Haskell, lists are ubiquitous. In real-world Haskell, monads are ubiquitous. So it should be an easy task to fill a list with values obtained in a monad, i.e. a list of n random numbers (in the Rand monad), or getting some input from the user, and storing it in a list, until the user chooses to finish the input. The task For now, assume a function getInput :: IO Int; we want to run it 10000 and then have the list of the values returned. Sounds simple and straight forward, but it is actually really hard to get the run-time behaviour that we naively and optimistically expect. The expectation In an imperative setting, we would have a simple counting loop. In the loop, we run getInput, allocate a new cell for the linked list, and append it to the list. Clearly, this

• consumes no stack space and
• at the end of the loop, the list is already fully evaluated in memory, ready to be used.

Can we achieve that with Haskell? Let s look at a few alternatives (complete code available)
Attempt 1: The accumulator I ll start with the probably worst attempt. Unfortunately, from what I have seen from correcting Haskell exams, this is what beginners would write, especially if they have the above imperative algorithm in mind, and know about the convert loops to recursion with accumulators approach to solving Haskell problems:

accumAppend :: Int -> IO [Int]
accumAppend n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput; go (n-1) (l++[x]) 

To check the space usage of this, I pass -Kn to the GHC runtime and see what limit is just enough to let this program calculate accumAppend 10000 and fully evaluate the resulting list. It turns out that we need 328336 bytes of stack space. So what about the second requirement; that the list is fully evaluated? Experienced Haskellers see it immediatelly: The accumulator is not used in the loop, so each iteration adds a thunk around the previous value, that would add the next entry to the end. Only when the list is actually used, these are run, each traversing the whole list, causing quadratic cost. We can verify this using ghc-vis: Clearly (and not surprisingly) this is not the best solution. Attempt 2: The accumulator (forced) Lets try to fix the second issue, by making sure that the list is always fully evaluated. We import Control.DeepSeq and write

accumAppendSeq :: Int -> IO [Int]
accumAppendSeq n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput ; go (n-1) $!! (l++[x]) 

As the accumulator gets fully evaluated before being passed to the next invocation of go, the result will not contain thunks. But note that this has also fixed the stack consumption: The code now runs the smallest setting for -K, 8 bytes. This works as both go and deepseq are tail-recursive. But this is still not the solution we are looking for, as the quadradic cost caused by using ++ is still there. Attempt 3: The accumulator (reversed) Since we know that l++[x] is expensive, but x:l is cheap, we can simply use this to update the accumulator. This has the slightly annoying consequence that the entries of the list are in the reverse order, but we can fix that in the base case:

accumReverse :: Int -> IO [Int]
accumReverse n = go n []
  where
    go 0 l = return (reverse l)
    go n l = do  x <- getInput ; go (n-1) (x:l) 

And indeed, we get noticably faster code that still runs in 8 bytes of stack. Unfortunately, we still don t construct the final list directly. Attempt 4: Naive recursion Maybe we can do without an accumulator. The most naive such code would be

listRecurse :: Int -> IO [Int]
listRecurse n = go n
  where
    go 0 = return []
    go n = do  x <- getInput; r <- go (n-1); return (x:r) 

and I d expect that most experienced Haskellers would write that code (if they are asked not to use a library function). And indeed, this code does create the final list directly. But it requires a large stack or 164616 bytes. The reason is that this is no longer tail recursive: After calling go again we need to take the result and prepend x Attempt 5: library functions Maybe we should simply not worry about the implementation, and use library functions? Clearly, the authors of such libraries have found the best solution. So how does

listReplicateM :: Int -> IO [Int]
listReplicateM n = Control.Monad.replicateM n getInput

fare? Unfortunately, not better than the naive recursion. In fact, the stack space requirement is the same 164616 bytes. The same holds when using special libraries for generating and consuming list, e.g. the Streams module from the vector package:

listStreams :: Int -> IO [Int]
listStreams n = MStream.toList $ MStream.replicateM n getInput

Attempt 6: Difference lists It is time to dig deeper into our box of tricks. What if we have lists with constant time Snoc (a.k.a. appending a new element)? Then Attempt 3 would not require reversing the list. Such lists are provided by difference lists; there are libraries for that, but we can simply implement them using lambdas:

accumDList :: Int -> IO [Int]
accumDList n = go n id
  where
    go 0 l = return (l [])
    go n l = do  x <- getInput; go (n-1) (l . (x:)) 

Indeed, no stack is required (8 bytes are enough). But we are cheating here: We are still not constructing the final list, but rather we combine functions that append elements to lists. So when accumDList returns, what we have in memory, is a thunk and sequence of partial function applications in the wrong order, and evaluating the thunk does basically the same thing as reverse in Attempt 3. It might be a bit faster, is still not satisfying enough: Attempt 7: Unsafe functions Even deepter in the box of tricks, somewhere in the grit and dust where one usually should not reach, there is a function that can help us out here: unsafeInterleaveIO. This allow us to modify the naive recursion in a way that first creates the Cons-cell, and then continues the loop:

listUnsafeInterleave :: Int -> IO [Int]
listUnsafeInterleave n = do  l <- go n; return $!! l 
  where
    go 0 = return []
    go n = do x <- getInput
              r <- unsafeInterleaveIO (go (n-1))
              return (x:r)

The stack consumption is 8 bytes. It is worth noting that the all to deepseq (in $!!) will actually drive the evaluation of go. Also, it guarantees that the unsafe effects of unsafeInterleaveIO are confined to the execution of listUnsafeInterleave, i.e. are actually safe. Run time statistics I ran criterion over the functions (generating lists of length 1000), and found these results:

• accumAppend: 6.5 ms
• accumAppendSeq: 8.2 ms
• accumReverse: 14.0 us
• listRecurse: 8.0 us (same as listReplicateM and listStreams)
• accumDList: 12.0 us
• listUnsafeInterleave: 174.0 us
  

Conclusion There are many ways to write a list-constructing list in Haskell, none are perfect they either fill the heap, or do not construct the list directly (usually requiring a second pass through it), or rely on IO-specific unsafe functions. The naive recursion performs the best, but may cause a stack overflow (note, though that from GHC 7.8, the stack will be unlimited). The next best option seems to be the difference lists
I m still wondering: What would be required from Haskell, GHC or the monads in question to have a fully satisfactory solution here, i.e. one that is as fast as the naive recursion, as stack efficient as the difference list solution, and allocates no thunks, only list cells?

For the last half-year I have used CyanogenMod on an Nexus 4 as my main phone. Recently the touch functionality stopped working on parts of the display, and the glass on the back has started to crack. It seems modern phones are not built to last. For comparison, before the N4 I used a Nokia N900 for around 3 years without any hardware damages (in my drawer now, still working). A few weeks ago I started looking for a replacement. My experience with CyanogenMod had been good, but the number of proprietary blobs on the N4 concerned me. Finding something better wasn t easy though, so I m documenting my experience here. My requirements were, briefly, that I wanted a phone that I could buy locally that had a free software community around it that produced a stable environment. I have modest requirements for things I wouldn t give up on: telephony, data connection (3G), email (IMAP+SMTP), chat (XMPP), and a web browser. I like the philosophy and openness around the Firefox OS but the more I have read about it, it seems unlikely that it would deliver what I need today. In particular none of the devices capable of running Firefox OS appealed to me, and the state of email reading seemed unclear. I m sure I ll revisit Firefox OS as an alternative for me in the future.

As I had been happy with CyanogenMod, but concerned about its freeness, it felt natural to move on and test the more free software friendly project Replicant. Replicant only supports a small number of devices. After talking with people in the #replicant IRC channel, it seemed the Samsung S3 would be a decent choice for me. The Samsung S2 would have worked as well, but it cost almost as much as the S3 where I looked. Despite the large number of Samsung S3 devices out there, it seems the prices even for used devices are high (around 2500 SEK in Sweden, ~380 USD). I ended up buying a brand new one for 3200 SEK (~500 USD) which felt expensive, especially after recalling the recent $199 sale for Nexus 4. Noticing that brand new Nexus 4 devices are still over 3000 SEK in Sweden comforted me a bit. I would have preferred a more robust phone, like the CAT B15, but the state of free software OSes on them seem unclear and I wanted something stable. So, enough about the background, let s get started. Building and installing Replicant on the device was straight forward. I followed the Replicant Samsung S3 Build instructions to build my own images. The only issue I had was that I had not set JAVA_HOME and the defaults were bad; make sure to set JAVA_HOME before building. I built everything on my Lenovo X201 running Debian Wheezy, with OpenJDK 6 as the Java implementation. Installing the newly built firmware was easy, I just followed the installation process documentation. I made sure to take a clockworkmod backup to an external SD card before wiping the old system. To get a really clean new device, I also re-formated /sdcard inside clockworkmod; I noticed there were some traces left of the old system there. I spent about one week testing various configurations before settling on something I could use daily. A fair amount of time was spent looking into backup and restore options for Android devices. My idea was that I would take a backup of the apps I ran on the N4 and transfer them to the S3. The Android Debug Bridge (adb) has a backup/restore command, however it (intentionally) ignores apps marked as allowBackup=false which a number of apps has. It doesn t seem possible to override that settings so much for the freedom to backup your own device. I then discovered oandbackup. It can backup your entire system, saving each app (together with associated data) into a separate directory, for simple review and inspection before restore. You can do batch backups and batch restore. I couldn t get it to automatically restore things, though, which would be neat for really automated re-installations (there is an open issue about this feature). After noticing that some apps did not like being moved from the N4 (running Android 4.2) to the S3 (running 4.0), I ended up installing most apps from scratch on a freshly installed Replicant. I use oandbackup to the external SD card so that I can quickly restore my phone. For backup/restore of SMS/MMS and Call Log, I use SMS Backup+ against my own IMAP server. Camera pictures are synced manually using adb when I am connected to my laptop. There is a number of apps that deserve to be mentioned because they are what I use on a daily basis. All of them come via the free software market F-Droid. For email (IMAP/SMTP), I use K-9 Mail which is feature rich but still easy to use. For chat, I use Xabber. I use NewsBlur s free software app to read RSS flows. For two-factor authentication, I use Google Authenticator. I haven t evaluated different PDF viewers, but the first one I tried (APV PDF Viewer) has worked fine so far. Handling a a synchronized address book and calendar deserve its own blog post because it is a challenging topic, but briefly, I m currently using a combination of aCal and DAVdroid. Finally, since Replicant is still work in progress, some words about stability and notes on what doesn t work. This is probably the most interesting part if you are considering running Replicant on an S3 yourself. Overall system stability is flawless, I hadn t had any crash or problem with the fundamental functionality (telephony, 3G, Camera). People have said graphics feels a bit laggy, but I cannot compare with the stock ROM and it doesn t get in the way of daily use. First some notes about non-free aspects:

• Bluetooth doesn t work by default. After installing /system/vendor/firmware/bcm4334.hcd (MD5 b6207104da0ca4a0b1da66448af7ed4c) pairing and testing with a Bluetooth headset worked fine.
• Front camera doesn t work by default. After installing /system/vendor/firmware/fimc_is_fw.bin (MD5 52eeaf0889bc9206860075cd9b7f80b9and) and /system/vendor/firmware/setfile.bin (MD5 0e6fdeb378fb154c39fd508ae28eaf2a) it works. The extensions are *.bin but I don t believe this code is executed on the main CPU.
• GPS doesn t work by default. After installing /system/bin/gpsd (MD5 6757ed2e2a283259e67c62e6b2b9cfef), /system/lib/libsecril-client.so (MD5 a836df0f947d2aa2ef20dcb213317517), /system/lib/hw/gps.exynos4.so (MD5 1ea1d67f297dd52d59d40dbad9b02a0a) it works. This is code that runs on the main CPU, and even more alarming, it embeds a copy of OpenSSL and talks to various online servers for A-GPS, and is thus a likely (and probably not too challenging) attack vector for anyone wanting to remotely exploit any phone.
• Wifi doesn t work, and I haven t gotten this to work. There is a list of non-free S3 firmware on the Replicant wiki however my stock ROM did not ship with those files. I don t believe any of the blobs run on the main CPU.
• NFC doesn t work, and I haven t gotten it to work. It seems the infrastructure for NFC is missing in Replicant 4.0, it doesn t even expose the NFC hardware permission capability. This is quite unfortunate for me, since I daily work with YubiKey NEO and would have preferred to replace Google Authenticator with the YubiOATH that uses the NEO for OATH secret storage.

Some other observations:

• Panorama mode in the Camera crashes; see issue about it.
• There is a number of smaller graphical issues. I believe these are related to the EGL but haven t understood the details. What I ve noticed are the following issues. The task switcher doesn t show mini screenshots of all running apps (the screenshots are just black). ZXing is not able to QR decode images (I m told this is because Replicant uses a RGB color plane instead of the required YUV color plane). Video playback in the gallery is laggy to the point of being unusable. Video playback on Youtube in the default web browser works in full screen (not laggy), but not embedded in the webpage.
• MTP has been a bit unreliable, my main laptop is able to import photos, but another laptop (also running Debian Wheezy) just stalls when talking to it. This may be a host issue, I have experience similar issues with a Nexus 7 2nd generation device.

I am quite happy with the setup so far, and I will continue to use it as my primary phone.

While quite old fashioned, SNMP is still an ubiquitous protocol supported by most network equipments. It comes handy to expose various metrics, like network interface counters, to be gathered for the purpose of monitoring. It can also be used to retrieve and modify equipments configuration. Variables exposed by SNMP agents (servers) are organized inside a Management Information Base (MIB) which is a hierarchical database¹. Each entry is identified by an OID. Querying a specific OID allows a manager (client) to get the value of an associated variable. For example, one common MIB-module is IF-MIB defined in RFC 2863. It contains objects used to manage network interfaces. One of them is ifTable whose rows are representing agent s logical network interfaces. Each row will expose the interface name, characteristics and various associated counters.

ifIndex	ifDescr	ifPhysAddress	ifOperStatus	ifOutOctets
1	lo		up	545721741
2	eth0	0:18:f3:3:4e:4	up	78875421
3	eth1	0:18:f3:3:4e:5	down	0

ifTable is indexed by its first column ifIndex. If you want to get the operational status of the second interface, you need to query IF-MIB::ifOperStatus.2 which is translated to OID .1.3.6.1.2.1.2.2.1.8.2 using information provided by the MIB definition.

Scripting SNMP An SNMP agent can deliver a lot of interesting information:

• interface counters with IF-MIB,
• IP addresses with IP-MIB,
• routing tables with IP-FORWARD-MIB,
• inventory with ENTITY-MIB,
• neighbors with LLDP-MIB.

You can gather those information manually with tools like snmpget and snmpwalk:

$ snmpwalk -v 2c -c public localhost IF-MIB::ifDescr      
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1

However, building robust scripts with them is quite challenging. For example, if you wanted to get the descriptions of all active interfaces as well as the total number of octets transmitted, you could do something like that:

#!/bin/sh
set -e
host="$ 1:-localhost "
community="$ 2:-public "
args="-v2c -c $community $host"
for idx in $(snmpwalk -Ov -OQ $args IF-MIB::ifIndex); do
    descr=$(snmpget -Ov -OQ $args IF-MIB::ifDescr.$idx)
    oper=$(snmpget -Ov -OQ $args IF-MIB::ifOperStatus.$idx)
    in=$(snmpget -Ov -OQ $args IF-MIB::ifInOctets.$idx)
    out=$(snmpget -Ov -OQ $args IF-MIB::ifOutOctets.$idx)
    [ x"$descr" != x"lo" ]   continue
    [ x"$oper" = x"up" ]   continue
    echo $descr $in $out
done

Hopefully, SNMP bindings in various languages are pretty common. For example, Net-SNMP ships with a Python binding:

import argparse
import netsnmp
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
args =  
    "Version": 2,
    "DestHost": options.host,
    "Community": options.community
 
for idx in netsnmp.snmpwalk(netsnmp.Varbind("IF-MIB::ifIndex"),
                            **args):
    descr, oper, cin, cout = netsnmp.snmpget(
        netsnmp.Varbind("IF-MIB::ifDescr", idx),
        netsnmp.Varbind("IF-MIB::ifOperStatus", idx),
        netsnmp.Varbind("IF-MIB::ifInOctets", idx),
        netsnmp.Varbind("IF-MIB::ifOutOctets", idx),
        **args)
    assert(descr is not None and
           cin is not None and
           cout is not None) #  
    if descr == "lo":
        continue
    if oper != "1": #  
        continue
    print("     ".format(descr, cin, cout))

This binding is quite primitive and has several drawbacks:

1. It exports everything as strings. See .
2. Error handling is just deficient. If you mispell something, like a variable name, you ll get snmp_build: unknown failure on the standard error. No exception. If a variable does not exist, you ll get None instead. See .

This inability to sanely handle failures makes this binding quite dangerous to use in scripts. Imagine making important modifications on the basis of the returned values. If you forget to check against None, your script may cause havoc!

Snimpy Because I didn t find any reliable Python binding for SNMP, I decided to write Snimpy with two goals in mind:

1. Leverage information contained in MIBs to provide a pythonic interface.
2. Any error condition should raise an exception.

Here is how the previous script could be written:

#!/usr/bin/env snimpy
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
m = M(options.host, options.community, 2)
load("IF-MIB")
for idx in m.ifDescr:
    if m.ifDescr[idx] == "lo":
        continue
    if m.ifOperStatus[idx] != "up":
        continue
    print("     ".format(m.ifDescr[idx],
                            m.ifInOctets[idx],
                            m.ifOutOctets[idx]))

You can also use a list comprehension:

load("IF-MIB")
print("\n".join([ "     ".format(m.ifDescr[idx],
                                    m.ifInOctets[idx],
                                    m.ifOutOctets[idx])
                  for idx in m.ifDescr
                  if m.ifDescr[idx] != "lo"
                  and m.ifOperStatus[idx] == "up" ]))

Here is another simple example to get the routing database from the agent:

load("IP-FORWARD-MIB")
m=M("localhost", "public", 2)
routes = m.ipCidrRouteNextHop
for x in routes:
    net, netmask, tos, src = x
    print(" :>15s / :<15s  via  :<15s  src  :<15s ".format(
        net, netmask, routes[x], src))

IP-FORWARD-MIB::ipCidrRouteNextHop is a more complex table with a compound index. Despite this, querying the table still seems natural. Have a look at Snimpy s documentation for more information. Behind the hood, SNMP requests are handled by PySNMP and MIB parsing is done with libsmi². Snimpy supports both Python 2, Python 3 and Pypy.

---

1. A MIB is defined using a subset of ASN.1 called SMI. However, it is not uncommon to refer to the definition as a MIB too.
2. Unfortunately, there is currently no robust SMI parser written in pure Python. For example, PySNMP relies on smidump which comes with libsmi. Snimpy uses a custom CFFI wrapper around libsmi.

A whopping 5 years ago, LWN ran a story about distributed bug trackers. This was during the early waves of distributed version control adoption, and so everybody was looking for other things that could benefit from decentralization. TL;DR: Not much has changed since. The potential benefits of a distributed bug tracker are similar to those of a distributed version control system: ability to fork any arbitrary project, easier collaboration between related projects and offline access to full project data. The article discussed a number of systems, including Bugs Everywhere, ScmBug, DisTract, DITrack, ticgit and ditz. The conclusion of our favorite grumpy editor at the time was that all of the available distributed bug trackers were still in their infancy. All of these piggyback on a version control system somehow - either by reusing the VCS database, by storing their data along with the source code in the tree, or by adding custom hooks that communicate with a central server. Only ScmBug had been somewhat widely deployed at the time, but its homepage gives me a blank page now. Of the trackers reviewed by LWN, Bugs Everywhere is the only one that is still around and somewhat active today. In the years since the article, a handful of new trackers have come along. Two new version control systems - Veracity and Fossil - come with the kitchen sink included and so feature a built-in bug tracker and wiki. There is an extension for Mercurial called Artemis that stores issues in an .issues directory that is colocated with the Mercurial repository. The other new tracker that I could find (though it has also not changed since 2009) is SD. It uses its own distributed database technology for storing bug data - called Prophet, and doesn't rely on a VCS. One of the nice features is that it supports importing bugs from foreign trackers. Some of these provide the benefits you would expect of a distributed bug tracker. Unfortunately, all those I've looked at fail to even provide the basic functionality I would want in a bug tracker. Moreso than with a version control system, regular users interact with a bug tracker. They report bugs, provide comments and feedback on fixes. All of the systems I tried make these actions a lot harder than with your average bugzilla or mantis instance - they provide a limited web UI or no web interface at all. Update: LWN later also publishe articles on on SD and on Fossil. Other interesting links are Eric Sink's article on distributed bug tracking (Erik works at Sourcegear who develop Veracity) and the dist-bugs mailing list.