Search Results: "ema"

30 November 2013

Gunnar Wolf: On errors in exams - Short rant

Blogging from a phone... first time ever. I don't want to forget some specifics for this :) I have just completed an exam to try to enter a postgraduate program (I'll talk more about it once it becomes real ). The exam is administered by CENEVAL, the same evaluation agency Where I presented my graduation equivalency exam some years ago - Only this exam is for all of the postgraduate studies on many national universities and is thus basically just a psychometric test. The exam had 162 questions, all to be filled in a optical reader sheet, on five subjects: mathematical reasoning, Spanish grammar and comprehension, Project management, Computers and technology, And English reading and understanding. It was all in all a fun exam to take, mostly due to the math reasoning part. But... on the Subject I Know I am an expert, I have to complain (and intend to find a easy to do so formally). First, I spotted two absolute mistakes (and answered based on What I knew others would, but knowing the answer is wrong technically). One was a subtlety, on how and why have hard drives should be defragmented (and part of my quip is that it's an obsolete habit, but besides, the answers were all erroneous), but a second one was... just wrong. It asked on what should not be part of an "Internet link" (can only guess they meant An URL). The 4 options were valid parts of a URL - including one very seldom used by most people, but very often by many of us: the @ sign. Anyway, answered it, but my other main gripe is that most of the section was in specific use of Office software. Not only In Office-like, which would be bad enough to begin with, but on specific ways of using Mainly Excel And PowerPoint. Syntax issues, or the name of the menu under which to look for specific functions.
Anyway, I will wait the stipulated 10 days for the exam to be rated, but will anyway look die a way to contact the very opaque and secretive CENEVAL. Not to demand to be better treated, but to try to correct those known mistakes and errors.

29 November 2013

Steve Kemp: A difficult day

Today was my last day working at Bytemark, and I found it a lot harder than expected. For better or worse I finished earlier than expected; having been gradually removing my accounts and privileges over the past few weeks I'd revoked my OpenVPN key this morning. Mid-afternoon my openvpn connection tried to renegotiate session keys, or similar, and failed. So I stopped work a few hours early. That meant I managed to avoid sending my "goodbye world" email, which is probably for the best - after all a lovely company, lovely people, and a good environment, what can you say besides things that are lovely? I think I largely wrapped things up neatly, and I'm pleased that one of my photos is hanging on the office wall. (I look forward to seeing that actually, I've only rarely made canvas prints.) The only other thing of note this week has been the sharp rise in blogspam I've detected. Black Friday alive and well, on the internets ..

Tanguy Ortolo: Mutt: encrypt all messages sent to known PGP users

GnuPG logo This is one thing I have wanted to do for a long time: configure Mutt to encrypt all messages sent to addresses for which I have a valid public key. Well, here is an awk-based script to generate that configuration. (Yes, I know, a similar script was already written. But I did not see it at first, and I find mine more readable. :-) )Code
#! /bin/sh
set -e
# Specific hooks set global configuration values for Mutt, which are not
# reset for new messages, so a default hook is needed to set the default
# value (do not try encrypting all messages).
printf "send-hook . unset crypt_autoencrypt\n\n"
cat <<EOF
# This group contains all known PGP users, and can be used to define
# hooks, for instance to encrypt all messages sent to PGP users, or to
# add a specific signature to promote PGP usage to non-PGP users.
EOF
# Generate a machine-readable list of known keys
gpg2 --batch --with-colons --list-keys \
      awk -- '
        BEGIN  
            FS = ":"
            # A flag that indicates whether or not we are in a public
            # key with usable encryption capability
            usable = 0
         
        # Detect beginning of public keys with usable encryption
        # capability
        $1 == "pub" && $12 ~ /E/   usable = 1  
        # Detect beginning of public keys without usable encryption
        # capability
        $1 == "pub" && $12 !~ /E/   usable = 0  
        
        # Only consider user IDs from public keys with usable encryption
        # capability and with regular, marginal, full or ultimate
        # validity (n, m, f or u in the second field)
        usable && $1 == "uid" && $2 ~ /[nmfu]/ && $10 ~ /@/  
            # Remove everything before and after the email address in
            # brackets
            sub(/^[^<]*</, "", $10)
            sub(/>[^>]*$/, "", $10)
            # Finally print the email address
            print $10
         ' \
      sort -u \
       
        # Group addresses by ten to reduce the number of Mutt commands
        finished=0
        while [ $finished -ne 1 ]
        do
            addresses=""
            for i in 0 1 2 3 4 5 6 7 8 9
            do
                if read address
                then
                    addresses="$addresses $address"
                else
                    finished=1
                    break
                fi
            done
            # Add PGP users' addresses to a pgp-users group
            printf "group -group pgp-users -addr %s\n" \
                "$addresses"
        done
     
# Build a hook to try encrypting messages sent to PGP users
printf "\n# Encrypt all messages sent to known PGP users\n"
printf "send-hook \"%%C pgp-users\" set crypt_autoencrypt\n" \
printf "\n# Add a specific signature to promote PGP usage to non-PGP users\n"
printf "#send-hook \"!%%C pgp-users\" set signature=~/.signature.use-pgp\n"
Usage Script usage This script generates uses GnuPG to print a list of all known keys, takes only valid ones, and uses them to build a series of configuration instructions for Mutt, that define hooks that are triggered when you write a message to them. This configuration has to be written to your Mutt configuration file:
$ bin/gen-pgp-hooks.sh >> ~/.muttrc
Or better (far better, in fact), to a dedicated configuration file you will refer to in your main one:
$ bin/gen-pgp-hooks.sh > ~/.muttrc.crypt-recipients
$ echo "source ~/.muttrc.crypt-recipients" >> ~/.muttrc
You may want to adapt this script to it directly writes to that file rather than having to redirect it. Also, remember to run it when you refresh your public keyring or add new keys to it. Mutt usage These hooks will make Mutt try to encrypt all messages you send to at least one known PGP user. That means it will try it even for messages you send to a PGP user and to non-PGP ones. In these cases, it will not find keys for all the recipient, and ask you which key to use; as you have none, you will have to quit that prompt by typing ^G, and manually disable encryption by accessing the PGP options menu with the key p.

28 November 2013

Dirk Eddelbuettel: RcppCNPy 0.2.1

Version 0.2.1 of RcppCNPy is now on CRAN earlier. This version is a minor upgrade with a few small changes as detailed in the NEWS file:
Changes in version 0.2.1 (2013-11-28)
  • Synchronized code with the cnpy repository
  • Added new function to test from R whether integers supported
  • Updated tests for integer support, if available
  • Updated vignette and discussion about need for rebuilding only RcppCNPy with the -std=c++11 flag if integer support is desired
  • Updated tests for integer support, if available
  • Updated THANKS file
  • Ensure that inclusion of Rinternal.h does not affect other headers by adding RF_NO_REMAP
CRANberries also provides a diffstat report for 0.2.1 relative to 0.2.0. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Debian Med: Bits from Debian Med team (by Andreas Tille)

Some of these bits are a bit outdated but there is no point in hiding this information just because it is old. FOSDEM talkAndreas Tille had a talk at FOSDEM which is also Video recorded. A quite similar but updated talk was held at DebConf 13 (see below). Jabber interview by reporter from IndiaThere was an interesting interview between Rajeev Nair and Andreas Tille at So, 17.02.2013 Rajeev Nair is Business head of an health care journal in India held via jabber. Since this might serve as some nice FAQ about Debian Med here is a complete log of this interview (permission to publish it was granted). Article about Debian MedAs a consequence of the interview above the Debian Med team has assembled an article which was originally intended to be published in the Health Cafe journal but somehow this never happened. So we ended up with a nice article targeting at interested readers about Free Software in medicine with not necessarily informatics background. Since the article is considered to be of good quality and has consumed some time of several team members we are seeking for ideas for relevant places where to publish it. It is available as PDFas well as in SVN. Yearly sprint of Debian Med teamThis year we had our third sprint of the Debian Med team and all participants consider it a great success again. The sprint was in end of February and the individual reports are linked from the according wiki page. Since Andreas Tille was able to prove the nice effect of having face to face meetings in Debian teams in the team metrics graphs presented in his talks at DebConf (see below) we will keep on with this good tradition. The next sprint is scheduled for Friday 31st January until Sunday 2nd February in Stonehaven near Aberdeen. If you are interested to meet the people behind Debian Med and want to join us in developing packages which are helpful in health care and bioinformatics you are invited to join us. DebConf 13 in VaumarcusAndreas Tille had two talks with specific influence to Debian Med development(Debian Med as a success story for other Blends, How to attract new developers for your team (MoM, SoB). For those who want to read a full (and lengthy) report about DebConf 13 by Andreas Tille this is online in this blog article. Codefest of the Open Bioinformatics FoundationIvo Maintz and Steffen M ller organised the Codefest of the Open Bioinformatics Foundation in Berlin. About 40 contributors from all over the world attended, connecting the Debian community strongly with upstream. The event triggered a joint paper of participants to Sprints, Codefests and Hackathons alike, presented at the Nettab conference on Semantic, Social, and Mobile Applications for Bioinformatics and Biomedical Laboratories in Venice. Reupload unattended packagesAndreas Tille has mentored the GSoC student Emmanouil Kiagias and as a byproduct some Blends metadata were now kept in UDD. This has inspired some new tools and one of this was an UDD query to return all packages in the interest of the Blend ordered by date when they were uploaded. It has shown that there were packages hanging around in the package pool since five years. So some effort was done to check those packages which were not necessarily "buggy" according to BTS entries and so we were able to fix the following issues in about 20-30 packages: Since it was impressive how many things that deserve fixing are hidden in such long unattended packages it is recommended also to other teams to spend some time into these. The UDD query can be easily tweaked for a team maintainer address. Debian Med Bug Squashing Advent Calendar 2013The Debian Med team will again do the Bug Squashing Advent Calendar. Feel free to join us in our bug squashing effort where we close bugs while other people are opening doors. :-) Kind regards and see you at Debian Med sprint
Andreas.

26 November 2013

Daniel Pocock: Real-time streaming market data with free software

Here is an overview of the real-time market data architecture based on OpenMAMA: All these components may be running on a single host or they may be distributed across different servers and workstations in a LAN. The OpenMAMA market data bus ties them together. External data sources These are external sources of data. These could be currency trading firms, stock brokers, bullion dealers or even Bitcoin exchanges. Some exchanges don't provide data services directly to the general public and they distribute their price data through third-party data vendors such as Reuters and Bloomberg. Feed handlers The feed handlers manage the connections to external data sources. These are typically daemon processes that implement the wire protocols used by the external vendors for transmitting data over the public internet or a leased line. They typically use a point-to-point topology such as TCP connections. The feed handlers publish the data into the market data bus. In practice, the feed handler may be a proprietary application provided by the data vendor or it may simply be a Python script that fetches exchange rates from a URL every five minutes. OpenMAMA The market data bus is a distributed framework that is accessible to all of the local servers and workstations in the LAN. At the lowest level a messaging middleware solution is used to transport the data. The Avis Event Router is a free middleware. There is upcoming support for Apache QPID as well. Some commcercial middleware is supported too - here is the full list OpenMAMA does not provide a daemon or server process of its own. It is a set of libraries that operate on top of the middleware transport. In very general terms, the OpenMAMA libraries let applications publish messages (such as price ticks) or subscribe to receive the messages published by other applications. Spreadsheet application (for example, LibreOffice) LibreOffice is introducing a new data import feature for streaming time series data. This appears to provide a useful integration point for OpenMAMA and discussion is taking place in the development community. The spreadsheet can operate in various ways. A simple application may simply poll the currency prices when the spreadsheet is opened or refreshed. A more demanding application may see the spreadsheet come alive, recalculating all cells on every tick from the data vendor (several times per second). This latter scenario is more common in dealing rooms and hedge funds. Accounting/billing software There are many free accounting applications today, including PostBooks and GnuCash and some more heavyweight solutions like Adempiere and OpenERP. In a world where credit cards and the world wide web have made international trade an everyday activity, many people are using this software to track expenses and accounts in more than one currency. For example, a British business may be paying some suppliers in Euros and a Canadian may be charging some customers in US dollars. A Swiss person may be keeping some of their savings in gold bullion in the vault of one of their world reknowned Swiss banks. A common requirement for all of these users is the balance sheet. Using real-time market data sources, the balance sheet can be refreshed at any time from the live market prices. A home user may just find it convenient that they can open the balance sheet and always see their net worth immediately without having to manually cut and paste the currency prices from a web page. A business user operating in a competitive industry with low margins may be checking the balance sheet several times per day to ensure they remain solvent and profitable. Web shops Many web shops now offer the customers the convenience of viewing prices in their own currency. It is important to make sure these prices are accurate, especially when dealing with volatile currencies or when the products have a low profit margin. The web application server can receive the market data price ticks just like any other application in the network. Using this data, it can insert live prices into the pages as they are generated. If the currencies are particularly volatile, the web application could stream the prices to the browser using WebSockets and a JavaScript client embedded in the page would update the price field in real time. In fact, this web-based streaming price update mechanism is exactly how many online financial trading services offer live market prices to their customers. Real-time valuation server In a large organisation there may be many users looking at the same values. For example, in a trading desk, many users may need to see the desk's overall position in each market. In a busy web shop, many concurrent users may need to see the prices of common products. Rather than recalculating these values for each user in parallel, a common solution involves setting up a server to receive raw values (such as currency prices) from the data feed, calculate values needed by local users and then broadcast those values over the market data bus. How applications can integrate with real-time market data Here are some steps for getting started:

Daniel Pocock: Real-time streaming market data with free software

Here is an overview of the real-time market data architecture based on OpenMAMA: All these components may be running on a single host or they may be distributed across different servers and workstations in a LAN. The OpenMAMA market data bus ties them together. External data sources These are external sources of data. These could be currency trading firms, stock brokers, bullion dealers or even Bitcoin exchanges. Some exchanges don't provide data services directly to the general public and they distribute their price data through third-party data vendors such as Reuters and Bloomberg. Feed handlers The feed handlers manage the connections to external data sources. These are typically daemon processes that implement the wire protocols used by the external vendors for transmitting data over the public internet or a leased line. They typically use a point-to-point topology such as TCP connections. The feed handlers publish the data into the market data bus. In practice, the feed handler may be a proprietary application provided by the data vendor or it may simply be a Python script that fetches exchange rates from a URL every five minutes. OpenMAMA The market data bus is a distributed framework that is accessible to all of the local servers and workstations in the LAN. At the lowest level a messaging middleware solution is used to transport the data. The Avis Event Router is a free middleware. There is upcoming support for Apache QPID as well. Some commcercial middleware is supported too - here is the full list OpenMAMA does not provide a daemon or server process of its own. It is a set of libraries that operate on top of the middleware transport. In very general terms, the OpenMAMA libraries let applications publish messages (such as price ticks) or subscribe to receive the messages published by other applications. Spreadsheet application (for example, LibreOffice) LibreOffice is introducing a new data import feature for streaming time series data. This appears to provide a useful integration point for OpenMAMA and discussion is taking place in the development community. The spreadsheet can operate in various ways. A simple application may simply poll the currency prices when the spreadsheet is opened or refreshed. A more demanding application may see the spreadsheet come alive, recalculating all cells on every tick from the data vendor (several times per second). This latter scenario is more common in dealing rooms and hedge funds. Accounting/billing software There are many free accounting applications today, including PostBooks and GnuCash and some more heavyweight solutions like Adempiere and OpenERP. In a world where credit cards and the world wide web have made international trade an everyday activity, many people are using this software to track expenses and accounts in more than one currency. For example, a British business may be paying some suppliers in Euros and a Canadian may be charging some customers in US dollars. A Swiss person may be keeping some of their savings in gold bullion in the vault of one of their world reknowned Swiss banks. A common requirement for all of these users is the balance sheet. Using real-time market data sources, the balance sheet can be refreshed at any time from the live market prices. A home user may just find it convenient that they can open the balance sheet and always see their net worth immediately without having to manually cut and paste the currency prices from a web page. A business user operating in a competitive industry with low margins may be checking the balance sheet several times per day to ensure they remain solvent and profitable. Web shops Many web shops now offer the customers the convenience of viewing prices in their own currency. It is important to make sure these prices are accurate, especially when dealing with volatile currencies or when the products have a low profit margin. The web application server can receive the market data price ticks just like any other application in the network. Using this data, it can insert live prices into the pages as they are generated. If the currencies are particularly volatile, the web application could stream the prices to the browser using WebSockets and a JavaScript client embedded in the page would update the price field in real time. In fact, this web-based streaming price update mechanism is exactly how many online financial trading services offer live market prices to their customers. Real-time valuation server In a large organisation there may be many users looking at the same values. For example, in a trading desk, many users may need to see the desk's overall position in each market. In a busy web shop, many concurrent users may need to see the prices of common products. Rather than recalculating these values for each user in parallel, a common solution involves setting up a server to receive raw values (such as currency prices) from the data feed, calculate values needed by local users and then broadcast those values over the market data bus. How applications can integrate with real-time market data Here are some steps for getting started:

Axel Beckert: Next Debian Meetup in Zurich

The first Debian meetup in Zurich last month was quite a success and I look forward to further Debian meetups in Zurich every first Tuesday of the month. The next meetup will be

on Tuesday, 2013-Dec-03 starting at 18:30 CET
at St. Gallerhof, Konradstrasse 2, 8005 Zurich
Please note that this is a different location than last month. Everybody who is interested in Debian is welcome to join us. Registering is not necessary. There was also some interest in Debian meetups in other Swiss cities, namely Bern and somewhere around Lac Leman. In case you want a Debian meetup elsewhere in Switzerland, too, or if you re interested in any Debian meetup in Switzerland, feel free to join the Swiss Debian Community Mailing List and help organising other Swiss Debian meetups.

Daniel Pocock: The free software country?

Loch Ness Monster, Scotland There is big news in Europe right now, especially the UK, about the Scottish nation's manifesto for independence. I thought this was really amazing, a country that is going to move to free software and protect its citizens from horrors like DRM. I couldn't wait to find out more, for example, will they use Linux or *BSD? After all, how many nations today can really consider themselves independent when they are trapped in the use of complex and opaque systems that require ongoing royalty payments to a foreign corporation, much like ancient villages paying their tithes to Rome? Today it is actually worse: these independent nations are not just sacrificing their cash - the new Rome is also using those same secret, proprietary systems to raid their subjects' privacy. So I went to the manifesto's web site to get the details. I couldn't even find the words "software" or "technology" listed in the policy index on the front page - and software doesn't even appear in the search results. A section on Culture, Communications and Digital only really looks at the issues for the media (BBC after independence). They don't even make any comment about the British Telecom (BT) pension crisis and whether Scotland will abandon BT's services, leaving only those customers south of the border to continue subsidising the deficit through higher phone bills. There is some hope, with the dot scot (.scot) top level domain coming in 2015 regardless of the referendum outcome. Does this mean an independent Scotland is going to be run without technology at all then? That could be an interesting way to avoid the type of infiltration of communications technology that has been taking place in Brussels. Or do the people calling the shots simply fail to realise the impact of software and communications technology on a modern day concept of independence? Party date set For those who just care about gatecrashing the post-independence party, start booking your leave for March 2016.

24 November 2013

Gregor Herrmann: RC bugs 2013/47

no exciting achievements but I still try to keep an eye on RC bugs:

22 November 2013

Vincent Sanders: Error analysis is the sweet spot for improvement

Although Don Norman was discussing designers attitude to user errors I assert the same is true for programmers when we use static program analysis tools.

The errors, or rather defects in the jargon, that a static analysis tools produce can be considered low cost well formed bug reports available very early in the development process.

When I say low cost it is because they can be found by a machine without a user or fellow developer wasting their time finding them. Well formed comes because the machine can describe exactly how it came to the logical deduction leading to the defect.
IntroductionStatic analysis is in general terms using the computer to examine a program for logic errors beyond those of pure syntax before it is executed. Examining a running program for defects is known as dynamic program analysis and while a powerful tool in its own right is not the topic of discussion.

This analysis has historically been confined to compiled languages as their compilers already had the Abstract Syntax Tree (AST) of the code available for analysis. As an example the C language (released in 1972) had the lint tool (released in 1979) based on the PCC compiler.

Practical early compilers (I am generalising here as the 19070s were a time of white hot innovation in computing and examples of just about any innovation in the field could probably be found) were pretty primitive and produced executables which were less good than hand written assembler output. Due to practical constraints the progress of optimising compilers was not as rapid as might be desired so static analysis was largely used as an external process.

Before progressing I ought to explain why I just mixed the concept of an optimising compiler and static analysis. The act of optimisation within those compilers requires program analysis, from which they can generate defect reports which we all know and love as compiler warnings, also explaining why many warnings only appear at higher optimisation levels where deeper analysis is required.

The attentive reader may now enquire as to why we would need external analysis tools when our compilers already perform the task. The answer stems from the issue that a compiler is trying to reconcile many desirable traits including:
The slow progress in creating optimising compilers initially centred around the problem of getting the compiled output in a reasonable time to allow for a practical edit-compile-run-debug cycle although the issues more recently have moved more towards the compiler implementation costs.

Because the output generation time is still a significant factor compilers limit the level of static analysis performed to that strictly required to produce good output. In standard operation optimising compilers do not do the extended analysis necessary to find all the defects that might be detectable.

An example: compiling one 200,000 line C program with the clang (v3.3) compiler producing x86 instruction binaries at optimisation level 2 takes 70 seconds but using the clang based scan-build static analysis tool took 517 seconds or more than seven times as long.
Using static analysis
As already described warnings are a by-product of an optimising compilers analysis and most good programmers will endeavour to remove all warnings from a project. Thus almost all programmers are already using static analysis to some degree.

The external analysis tools available can produce many more defect reports than the compiler alone as long as the developer is prepared to wait for the output. Because of this delay static analysis is often done outside the usual developers cycle and often integrated into a projects Continuous Integration (CI) system.

The resulting defects are usually presented as annotated source code with a numbered list of logical steps which shows how the defect can present. For example the steps might highlight where a line of code allocates memory from the heap and then an exit path where no reference to the allocated memory is kept resulting in a resource leak.

Once the analysis has been performed and a list of defects generated the main problem with this technology rears its ugly head, that of so called "false positives". The analysis is fundamentally an undecidable problem (it is a variation of the halting problem) and relies on algorithms to generate approximate solutions. Because of this some of the identified defects are erroneous.

The level of erroneous defect reports varies depending on the codebase being analysed and how good the analysis tool being used is. It is not uncommon to see false positive rates, even with the best tools, in excess of 10%

Good tools allow for this and provide ways to supply additional context through model files or hints in the source code to suppress the incorrect defect reports. This is analogous to using asserts to explicitly constrain variable values or a type cast to suppress a type warning.

Even once the false positives have been dealt with there comes the problem of defects which while they may be theoretically possible take so many steps to achieve that their probability is remote at best. These defects are often better categorized as a missing constraint and the better analysis tools generate fewer than the more naive implementations.

An issue with some defect reports is that often defects will appear in a small number of modules within programs, generally where the developers already know the code is of poor quality, thus not adding useful knowledge about a project.

As with all code quality tools static analysis can be helpful but is not a panacea code may be completely defect free but still fail to function correctly.
Defect DensityA term that is often used as a metric for code quality is the defect density. This is nothing more than the ratio of defect to thousands of lines of code e.g. a defect density of 0.9 means that there is approximately one defect found in every 1100 lines of code.

The often quoted industry average defect density value is 1, as with all software metrics this can be a useful indicator but should not be used without understanding.

The value will be affected by improvements in the tool as well as how lines of code are counted so is exceptionally susceptible to gaming and long term trends must be treated with scepticism.
Practical examplesI have integrated two distinct static analysis tools into the development workflow for the NetSurf project which I shall present as case studies. These examples show a good open source solution and a commercial offering highlighting the issues with each.

Several other solutions, both open source and commercial, exist many of which have been examined and discarded as either impractical or proving less useful than those selected. However the investigation was not comprehensive and only considered what was practical for the project at the time.
clangThe clang project is a frontend to the LLVM project providing an optimising compiler for the C, C++ and objective C languages. As part of this project the compiler has been enhanced to run a collection of "checkers" which implement various methods of analysis on the code being compiled.

The "scan-build" tool is provided to make the using these features straightforward. This tool generates defect reports as a series of html files which show the analysis results.


NetSurf CI system scan-build overview
Because the scan-build takes in excess of eight minutes on powerful hardware the NetSurf developers are not going to run this tool themselves as a matter of course. To get the useful output without the downsides it was decided to integrate the scan into the CI system code quality checks.

NetSurf CI system scan-build result list
Whenever a git commit happens to the mainline branch and the standard check build completes successfully on all target architectures the scan is performed and the results are published as a list of defects.

The list is accessible directly through the CI interface and also incorporates a trend graph showing how many defects were detected in each build.

A scan-build report showing an extremely unlikely path to a defect
Each defect listed has a detail link which reveals the full analysis and logic necessary to cause the defect to occur.

Unfortunately even NetSurf which is a relatively small piece of software (around 200,000 lines of code at time of writing) causes 107 defects to be emitted by scan-build.

All but 25 of the defects are however "Dead Store" where the code has a value assigned but is never checked. These errors are simply not interesting to the developers and are occurring in code generated by a tool.

Of the remaining defects identified the majority are false positives and several (like the example in the image above) are simply improbable requiring a large number of steps to reach.

This shows up the main problem with the scan-build tool in that there is no way to suppress certain checks, mark defects as erroneous or avoid false positives using a model file. This reduces the usefulness of these builds because the developers all need to remember that this list of defects is not relevant.

Most of the NetSurf developers know that the project currently has 107 outstanding issues and if a code change or tool improvement were to change that value we have to manually work through the defect list one by one to check what had changed.
CoverityThe coverity SAVE tool is a commercial offering from a company founded in the Computer Systems Laboratory at Stanford University in Palo Alto, California. The results of the original novel research has produced a good solution which improved on analysis tools previously available.

Coverity Interface showing summary of NetSurf analysis. Layout issues are a NetSurf bug
The company hosts a gratis service for open source projects, they even provide scans for the Linux kernel so project size does not appear to be an issue.

The challenges faced integrating the coverity tool into the build process differed from clang however the issue of execution time remained and the CI service was used.

The coverity scanning tool is a binary executable which collects data on the build which is then submitted to the coverity service to be analysed. This tool obviously relies upon the developer running the executable to trust coverity to some degree.

A basic examination of the binary was performed and determined the executable was not establishing network connections or performing and observably undesirable behaviour. From this investigation the decision was made that running the tool inside a sandbox environment on a CI build slave was safe. The CI system also submits the collected results in a compressed form directly to the coverity scan service.

Care must be taken to only submit builds according to the services Acceptable Use Policy which limits the submission frequency of NetSurf scans to every other day. To ensure the project stays within the rules the build performed by the CI system is manually controlled and confined to a subset of NetSurf developers.

Coverity connect defect management console for NetSurfThe results are presented using the coverity connect web technology based defect management tool. Access to the coverity connect interface is controlled by a user management system which precludes publicly publishing the results within the CI system.

Unfortunately NetSurf itself does not currently have good enough JavaScript DOM bindings to support this interface so another browser must be used to view it.

Despite the drawbacks the quality of the analysis results is greatly superior to the clang solution. The false positive rate is very low while finding many real issues which had not been previously detected.

The analysis can be enhanced by use of collection configuration and modelling files which remove intended constructions from consideration reducing the false positive rate to very low levels. The ability to easily and persistently suppress false positives through the web interface is also available.

The false positive management capabilities coupled with a user interface that makes understanding the defect path simple make this solution very practical and indeed the NetSurf developers have removed over 50 actual issues within a relatively short period since the introduction of the tool.

Not all of those defects could be considered serious but they had the effect of encouraging deeper inspection of some very dubious smelling source.
ConclusionsThe principle conclusions of implementing and using static analysis have been:

When I started looking at this technology I was somewhat dubious about its usefulness but I have definitely changed my mind. It is a useful addition to any non-trivial project and the return on time and effort should be repaid handsomely in all but already perfect code (if you believe you have such code I have a bridge to sell you).

21 November 2013

Craig Small: Quieting dbus

I run a program called logcheck which regularly scans the logfiles looking for interesting things . One of these interesting things (to logcheck, not to me at first) was dbus complaining about mythtv. The logs look something like this:
Nov 17 05:08:18 elmo dbus-daemon[1621]: dbus[1621]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.236" (uid=999 pid=3669 comm="/usr/bin/mythfilldatabase --verbose general --logl") interface="org.freedesktop.NetworkManager" member="GetDevices" error name="(unset)" requested_reply="0" destination="org.freedesktop.NetworkManager" (uid=0 pid=1930 comm="/usr/sbin/NetworkManager ")
What this is is the dbus daemon rejecting a message being sent from mythfilldatabase that wants to go to NetworkManager. Looking at NetworkManager s dbus configuration, its doing all the right things. So it s a simple matter of just fixing up the dbus configuration and you re done. Then I hit the first snag; the dbus documentation is a little sad. I remember seeing a show where they MRI scanned some artifacts to get to the writing and then used a maker-bot to recreate the hidden writing. They then painstakingly translated it using a team of experts to work out the English translation. Working out the dbus configurations is not quite that difficult, but it s close. The easiest way is to copy something and see if it works, which is what I did. I created a file /etc/dbus-1/system.d/mythtv.conf which permitted the myth user access to the GetDevices method. It looks like:
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
        <policy user="mythtv">
                <allow send_destination="org.freedesktop.NetworkManager"
                       send_interface="org.freedesktop.NetworkManager"
                       send_member="GetDevices"/>
        </policy>
</busconfig>
Reloaded dbus and all went quiet.

19 November 2013

Francois Marier: Things that work well with Tor

Tor is a proxy server which allows its users to hide their IP address from the websites they connect to. In order to provide this level of anonymity however, it introduces latency into these connections, an unfortunate performance-privacy trade-off which means that few users choose to do all of their browsing through Tor. Here are a few things that I have found work quite well through Tor. If there are any other interesting use cases I've missed, please leave a comment!

Tor setup There are already great docs on how to install and configure the Tor server and the only thing I would add is that I've found that having a Polipo proxy around is quite useful for those applications that support HTTP proxies but not SOCKS proxies. On Debian, it's just a matter of installing the polipo package and then configuring it as it used to be recommended by the Tor project.

RSS feeds The whole idea behind RSS feeds is that articles are downloaded in batch ahead of time. In other words, latency doesn't matter. I use akregator to read blogs and the way to make it fetch articles over Tor is to change the KDE-wide proxy server using systemsettings and setting a manual proxy of localhost on port 8008 (i.e. the local instance of Polipo). Similarly, I use podget to automatically fetch podcasts through this cron job in /etc/cron.d/podget-francois:
0 12 * * 1-5 francois   http_proxy=http://localhost:8008/ https_proxy=http://localhost:8008/ nocache nice ionice -n7 /usr/bin/podget -s
Prior to that, I was using hpodder and had the following in ~/.hpodder/curlrc:
proxy=socks4a://localhost:9050

GnuPG For those of us using the GNU Privacy Guard to exchange encrypted emails, keeping our public keyring up to date is important since it's the only way to ensure that revoked keys are taken into account. The script I use for this runs once a day and has the unfortunate side effect of revealing the contents of my address book to the keyserver I use. Therefore, I figured that I should at least hide my IP address by putting the following in ~/.gnupg/gpg.conf:
keyserver-options http-proxy=http://127.0.0.1:8008
However, that tends to makes key submission fail and so I created a key submission alias in my ~/.bashrc which avoids sending keys through Tor:
alias gpgsendkeys='gpg --send-keys --keyserver-options http-proxy=""'

Instant messaging Communication via XMPP is another use case that's not affected much by a bit of extra latency. To get Pidgin to talk to an XMPP server over Tor, simply open "Tools Preferences" and set a SOCKS5 (not Tor/Privacy) proxy of localhost on port 9050.

GMail Finally, I found that since I am running GMail in a separate browser profile, I can take advantage of GMail's excellent caching and preloading and run the whole thing over Tor by setting that entire browser profile to run its traffic through the Tor SOCKS proxy on port 9050.

18 November 2013

Daniel Pocock: $75,000 to explore free software during maternity leave?

A number of countries offer women various social benefits during maternity leave. A new scheme in Australia promises to give virtually all women one hundred percent of their salary (capped at a whopping $12,500 per month) and extend maternity leave to 6 months. Some believe it has been funded by abolishing fibre-to-the-home, but that is another topic... This is a dramatic upgrade - in both cash and time - from the existing scheme where women have 18 weeks leave at the minimum wage. In Australia, the "minimum wage" figure is a fantasy, well below the real cost of living. In practice, the existing scheme and economic pressure force many women back into their job in less than 18 weeks. The new scheme with a one hundred percent salary replacement eliminates much of that pressure. Many European countries offer women some payment that pays the real cost of living, for example, EUR 2,400 per month in France. That is more than the minimum wage in Australia, enough to cover food, clothing and rent in most French cities. Australia's new system, in comparison, will give women on high salaries enough cash to immediately hire a nanny and other discretionary expenses that will maintain their lifestyle as much as they can within reason while dealing with a new baby. The cap is 300% higher than in France and it is 100% wage replacement, whereas many European schemes only give women a percentage of their salary. For this reason, many people have described it as the "Rolls Royce" of maternity pay. Just who are the women in these higher salary brackets? It is hardly any surprise that many of them are in IT and engineering professions. The upper salary range of the new maternity leave scheme appears to squarely match with the cashflows of female software developers, web designers, project managers and potentially many other professionals with a background in science or engineering. Many women worry about being out of the workforce for too long and the potential impact on their skills and professional networks. In computing jobs, technology can change a lot in 6 months. There are many men and women who enjoy computing in their free time as much as in their job. For female developers with this time out of the office, the world's most comfortable maternity leave scheme could be an opportunity to spend a little time exploring the free software world. Just to be clear, I'm not suggesting that women should be reaching for the laptop when they first go into labor and getting plugged in to the web before the umbilical cord has been severed. Some people find technology to be a rewarding pursuit throughout their whole life, while others simply see it as a job. For those who find free software interesting, some may simply pursue something creative that has no relation to the IT projects in their job, some may bring some new experiences back to their job. Others may choose to reconsider their career, performing free software development work on a freelance basis rather than going back to an office environment. The "Rolls Royce" maternity pay scheme could provide a comfortable buffer for them as they evaluate their next steps and make such decisions.

Russ Allbery: Review: Anathem

Review: Anathem, by Neal Stephenson
Publisher: William Morrow
Copyright: 2008
ISBN: 0-06-147409-6
Format: Hardcover
Pages: 935
Fraa Erasmas is a little less than twenty years old and is an avout of the decade math of the Concent of Saunt Edhar. As Anathem opens, he's approaching his first apert. All of those words are left for the reader to puzzle out for themselves, unless you cheat and read the glossary (which I would recommend doing only with caution, but more on that in a moment), since Erasmas, the first-person narrator of this story, writes (mostly) as if the reader is familiar with the mathic world and its concepts. Translated into English, it means that Erasmas was brought into something akin to a scientific monastery (the "math") at around the age of nine and has been there for ten years, since his math is a decade math and therefore remains closed to all outside (s cular) ideas and contact for periods of ten years at a time. He is still a student there, not yet declaring allegiance to one of the chapters. That's something that he will be expected to do after the first apert. Every ten years, the decade math opens its doors to the outside world for a period of ten days (apert), and the fraas and suurs can leave the math and travel through the outside world as they wish. The start of Anathem is an exercise in alienation and reorientation. Stephenson has constructed a complex society with a long history and its own specific technical jargon and throws the reader into it with a minimum of orientation. Despite being familiar with SF novels that use this technique, I did read the preface and recommend that other readers do as well unless they particularly love piecing together a world from hints and clues. The timeline, in particular, was invaluable and something I referred to throughout the story. But even with the preface, you should expect to read the first part of Anathem interpolating meanings from context or just letting things slip past until you later come to understand them. A lot of people bounce off of Anathem here, and it's one of the reasons why I'd been postponing reading it (the other being that it's a substantial doorstop and I wanted to give it my undivided attention). But it's really not as bad as I feared. One gets a feel for the terms fairly quickly, and the more difficult ones Stephenson defines as he goes with excerpts from a dictionary of the mathic world. I found it slightly disorienting for a chapter or two, and then it started growing on me. The terminology is sometimes different just for the sake of being different (jeejahs are indistinguishable from smartphones for all practical purposes), but usually the invented words provide either important links to the past of the constructed world of Arbre or are technical terms that add precision and clarity once you know their meanings. It can be a little bit frustrating to remember the mapping of famous theories and theorems to the Orth names, but only mildly. (And Diax's Rake is so useful of a name for something that English doesn't name that I may start using it myself.) This is also why I recommend against reading the glossary. The definitions of terms are linked to important parts of history, which are, in turn, linked to important parts of the plot, and the glossary therefore risks spoilers. I liked having the term introduced by the frequent dictionary excerpts scattered through the chapters, and enjoyed the feeling of the terminology unfolding with the plot. The glossary is there if you really get lost, but I think the experienced SF reader (particularly if you have enough history background to pick up on the obvious parallels and start mapping bits to Greece, Rome, the Catholic Church, etc.) will be able to navigate the language without that much difficulty. I also stopped caring about the terminology because the story is so engrossing that it pulled me right into the book, closed over me, and made Erasmas's world feel real, precious, and fascinating. Whether that's going to be true of other people is a surprisingly difficult question, and one that I'll try to tackle as part of this review. But I will say up-front that I think this is the best book that Stephenson has ever written, even better than Snow Crash, despite some undeniable flab and one extremely irritating choice. Anathem is, at its heart, a novel about the scientific method. I think it's the first book by Stephenson that elegantly mingles his tendencies (and obsessions) as a writer with the interests (and obsessions) of the characters and makes the whole novel feel coherent. All of Stephenson's books are prone to discursive infodumps, but they've usually had to be shoehorned in around the characters. Even with the Baroque Cycle, a series full of natural philosophers, the digressions are jammed in around the plot like the filling of an over-packed box and frequently stick out at odd angles or dribble out on the floor. With Anathem, Stephenson has populated a book full of characters who have completely believable and engrossing reasons to digress into scientific debate and analysis, can put them into dialogue and thereby avoid some of the strain of the infodump, and (most importantly) largely restrains himself from narrative digressions via first-person narration and careful attention to when the characters themselves would follow the same digression. In short, this is a book full of very smart and very well-educated people who figure things out from first principles using agreed-upon theory, thrown into an unknown and dangerous situation that requires a great deal of deep thinking, science, engineering, and philosophy. All of Stephenson's quirks are still here, on prominent display, but rarely have I ever seen a better match of writer, characters, plot, and world background than this book. I would go so far as to say that, if you don't like this book, you're unlikely to like any long Stephenson. I haven't said a lot about the plot. Erasmas's apert is only the first part, and barely touches on any significant plot elements. It's mostly there to set some groundwork, introduce characters, and get the reader oriented. I'm not going to say much more, since the way the novel transitions from routine to unusual to emergency, and the resulting scramble to understand the emergency, is the heart and soul of this book and should not be spoiled. (I spoiled myself slightly by reading some analysis that I shouldn't have and regretted it, although it wasn't too much of a problem.) This is one of Stephenson's best-plotted novels. It develops slowly enough for the reader (and the characters) to think hard about it and make some good and bad guesses, but fast enough to stay engrossing. Most of it is very well-paced provided that you find the background of science and philosophy of science interesting. (If you don't, this is probably not a book you're going to enjoy.) I say "most of it" because, as mentioned earlier, this book is a bit flabby. There are a couple of sections where the theory and analysis are pushed aside by exigent circumstances and more action-oriented plot elements, both of which reminded me that Stephenson is very bad at writing action sequences and both of which I wished were shorter. And there are a couple of sections where Erasmas is getting oriented to a new situation or working up the courage to do something where I wished he'd get on with it already. But for a novel of nearly 1,000 pages, the amount of tension and reader interest Stephenson maintained for me was quite impressive. But as good as the plot is, the best part of this book is its full-bore, no-apologies embrace of understanding things. This is a book about making sense of the world, and about what it means to make sense of the world, and how you go about it, and what preconditions are required to do so. It is not, contrary to so much science fiction, a book about knowing things. The people who are already experts appear in this book, but generally only in the context of figuring out new things they've not previously been exposed to. Rather, it's a book about learning things, the thrill that comes from applying theory to understand something new, and the satisfaction of working through something from first principles or previously-established theory, arriving at testable predictions, and then testing them. It's the first novel I've read that captures some of the joy and delight that I got from reading George Gamow's One Two Three... Infinity as a child. Stephenson novels have always been about showing the reader neat things he learned, but I think previous books relied on the reader to bring their own enthusiasm. Anathem, at least for me, goes a step farther and uses the passion and approach of its characters to create that enthusiasm. It's one of those books that I not only enjoyed but went away from feeling like it made me a better person in some subtle but detectable way. Sadly, this is also where the irritating part comes in. I'm going to be very careful and indirect here, since the details are all extremely significant spoilers. You may want to skip to the last paragraph of this review if you want to avoid any knowledge about the end of Anathem. Stephenson pulls several rabbits out of the hat in this book, but most of them are well-defended and nicely handled bits of misdirection. There are a few where I thought the science or engineering was dubious, but even those are usually backed with enough hand-waving about Arbre technology that I could swallow it enough for the purposes of this book. (The timeline here is very helpful in allowing one to write off a few things as unknown technology.) But he puts, at the core of the plot, one of the most scientifically dubious bits of the whole book, and there's simply no avoiding that by the end of the story. Now, this doesn't mean that he just expects the reader to swallow it. Much of Anathem is a carefully-constructed defense of the bits he uses for the plot, and the defense is not half-bad within his fictional world. There are substantial in-book justifications for believing these techniques will work on Arbre even if they wouldn't in our scientific domain. So it's not so much the plausibility that bothered me; the idea is at least as plausible as some of the FTL drive concepts that I've swallowed without complaint. I do hold Anathem to a higher standard because the book is about science in a way that most science fiction isn't, but even with that, I think Stephenson mostly managed to dance across the thin ice he built. The problem is more subtle but more serious: if taken seriously on its own terms, the approach Stephenson takes to the plot destroys science in his universe. I can't really say more than that, since all of the details are huge spoilers, but the more I thought about it, the more irritated and annoyed I got. Anathem is otherwise a brilliant defense of the scientific method, and it felt like Stephenson injected a poison pill in the center of it, leading necessarily to a world where the scientific method no longer works. This does not happen on camera, or even between the pages of the book; you have to think about the implications for a while before you realize that's what happened. But once you do, it's a bitter pill to swallow, and it's hard to escape the feeling that it fundamentally undermines Stephenson's whole project here. And, to make it worse (although also oddly better in a way), I think it was unnecessary. He would have had to reconstruct the ending, but I can't shake the feeling that there were ways to write the ending, even maintaining the same character strengths, without having to pull that particular rabbit out of the hat. Because I think it's, in the end, unnecessary, I was still able to enjoy the whole book, including the ending, despite this, but while the rest of Anathem is a wholehearted 10, that choice knocked a full point off my impression of the book. It's profoundly irritating precisely because Stephenson did such an excellent job with the rest of it. But, that said, this book is still exceptional. I think it's the book that Stephenson was meant to write. Not only did he get the overall construction nearly perfect, he gets so many of the details right, from the history (which for a while I thought was merely clever, once I started seeing the correspondences that moved it out of the realm of pure invention, but which transitioned back to profound with later revelations in the book) to the language to the way that he excerpted some theoretic digressions into very well-done appendices. If you've ever thought of a theory as beautiful, or like reading about smart people debating the nature of understanding and knowledge, I cannot recommend this book too highly. Despite the highly irritating flaw at its center, and despite a few sections that show Stephenson is not a well-rounded writer, I think it's brilliant. Rating: 9 out of 10

17 November 2013

Steve Kemp: All change

If this post is visible I should have migrated the following virtual machines to a new home: These previously existed on a machine at Bytemark, running under screen and KVM. Now they exist upon a different Bytemark-rented host. TODO: Move 4096.io, configure an auto-builder guest (I have a slaughter policy for that), and allocate a /48 so that I regain IPv6 support (/56 would do, I guess. I want a /64 for each guest.).

16 November 2013

Lars Wirzenius: Debian: developing it wrong

This is an essay form of a talk I have given today at the Cambridge Mini-debconf. The talk was videoed, so will presumably show up in the Debconf video archive eventually. Abstract Debian has a long and illustrious history. However, some of the things we do perhaps no longer make as much sense as they used to do. It's a new millennium, and we might find better ways of doing things. Things that used to be difficult might now be easy, if we dare look at things from a fresh perspective. I have been doing that at work, for the Baserock project, and this talk is a compilation of observations based on what I've learnt, concentrating on things that affect the development workflow of package maintainers. Introduction and background I have been a Debian developer since August, 1996 (modulo a couple of retirements), and have used it a little bit longer. I have done a variety of things for Debian, from maintaining PGP 2 packages to writing piuparts to blathering excessively on the mailing lists. My day job is to develop the Baserock system at Codethink. Baserock is a set of tools and workflows for developing embedded and appliance Linux systems. If you squint, it looks a bit like a source-based Linux distribution. I have worked on Baserock since September, 2011. Some of Baserock's design has been influenced by my experience with Debian. With Baserock I have the chance to fix all the things that are wrong in Debian, and this talk is me giving back to Debian by pointing out some the things I feel should be fixed. I don't have solutions for these problems: this is a bug report, not a patch. It's also perhaps a bit of a rant. I am specifically concentrating here on technical and tooling issues that affect the development process of Debian. I excluding social issues. I am also not trying to get Debian to switch to Baserock. Baserock is targeting embedded and appliance systems, and makes simplifying assumptions based on those targets, which Debian does not get to do. I am pointing out problems, and I am outlining solutions as implemented in Baserock, when I think the concept carries over well to Debian. Build tools should be intelligent, packaging should be dumb In Debian, the fundamental tool for compiling the upstream code and assembling a binary package is dpkg-buildpackage. It uses debian/rules, which is a Makefile with targets with specific names and semantics. By executing the right targets in the right order, dpkg-buildpackage tells the source package to build the binary package. On the one hand, this is a nice design, because it abstracts away the large variety of upstream build systems into one API for dpkg-buildpackage to use. On the other hand, it puts all the intelligence for how packages are built into the packages. Making packaging intelligent, rather than the build tool, means packagers need to do more work, and there's more work to do when, say, the Debian policy changes, or when there are other changes that affect a large number of packages. If packaging is intelligent, then every package needs changes. If the build tool is intelligent, then you change the build tool and re-build everything. In Baserock we put as much intelligence as we can into the Morph tool, which drives the build process. It turns out that today, unlike in 1995, most upstream projects use one of a handful well-known build systems: autotools, cpan, cmake, Python distutils, etc. With just a little extra logic in Morph we avoid having any delta against upstream in Baserock. This doesn't work for quite all upstream projects, of course, but we've spent the 2% (two, not twenty) of effort that solves 80% of the problem. In recent years, the dh approach to packaging has made a lot of packages be able to have only a minimal, 3-line debian/rules file. This is excellent. Wouldn't it be nice if even that wasn't needed? It'd save thousands of files in source packages across the archive, at least. It would be easy to: if the file is missing, have dpkg-buildpackage assume the 3-line version by default. Getting rid of a single file is, of course, not a particularly big win. The big win is the change in mindset: rather than dealing with all new issues in development by adding yet more fields to debian/control and more optional, competing tooling outside the core toolset, if you improve the tools everyone uses, then everyone's packages gets better. The goal should, in my opinion, be that for the large number of packages where upstream uses a well-known, well-behaved build system, and uses it in a reasonably sensible way, the Debian source package should not require anything added to make the package build. There will still be a need to add some stuff, such as the debian/copyright file, to make a good Debian package, but just getting the package to build should require nothing extra. (Side note: wouldn't it be nice if there was a well-known, widely used way to mark up copyright information so that debian/copyright could be constructed and updated automatically?) Configuration file handling on upgrades: add ucf to dpkg already In the 1990s, dpkg had excellent handling of configuration files and merging local changes with changes from the new package version, but it was excellent only because it tried to do that at all, and mostly nothing else did. It hasn't changed much, since, and it's not excellent on any absolute scale. We have the ucf tool, which can do a better job, but requires to be added to each package wanting to use it. Why don't we make dpkg smarter, instead? If ucf is not not good enough to be merged wholesale into dpkg, let's write something better. Making every package maintainer use ucf manually is just wasteful. This is not the kind of thing that should be done for each package separately: the package manager should be smart so that packaging can be stupid. The goal should be that dpkg is smart enough in its configuration file handling that having the package do it is a very rare special case. Clean building shouldn't be hard The basic tool for building a package is dpkg-buildpackage. It is ever so slightly cumbersome to use, so there are some wrappers, most importantly debuild. However, if you're making a build intended to be uploaded to the Debian archive, you should be doing a clean build. This means having to learn, configure, and use yet more tools. A clean build is important: security updates, development, debugging, quality control, porting, and user support become more difficult if we don't know how a package was built, and can't reproduce the build. It gets harder to keep build dependencies correct, making it harder for everyone to build things. Luckily, Debian has solved the clean build problem. Unluckily, it has solved it multiple times, creating the problem of having to choose the clean building approach you want to use. The default way of building is not clean, so then you have to remember to use the non-standard way. You also get to spend time maintaining the clean build environments yourself, since that doesn't seem to be fully automated. None of this is hard, as such, but it's extra friction in the development workflow. The primary approaches for clean building in Debian seem to be pbuilder, cowbuilder, and sbuild. I happen to use pbuilder myself, because it's what's been around the longest, but I make no claim of having made an informed choice. That is part of the problem here: why should I have to spend the effort to become informed to make a choice well? Why is the default way of building not clean? Don't say performance: Morph sets up a clean staging area in seconds, and does not offer you a choice of not doing so. It's a chroot, with everything but the build tree hardlinked from cached, unpacked build dependencies, and protected using read-only bind mounts. What's more, this approach avoids having to maintain the pbuilder base tarballs or sbuild chroots manually. It's all automatic, and up to date, for every build. Furthermore, the staging area contains only the specified build dependencies, and not anything else, meaning a build fails if a build dependency is missing, rather than succeeding because it happens to be in the default set of packages the build tool installs. Mass building shouldn't be hard Suppose you want to try out a large-scale change in Debian. It might be trying out a new version of GCC, or using llvm's clang as the default C compiler, or updating glibc, or doing a large library transition such as a new version of GTK+, or trying a new dpkg version, or even something more exploratory such as changing the default optimisation flags for all packages to see what breaks. All of these will require you to at least rebuild all affected packages. Ideally you'd test the built packages as well, but let's concentrate on the building for now. Here's what you do. You make the changes you want to try out, and build those packages. You create a dedicated APT repository, and upload your packages there. You configure your build environment to favour that APT repository. You make a list of all reverse build dependencies of the packages you changed. You write a script to build all of those, preferably so that if you change A, and rebuild B, then you also rebuild C which build depends on B. Each rebuilt package you also upload to your APT repository. You keep track of the build log, and success or failure, of each build. The people in Debian who do this kind of stuff regularly presumably have tools for doing it. It shouldn't be a rare, special thing, though. If my package has reverse build dependencies, I should at least consider testing building when I'm making changes. Otherwise, it might take years until the reverse build dependencies are rebuilt, and the problem is only found then, making it harder to fix. To be fair, building a lot of packages takes a lot of resources. It's not feasible to rebuild everything in Debian every time there's any change to, say, eglibc. However, it's feasible to do it, for large subsets of the archive, without huge hardware investments. One VCS to rule them all In 1995 there was really only one relevant version control system: CVS. It was not a great tool. In 2000, another contender existed: Subversion. It fixed some problems in CVS, but still wasn't a great tool. In 2005, there was a great upheaval and distributed version control systems started to become mainstream. There were a large handful of them. In 2010, it was becoming pretty clear that git had won. It's ugly, but it's powerful. I'm not going to debate the relative merits of different version control systems. Until recently, I was a Bazaar boy, and all of my personal projects were kept in Bazaar. (I have recently switched everything to git.) There are, however, strong benefits from everyone using the same system. Developers don't need to learn a dozen version control systems. Tools that operate on many repositories are easier to write and maintain. Workflows become simpler if one system can be assumed. Debian has a strong historical tendency to choose every option. This is sometimes a good thing, and sometimes a bad thing. For keeping source packages in version control I believe it is to be a bad thing. The status quo is that a Debian source package may not be in version control at all, or it might be in any version control system. This is acceptable when everyone only ever needs to maintain their own packages. However, in a distribution the size of Debian, that is not the case. NMUs, security support, archive wide transitions, and other situations arise when I might need to change yours, or you might need to change mine. We try to work around this by having a complicated source package format, using quilt to maintain patches to upstream semi-manually in a debian/patches directory. This is an awkward workflow. It's a workflow that trips up those that are not used to it. (I know quilt is a patch management system, not a version control system. I think git does it much better anyway.) It would be oh so much easier if everyone kept their source packages in the same, real version control system. I don't even care what it is, as long as it is powerful enough to handle the use cases we have. Imagine a world where every Debian source package is kept, for argument's sake, git, and everyone also uses the same layout and roughly the same workflow to maintain it. What would this mean? It would mean that if you want to inspect the history of your package, you know how to do that. If you want to merge in some bugfix from upstream code, you know how to do that, without having to figure out which of the several source package formats are in use. It would make feasible the development of more powerful, more higher level tooling. For example, it would allow Debian to have what we call system branches in Baserock. In Debian we have stable, testing, unstable, and experimental. We may get something like Ubuntu's PPAs, or perhaps an improved version of those. These are very poor versions of system branches, just like quilt is a poor way to manage patches and deltas against upstream. For example, you can upload an experimental version of gccto experimental, but then nobody else can upload another experimental version. You can set up your own PPA for this, but you'll still be affected by all the uploads to unstable while you're working. A Baserock system branch is a branch of the entire system, or the entire distribution in the case of Debian. It is isolated from other system branches. In branch in an individual repository is a well-known concept. A system branch is conceptually like branching every repository in the distribution at once. The actual implementation is more efficient, of course. This would be possible to implement without standardising on one version control system, but it would be much harder to implement, and would have to live with the lowest common denominator for features. CVS and Subversion, for example, don't really to merges, where Bazaar, Mercurial, and git do. Possible does not mean feasible. Any work you do in a system branch is isolated. Your work doesn't affect others, and theirs doesn't affect yours, until a merge happens. This is a simple, but very powerful tool. Cheap system branches, and powerful merging, makes it possible to do experiments safely, with little fuss. Combine that with being able to build everything cleanly and quickly, and get into a situation where there's no need to have make technical decisions based on arguments on mailing lists, and instead they can be done by choosing by looking at working code. I don't know how this could be implemented in Debian, but think about it. If Debian could have this, it might many archive-scale changes easier. debian/rules clean: really? One of the silliest things we require of packages is that they have a debian/rules clean rule that cleans up after a build perfectly so that we can do repeated builds in the same source tree. Let's just use git clean -fdx instead. This is a problem that is superbly well suited for automation. There is no point whatsover making packagers do any manual work for this. Large scale semi-mechanical changes require too much effort About a decade ago, we decided to follow a new version of the Filesystem Hierarchy Standard and transition from /usr/doc to /usr/share/doc. This was an almost entirely mechanical change: in many cases, a mere rebuild would fix it, and in almost every other case it was just a minor tweak to the packaging. A one-line change. It took us seven years to do this. Seven years. Think about it. In a recent discussion about building R binary data files from source at package build time it was suggested that we take 2-3 release cycles to get this done. That's four to six years. Think about it. These are not isolated cases. Every time we need to make a change that affects more than a small handful of packages, it becomes a major undertaking. Most of the time all the people involved in this are agreeable to the change, and welcome it. The change takes long because it requires co-ordinating a large number of people. Sometimes people are just busy. Sometimes they've left the project, but haven't properly orphaned their packages. Waiting for a timeout for answers about packages drags the process longer. Mechanical changes, or semi-mechanical ones, which are very easy and very quick to do should not take years. They should take an evening, no more. There's no end of changes we might want to do like this. In 2005 and 2006 I filed a few hundred bugs from failed piuparts runs. They're still being fixed, even when the fix is simple, such as adding a postrm script to remove, when the package is purged, a configuration file created by postinst, or starting the service with invoke-rc.d rather than running an init.d script directly. Mostly one-line changes. There's more mechanical changes that might happen. For example, changing the value of the VCS-Browser field when the Debian version control server changes domain names. It's not just about simple, mechanical changes, either. Transitions of important library packages, for example, which require changes to reverse dependencies due to a changed API are another source of pain. What should be a few evenings of build fixes can drag out to months of co-ordination. This is caused partly by technical issues, and partly due to social issues. The main social issue is that we have quite a strong ownership of packages, and NMUs are to be done carefully, and only in some cases. This is both good and bad, and I won't discuss that aspect now. The technical issues are that our tools are primarily intended for maintaining individual packages, rather than all of them together, making it harder to make even simple changes in large number of packages. In addition to easy mass building and system branches, as outlined above, large-scale changes would require testing tools so that you don't build the new package versions, but also test things automatically. Essentially, CI at the distribution level. Conclusion I've listed above a small variety of problems I see in the Debian development processes and tools. They're not the important part of this talk. The important part is that it we, the Debian developers, should look at our tools, and workflows, critically, and improve them when we can. Even small improvements are very useful, when they affect each of our twenty thousand source packages. The important change I argue for here is one in mindset, rather than any specific problem in any tool. We need a mindset of constant, incremental improvement for our tools.

Russ Allbery: Review: Blue Remembered Earth

Review: Blue Remembered Earth, by Alastair Reynolds
Series: Poseidon's Children #1
Publisher: Ace
Copyright: January 2012
Printing: June 2012
ISBN: 0-441-02071-2
Format: Hardcover
Pages: 505
Geoffrey Akinya is one heir to the vast Akinya business empire, which straddles the solar system and has created vast riches from exploration and mining. He wants nothing to do with it. His passion is elephants: long and methodical study of wild herds in Africa, including a slow and careful investigation into their thought processes. His obnoxious and superior cousins have a passion for business and finance and can have the running of the company for all he cares. But Geoffrey's grandmother Eunice, the family matriarch and driving force behind much of their business expansion, has died. Her ashes are brought back to Africa from the Winter Palace in orbit around the Moon where she lived out her final days and scattered in a family ceremony. And, shortly thereafter, Geoffrey's cousins convince him to investigate a safe-deposit box left on the Moon by Eunice to ensure that it doesn't contain anything damaging to the family. What it contains is the first step in a puzzle. Despite himself, Geoffrey and his sister Sunday an artist and family black sheep who lives in a region of the Moon that is one of the sole holdouts against the ubiquitous monitoring common in the rest of the inhabited solar system are slowly pulled into unraveling that puzzle. This leads them into an uneasy alliance with one of the major political forces in the solar system and some startling revelations about their grandmother's actual plans. Blue Remembered Earth is another entry in the currently-popular sub-genre of solar system SF. Interstellar travel is still a dream, but a combination of improved technology, suspended animation, and heavy use of robots and robotic factories has let humans slowly expand into more of the solar system. On Earth, powers have risen and fallen, and the leap into the solar system coincided with a surging and powerful Africa. Extensive undersea colonization has established a new international consortium of sea-floor countries, and Earth politics have realigned into a diplomatic and business struggle between surface and oceanic nations. Unusually for this sort of fiction, most characters of significance in this story are African. The United States and Europe have quietly disappeared in the way that all non-US countries tend to in most other SF. Reynolds doesn't do very much with this setup (and the characters don't seem distinctly African to me). It's just there as the unremarked normal. I liked that. I kind of wish more of African culture had come through in the story, but there are enough SF novels with an assumed and unremarked white (and usually US) future that an assumed and unremarked black African culture is a nice change of pace. I picked up this book in part because it was described as a better 2312. That's a fairly accurate summary (although only helpful to those who have read Kim Stanley Robinson's novel). The scope and setting are similar, although Reynolds's future cuts back on the scope of off-world human colonization, relies more heavily on robotics, and keeps the power base on Earth. Both books are structured as a mystery, although in Blue Remembered Earth that mystery is deliberately planted for the characters. Reynolds is nowhere near as good at set pieces as Robinson, sadly. There's nothing here at the level of some of the scenes on Mercury in 2312. But he is much better at plotting and characterization, and Blue Remembered Earth tells a coherent story with increasing tension and a satisfying ending. Reynolds here uses an interesting strategy for spinning a mystery out of pieces of knowledge and discovery: set the discovery process prior to the novel, use protagonists who were unaware of it, and turn the details into a mystery that they have to investigate. This is an interesting way to address the problem that most discovery processes are slow and not particularly dramatic. It is artificial Eunice creates the plot for the novel but that worked for me. By the end of the book, I think I understand her reasons for doing so, and both Geoffrey and Sunday have good reasons to not be aware of any of the details before they start following the clues that Eunice leaves behind. The mystery of course sends Geoffrey and Sunday on a bit of a tour of the solar system as well as a tour of the past of their family business and the politics of the present. Here, Reynolds draws on an old SF idea: a conflict between groups that want to mechanize space exploration and groups that want to adapt humans for space. Long-time SF readers will be familiar with this approach, and I don't think Reynolds does anything startlingly new with it, but I thought it was a well-executed side plot. And, for once, an SF author realizes that Earth's oceans are as compelling of a place for that conflict to play out as outer space. There are, of course, significant mysteries to be uncovered, but it takes much of the book before those revelations start (even if one of them is almost spoiled in the dust jacket summary). The story up until that point is driven by character, particularly Geoffrey's stubborn annoyance at the rest of his family and his complete lack of interest in being the protagonist of a novel. I should probably warn here that many of the Amazon reviews disliked this book because of the characters, particularly Geoffrey. Apparently he comes across to some as whiny, resentful, and annoying. I, however, didn't have this reaction at all, possibly in part because I wholeheartedly share his opinion of both big business and his cousins. I thoroughly enjoyed reading an SF novel whose main characters are not driven either typical capitalist motives or a desire for personal exploration, but instead are pulled into the plot through a sense of loyalty and an inability to get out of the way of the problem. I also liked the way Reynolds manages to broaden and deepen the reader's appreciation for the characters as the story unfolds, and even salvages some characters I thought were unsalvageable. The last third of the book features some excellent plot twists and some very nicely-handled revelations. This is apparently the first book of a trilogy, and I had a suspicion at the end of the book that sequels were coming, but I think it stands on its own quite well. Most of the questions raised by the plot are resolved by the end of the book, and those that remain are of a size that other books have left unanswered intentionally. Despite not reaching the same moments of awe-inspiring beauty, and despite more workman-like descriptions, the stronger plot and more interesting characters do make Blue Remembered Earth a better 2312 for me. Given that 2312 won a Nebula, that's saying something (although I don't agree with that award). It's a bit deliberate in its pacing, and the plot built around a constructed puzzle can feel a bit artificial, but I think it's a solid and satisfying near-space SF novel. Followed by On the Steel Breeze. Rating: 7 out of 10

15 November 2013

Rudy Godoy: New GPG Key 65F80A4D

A bit late but I ve created a new 4096 RSA GPG key. I ve published a transition document as well. I hope I can meet fellow Debian developers soon to get my new key signed. So, if you are in town (Arequipa, Peru) drop me an email!

14 November 2013

Russ Allbery: Review: Water Logic

Review: Water Logic, by Laurie J. Marks
Series: Elemental Logic #3
Publisher: Small Beer Press
Copyright: 2007
ISBN: 1-931520-23-2
Format: Trade paperback
Pages: 332
Water Logic is the third book in the Elemental Logic series, starting with Fire Logic. As with Earth Logic, it's best to read all three books in close temporal proximity, since Water Logic offers little recap of what happened in the previous books and relies on a large cast of characters who are not reintroduced. It's also hard to review without spoilers for the previous books. Earth Logic ended with a monumental shift in Shaftal politics, and half of Water Logic is deeply concerned with the implications of that shift and where Shaftal should go from here. Karis and Zanja are, of course, deeply involved, but much of the political story is told from the perspective of the character introduced in Earth Logic, and from the perspective of another earth blood, Meriseth. Water bloods haven't been seen much so far in the series. Water magic is about repeated patterns and the malleability of time, and the plot of Water Logic eventually entangles Zanja in events far in the past of Shaftal. This proceeds in parallel with more of Marks's excellent analysis of the meaning of war and peace, diplomacy, cooperation, multiculturalism, adaptation, and empathy. The two sides of the book do to some extent reinforce each other, but the connections are tricky and less obvious. Parts of this book are excellent. Wars are complicated and powerful, and don't end easily. The mutual incomprehension and mutual (if intermittant) struggle for understanding between the natives of Shaftal and the Sainnites is beautifully presented. I particularly loved Mariseth's approach throughout the book. Earth bloods are perhaps Marks's best invention, and seeing another (besides Karis) at work, with the same stubborn patience but with different tactics and different levels of power, was a delight. I feel like reading, and thinking about, Seth's decisions makes me a better person. As with the previous books, though, parts of Water Logic are a bit boring, or at least don't seem to coherently move in a conclusive direction. Zanja has long journeys across the landscape as well as several adventures after she's grabbed by water logic that seem only distantly connected to the rest of the story. Marks does pull this mostly together in the end, and I could detect other thematic linkages while mulling the book over after finishing it, but the book feels a bit disjointed while reading it. And there are some elements, such as the cowdogs, that are neat bits of world-building but that seem tossed in just because they're neat and don't feel strongly connected to the plot. The other complaint I have, which would probably have been lessened by reading the books closer together, is that the cast is a bit too large. All of Karis's extended family are, of course, still around, which I found more confusing than helpful. Named and important characters keep popping in and then disappearing, with no introduction and no help for the reader who doesn't remember the exact details of their roles from previous books. It's a bit frustrating, and it makes the book feel a bit cluttered, although it does a great job of showing the dynamics of a large and comfortable family. (Marks shows the dynamics of family webs with multiple adults better than any other fantasy author I know.) Karis is a bit more sidelined in this book, casting her in the role of force of nature, and I think that works well. It took me a bit to warm to Zanja again, but by the end of the book I was greatly enjoying her sections. But the star, for me, is Seth. Even though some of her segments should have been the boring parts of the book, since they were mostly concerned with quitodian issues, they were the parts I wanted to read about. I was very touched by the ending, even if I think a bit too much authorial hand-waving was required. There is material left for a sequel at the end of Water Logic, including a villain who has yet to fully appear, but it's currently not clear whether Air Logic will ever materialize. But, as with Earth Logic, Water Logic provides a satisfying ending and leaves the world in a stable enough place that I didn't feel cheated out of the end of the story. I think Fire Logic remains the strongest book of this series, but Water Logic is very much worth reading if you've read the previous books. Unfortunately, the books apparently didn't sell well, so this book is from a small press instead of Tor. You may have to order it on-line instead of finding it a local bookstore. But the whole series is very much worth reading. It's a refreshingly original take on fantasy and is much more thoughtful about human relationships and politics than most fantasy works. Rating: 7 out of 10

Next.

Previous.