Search Results: "zed"

11 December 2013

Daniel Pocock: Get WebRTC going faster

On Saturday, Lumicall began offering free calls from browser to mobile using the free and open WebRTC technology. It should be no surprise that the service has been popular. Is it really free and open? The only way to prove this technology is free is to help people implement this for themself. On Monday, I uploaded reSIProcate v1.9.0 beta7 packages to Debian. The reSIProcate SIP proxy, repro, is one of the core components of the solution behind the free Lumicall service. Simply install the repro and resiprocate-turn-server packages using apt-get and make the following changes to the configuration (use your own IP addresses of course). I've taken this diff from my own runtime environment, only hiding my passwords, so that you can see exactly how I got it working:
--- repro.config.orig	2013-12-11 17:36:27.179228324 +0100
+++ repro-ws.sip5060.net.config	2013-12-11 17:48:24.159938649 +0100
@@ -143,6 +143,41 @@
 # Transport6TlsClientVerification = None
 # Transport6RecordRouteUri = sip:h1.sipdomain.com;transport=WS
 
+Transport1Interface = 195.8.117.57:80
+Transport1Type = WS
+Transport1RecordRouteUri = auto
+
+Transport2Interface = 2001:67c:1388:1000::57:80
+Transport2Type = WS
+Transport2RecordRouteUri = auto
+
+Transport3Interface = 195.8.117.57:5060
+Transport3Type = TCP
+Transport3RecordRouteUri = auto
+
+Transport4Interface = 2001:67c:1388:1000::57:5060
+Transport4Type = TCP
+Transport4RecordRouteUri = auto
+
+Transport5Interface = 195.8.117.57:443
+Transport5Type = WSS
+#Transport5RecordRouteUri = auto
+Transport5TlsDomain = ws.sip5060.net
+Transport5TlsClientVerification = None
+Transport5RecordRouteUri = sip:ws.sip5060.net;transport=WSS
+Transport5TlsCertificate = /etc/ssl/ssl.crt/ws.sip5060.net-bundle.crt
+Transport5TlsPrivateKey = /etc/ssl/private/ws.sip5060.net-key.pem
+
+Transport6Interface = 2001:67c:1388:1000::57:443
+Transport6Type = WSS
+#Transport6RecordRouteUri = auto
+Transport6TlsDomain = ws.sip5060.net
+Transport6TlsClientVerification = None
+Transport6RecordRouteUri = sip:ws.sip5060.net;transport=WSS
+Transport6TlsCertificate = /etc/ssl/ssl.crt/ws.sip5060.net-bundle.crt
+Transport6TlsPrivateKey = /etc/ssl/private/ws.sip5060.net-key.pem
+
+
 # Comma separated list of DNS servers, overrides default OS detected list (leave blank 
 # for default)
 DNSServers =
@@ -455,7 +490,7 @@
 ForceRecordRouting = false
 
 # Assume path option
-AssumePath = false
+AssumePath = true
 
 # Disable registrar
 DisableRegistrar = false
@@ -481,7 +516,7 @@
 # WARNING: Before enabling this, ensure you have a RecordRouteUri setup, or are using
 # the alternate transport specification mechanism and defining a RecordRouteUri per
 # transport: TransportXRecordRouteUri
-DisableOutbound = true
+DisableOutbound = false
 
 # Set the draft version of outbound to support (default: RFC5626)
 # Other accepted values are the versions of the IETF drafts, before RFC5626 was issued
@@ -505,7 +540,7 @@
 # WARNING: Before enabling this, ensure you have a RecordRouteUri setup, or are using
 # the alternate transport specification mechanism and defining a RecordRouteUri per
 # transport: TransportXRecordRouteUri
-EnableFlowTokens = false
+EnableFlowTokens = true
 
 # Enable use of flow-tokens in non-outbound cases for clients detected to be behind a NAT.  
 # This a more selective flow token hack mode for clients not supporting RFC5626.  The 

This is a diff against the /etc/repro/repro.config file distributed in the Debian package version 1.9.0~beta7-1. In the example above, I've included WSS transport defintions for WebSockets over TLS. Use the standard procedure for creating webserver SSL certificates to create certificates for repro and make sure you insert the correct filenames in the TLS parameters above. I've also duplicated every transport for IPv6. If you don't want TLS/WSS or IPv6, just comment those entries out (and renumber the remaining transports). Web-based SIP proxy setup Once you have repro running, go to the web admin interface (port 5080, username: admin, password: admin) and finish the setup using the web UI. The following steps are essential:
  • Add your domain (and restart the repro daemon after adding your domain)
  • Add some users
  • Add any routes to external services (optional - in my next blog I'll demonstrate how to route WebRTC calls to Asterisk using the Debian packages and less than 20 lines of configuration)
Set up reTurn, the TURN server Just put your IP addresses in /etc/reTurnServer.config and add the following line to /etc/reTurnServer-users.txt:
test:notasecret:reTurn:authorized
IMPORTANT: the realm in the users file (reTurn in the example and default config) must be identical to the AuthenticationRealm in the /etc/reTurnServer.config file. On your own web site Simply install your own apache server and clone the webrtc.lumicall.org demo site. Modify the file js/custom.js and include the settings for your own server.
# cd /var/www
# mkdir webcall
# cd webcall
# wget -r -nH http://webrtc.lumicall.org
# vi js/custom.js
In the custom.js, make sure you use a ws:// URL if you didn't set up SSL certificates and use a wss:// URL if you did. The IP or domain of your repro server must be in the ws:// or wss:// URL. Now navigate to the URL ending with /webcall on your server. For RHEL, Fedora and other RPM users Can somebody please assist with the review of the cajun-jsonapi dependency package so I can upload this new version of reSIProcate to Fedora? I'm also planning to make v1.9.0 available in EPEL6 when it is released in January. Questions? Please come and ask questions or discuss your experiences on the Free RTC mailing list sponsored by FSF Europe.

Gustavo Noronha Silva: WebKitGTK+ hackfest 5.0 (2013)!

For the fifth year in a row the fearless WebKitGTK+ hackers have gathered in A Coru a to bring GNOME and the web closer. Igalia has organized and hosted it as usual, welcoming a record 30 people to its office. The GNOME foundation has sponsored my trip allowing me to fly the cool 18 seats propeller airplane from Lisbon to A Coru a, which is a nice adventure, and have pulpo a feira for dinner, which I simply love! That in addition to enjoying the company of so many great hackers.
Web with wider tabs and the new prefs dialog

Web with wider tabs and the new prefs dialog

The goals for the hackfest have been ambitious, as usual, but we made good headway on them. Web the browser (AKA Epiphany) has seen a ton of little improvements, with Carlos splitting the shell search provider to a separate binary, which allowed us to remove some hacks from the session management code from the browser. It also makes testing changes to Web more convenient again. Jon McCan has been pounding at Web s UI making it more sleek, with tabs that expand to make better use of available horizontal space in the tab bar, new dialogs for preferences, cookies and password handling. I have made my tiny contribution by making it not keep tabs that were created just for what turned out to be a download around. For this last day of hackfest I plan to also fix an issue with text encoding detection and help track down a hang that happens upon page load.
Martin Robinson and Dan Winship hack

Martin Robinson and Dan Winship hack

Martin Robinson and myself have as usual dived into the more disgusting and wide-reaching maintainership tasks that we have lots of trouble pushing forward on our day-to-day lives. Porting our build system to CMake has been one of these long-term goals, not because we love CMake (we don t) or because we hate autotools (we do), but because it should make people s lives easier when adding new files to the build, and should also make our build less hacky and quicker it is sad to see how slow our build can be when compared to something like Chromium, and we think a big part of the problem lies on how complex and dumb autotools and make can be. We have picked up a few of our old branches, brought them up-to-date and landed, which now lets us build the main WebKit2GTK+ library through cmake in trunk. This is an important first step, but there s plenty to do.
Hackers take advantage of the icecream network for faster builds

Hackers take advantage of the icecream network for faster builds

Under the hood, Dan Winship has been pushing HTTP2 support for libsoup forward, with a dead-tree version of the spec by his side. He is refactoring libsoup internals to accomodate the new code paths. Still on the HTTP front, I have been updating soup s MIME type sniffing support to match the newest living specification, which includes specification for several new types and a new security feature introduced by Internet Explorer and later adopted by other browsers. The huge task of preparing the ground for a one process per tab (or other kinds of process separation, this will still be topic for discussion for a while) has been pushed forward by several hackers, with Carlos Garcia and Andy Wingo leading the charge.
Jon and Guillaume battling code

Jon and Guillaume battling code

Other than that I have been putting in some more work on improving the integration of the new Web Inspector with WebKitGTK+. Carlos has reviewed the patch to allow attaching the inspector to the right side of the window, but we have decided to split it in two, one providing the functionality and one the API that will allow browsers to customize how that is done. There s a lot of work to be done here, I plan to land at least this first patch durign the hackfest. I have also fought one more battle in the never-ending User-Agent sniffing war, in which we cannot win, it looks like.
Hackers chillin' at A Coru a

Hackers chillin at A Coru a

I am very happy to be here for the fifth year in a row, and I hope we will be meeting here for many more years to come! Thanks a lot to Igalia for sponsoring and hosting the hackfest, and to the GNOME foundation for making it possible for me to attend! See you in 2014!

10 December 2013

Russ Allbery: krb5-sync 3.0

krb5-sync is the software we run at Stanford to synchronize principal information from a central Heimdal realm to Active Directory, allowing users to use either a Linux-based Kerberos environment or Active Directory with the same account and password. The original intent of this release was to add a new feature to allow a subsidiary instance of an account in the MIT or Heimdal realm to be synchronized with the instance-less account in Active Directory. This allows, for example, an rra/windows instance to be used to set and maintain the password for an rra principal in Active Directory. In the process of implementing that, though, I ended up doing a significant overall of the code, since the plugin architecture was quite awkward and dated. The code now uses the MIT Kerberos data structures in a more natural and native way, since MIT Kerberos has now added direct support for plugins of this sort. Kerberos contexts and Kerberos error codes are used uniformly throughout the plugin, which provides consistent and more robust error handling and reporting. I also significantly enhanced the test suite, although it still needs more work to test the core functionality that has complex external dependencies. This release also drops support for all versions of MIT Kerberos prior to 1.9, which required an external patch; to run krb5-sync 3.0, you should upgrade to a recent version of MIT Kerberos. This allowed me to drop support for the legacy API. There are a couple of major backward-incompatible changes in this release (and both unfortunately are not handled automatically by the Debian package upgrade, since it's hard to find and safely modify KDC configuration). First, the ad_ldap_base configuration option is now mandatory when synchronizing account status and its meaning has changed. Previously, dc elements for the realm were appended to a provided partial base. Now, the complete DN of the root of the Active Directory tree should be provided. This is more flexible and more useful with a wider variety of Active Directory setups. Second, I took advantage of the backward-incompatibilities to change the module name to sync.so from krb5_sync.so, since the latter sounded weirdly redundant and verbose when installed in the Kerberos plugin directory. This will require a configuration change to the plugin configuration for the KDC or kadmin server. Also in this release are a couple of new options: ad_queue_only, which forces all changes to be queued for later processing instead of processed in real time, and syslog, which can be used to turn off the internal syslog logging of non-errors from the module. (This is mostly useful for test suites.) Now, password changes are queued on any Active Directory failure, not just a few oddly-distinguished ones. The previous behavior was rather specific to Stanford's needs, and queuing all password changes shouldn't pose any problems. Finally, the krb5-sync-backend utility program for manipulating the queued changes has been completely rewritten and is much cleaner. It now uses the Net::Remctl::Backend Perl module for command and option handling, so that module (provided with remctl 3.4 or later) must be installed. It also requires IPC::Run, which is available from CPAN. It uniformly supports a -d option to specify the queue location, and skips event files during processing that no longer exist. You can get the latest release from krb5-sync distribution page.

30 November 2013

Guido G nther: CrystalHD progress

Following up on my port of the crystalhd plugin to the gstreamer 1.0 api I realized that the CrystalHD repo is pretty dormant. After reading slomo's nice article about GStreamer and hardware integration and a short off list mail exchange I decided to split the GStreamer part out of the CrystalHD repo and to try to get the plugin into gst-plugins-bad. Since the kernel part is already in linux kernel's staging area there would not be much left in the repo except for the libcrystalhd library itself and the firmware blobs. So I split them out as well and started to clean them up a bit by moving it to autoconf/automake, dropping the need for a C++ compiler and adding symbol versioning among other things. So up to know video is still smooth with:
gst-launch-1.0 filesrc location=sample.mp4 ! decodebin ! xvimagesink
after jhbuilding up to gst-plugins-bad. There are #ifdefs for macosx and windows but I doubt they're functional but in case anybody is building libcrystalhd on these these platforms it'd be great to know if it still works. Should these efforts lead to the crystalhd plugin being merged into GStreamer getting the kernel driver out of staging would be a great next step. This blog is flattr enabled.

29 November 2013

Axel Beckert: PDiffs are still useful

probably just not as default. I do agree with Richi and with Michael that disabling PDiffs by default gives the big majority of Debian Testing or Unstable users a speedier package list update. I m though not sure, if disabling PDiffs by default would Additionally I want to remind you that PDiffs per se are nothing bad and should be continued to be supported: So yes, disabling PDiffs by default is probably ok, but the feature must be kept available for those who haven t 100 MBit/s fibre connection into their homes or are sitting just one hop away from the next Debian mirror (like me at work :-). Oh, and btw., for the very same reasons I m also a big fan of debdelta which is approximately the same as PDiffs, just not for package lists but for binary packages. Using debdelta I was able to speed up my download rates over EDGE to up to virtual 100 kB/s, i.e. by factor four (depending on the packages). Just imagine a LibreOffice minor update at 15 kB/s average download rate. ;-) And all these experiences were not made with a high-performance CPU but with the approximately 5 year old Intel Atom processor of my ASUS EeePC 900A. So I used PDiffs and debdelta even despite having a slight performance penalty on applying the diffs and deltas.

28 November 2013

Dirk Eddelbuettel: RcppCNPy 0.2.1

Version 0.2.1 of RcppCNPy is now on CRAN earlier. This version is a minor upgrade with a few small changes as detailed in the NEWS file:
Changes in version 0.2.1 (2013-11-28)
  • Synchronized code with the cnpy repository
  • Added new function to test from R whether integers supported
  • Updated tests for integer support, if available
  • Updated vignette and discussion about need for rebuilding only RcppCNPy with the -std=c++11 flag if integer support is desired
  • Updated tests for integer support, if available
  • Updated THANKS file
  • Ensure that inclusion of Rinternal.h does not affect other headers by adding RF_NO_REMAP
CRANberries also provides a diffstat report for 0.2.1 relative to 0.2.0. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 November 2013

Steinar H. Gunderson: Stockfish

The open-source chess program Stockfish recently climbed to the top of the TCEC rating lists. Even though it does not seem likely to win the actual TCEC tournament (it is 15.5 11.5 down in the 48-game superfinal against Komodo, a closed-source commercial program, after winning 4, losing 8 and drawing 15), this is a tremendeous achievement. This is not only because it is highly unusual that the strongest chess program is open-source (it's naturally hard to keep a competitive edge when everybody can see your secrets!), but also because it is a mere 9000 lines of C++ code. (This is without a GUI; it interfaces through a standardized protocol known as UCI.) In comparison, Crafty, a far weaker open-source program, is about four times that, GNU Chess is above 40000, and I seem to recall that the Deep Blue program was something like 65000 lines (in addition to, of course, all the code for the ASIC chess chips it contained, and stood for the majority of its chess strength!). So, the development team is really to be commended. It should also be said that TCEC is testing the latest beta versions of most of these engines and while Stockfish is developed in the open, on GitHub, the latest versions of e.g. Komodo and Houdini are private. (This is probably one of the reasons why Stockfish is slightly lower on e.g. the CCRL list, which also computes chess program rating lists by playing games between them.) Now, why does anyone care? Only a very small handful of players would be able to score even a draw against any of these programs in a 10-game match, so does it really matter which one we choose? Or is this some obscure pissing contest? The answer, of course, is that these engines are not primarily useful for competitive play; they are useful for analysis. Almost any player, right down from beginners to the very top players, can benefit from having someone stronger than them look at their games (or others' games!) and say where things went wrong. (Somehow analyzing losses always seem more instructive than analyzing wins. Nobody ever got better by winning, as the saying goes.) And the better a program is at playing chess, the more likely is its analysis to be correct. Of course, for the weaker players, a chess coach is likely to produce much more understandable guidance ( this creates a weak square on c7 is much more palatable than after these 18 moves here, white is 0.48 pawns stronger than you, so this was a bad move , and engines produce the latter sort of output), but they are always available, incredibly strong, and cost much less than a coach. :-) They also add a lot to the otherwise quite hard-to-comprehend activity of watching high-level human players; while the computers do not understand every kind of position equally well, they at least provide some sort of objective verdict on which player has the upper hand, something which can be really hard to discern unless you have very strong players around for commentary. So, in trying to improve my skills a bit, I've been analyzing my games with engines for a while, and recently I've switched from Rybka to Stockfish. (Rybka, a closed-source program, used to be the clear leader of the pack, but after a huge amount of drama involving accusations of who has copied code from who, it seems no longer to be actively developed, and Stockfish is now somewhat stronger.) This, unfortunately, brings me to the things I don't like too much about Stockfish; it seems to behave more erratically than Rybka did in interactive analysis mode. Its score jumps a lot more around when I move something (worse at remember the analysis from the previous position?), and sometimes, it changes its mind a lot when I move back and forth between moves maybe the huge selectivity that makes chess engines so strong these days also make them behave less usefully when only looking at a move for a few seconds. So, even though Stockfish is a fantastic chess engine, and will only get better in the future, maybe I'll go back to Rybka at some point. But then again, maybe I should then check out Komodo or any of the other engines out there or maybe talk to someone who actually knows chess, that would help a lot, too =)

22 November 2013

Vincent Sanders: Error analysis is the sweet spot for improvement

Although Don Norman was discussing designers attitude to user errors I assert the same is true for programmers when we use static program analysis tools.

The errors, or rather defects in the jargon, that a static analysis tools produce can be considered low cost well formed bug reports available very early in the development process.

When I say low cost it is because they can be found by a machine without a user or fellow developer wasting their time finding them. Well formed comes because the machine can describe exactly how it came to the logical deduction leading to the defect.
IntroductionStatic analysis is in general terms using the computer to examine a program for logic errors beyond those of pure syntax before it is executed. Examining a running program for defects is known as dynamic program analysis and while a powerful tool in its own right is not the topic of discussion.

This analysis has historically been confined to compiled languages as their compilers already had the Abstract Syntax Tree (AST) of the code available for analysis. As an example the C language (released in 1972) had the lint tool (released in 1979) based on the PCC compiler.

Practical early compilers (I am generalising here as the 19070s were a time of white hot innovation in computing and examples of just about any innovation in the field could probably be found) were pretty primitive and produced executables which were less good than hand written assembler output. Due to practical constraints the progress of optimising compilers was not as rapid as might be desired so static analysis was largely used as an external process.

Before progressing I ought to explain why I just mixed the concept of an optimising compiler and static analysis. The act of optimisation within those compilers requires program analysis, from which they can generate defect reports which we all know and love as compiler warnings, also explaining why many warnings only appear at higher optimisation levels where deeper analysis is required.

The attentive reader may now enquire as to why we would need external analysis tools when our compilers already perform the task. The answer stems from the issue that a compiler is trying to reconcile many desirable traits including:
The slow progress in creating optimising compilers initially centred around the problem of getting the compiled output in a reasonable time to allow for a practical edit-compile-run-debug cycle although the issues more recently have moved more towards the compiler implementation costs.

Because the output generation time is still a significant factor compilers limit the level of static analysis performed to that strictly required to produce good output. In standard operation optimising compilers do not do the extended analysis necessary to find all the defects that might be detectable.

An example: compiling one 200,000 line C program with the clang (v3.3) compiler producing x86 instruction binaries at optimisation level 2 takes 70 seconds but using the clang based scan-build static analysis tool took 517 seconds or more than seven times as long.
Using static analysis
As already described warnings are a by-product of an optimising compilers analysis and most good programmers will endeavour to remove all warnings from a project. Thus almost all programmers are already using static analysis to some degree.

The external analysis tools available can produce many more defect reports than the compiler alone as long as the developer is prepared to wait for the output. Because of this delay static analysis is often done outside the usual developers cycle and often integrated into a projects Continuous Integration (CI) system.

The resulting defects are usually presented as annotated source code with a numbered list of logical steps which shows how the defect can present. For example the steps might highlight where a line of code allocates memory from the heap and then an exit path where no reference to the allocated memory is kept resulting in a resource leak.

Once the analysis has been performed and a list of defects generated the main problem with this technology rears its ugly head, that of so called "false positives". The analysis is fundamentally an undecidable problem (it is a variation of the halting problem) and relies on algorithms to generate approximate solutions. Because of this some of the identified defects are erroneous.

The level of erroneous defect reports varies depending on the codebase being analysed and how good the analysis tool being used is. It is not uncommon to see false positive rates, even with the best tools, in excess of 10%

Good tools allow for this and provide ways to supply additional context through model files or hints in the source code to suppress the incorrect defect reports. This is analogous to using asserts to explicitly constrain variable values or a type cast to suppress a type warning.

Even once the false positives have been dealt with there comes the problem of defects which while they may be theoretically possible take so many steps to achieve that their probability is remote at best. These defects are often better categorized as a missing constraint and the better analysis tools generate fewer than the more naive implementations.

An issue with some defect reports is that often defects will appear in a small number of modules within programs, generally where the developers already know the code is of poor quality, thus not adding useful knowledge about a project.

As with all code quality tools static analysis can be helpful but is not a panacea code may be completely defect free but still fail to function correctly.
Defect DensityA term that is often used as a metric for code quality is the defect density. This is nothing more than the ratio of defect to thousands of lines of code e.g. a defect density of 0.9 means that there is approximately one defect found in every 1100 lines of code.

The often quoted industry average defect density value is 1, as with all software metrics this can be a useful indicator but should not be used without understanding.

The value will be affected by improvements in the tool as well as how lines of code are counted so is exceptionally susceptible to gaming and long term trends must be treated with scepticism.
Practical examplesI have integrated two distinct static analysis tools into the development workflow for the NetSurf project which I shall present as case studies. These examples show a good open source solution and a commercial offering highlighting the issues with each.

Several other solutions, both open source and commercial, exist many of which have been examined and discarded as either impractical or proving less useful than those selected. However the investigation was not comprehensive and only considered what was practical for the project at the time.
clangThe clang project is a frontend to the LLVM project providing an optimising compiler for the C, C++ and objective C languages. As part of this project the compiler has been enhanced to run a collection of "checkers" which implement various methods of analysis on the code being compiled.

The "scan-build" tool is provided to make the using these features straightforward. This tool generates defect reports as a series of html files which show the analysis results.


NetSurf CI system scan-build overview
Because the scan-build takes in excess of eight minutes on powerful hardware the NetSurf developers are not going to run this tool themselves as a matter of course. To get the useful output without the downsides it was decided to integrate the scan into the CI system code quality checks.

NetSurf CI system scan-build result list
Whenever a git commit happens to the mainline branch and the standard check build completes successfully on all target architectures the scan is performed and the results are published as a list of defects.

The list is accessible directly through the CI interface and also incorporates a trend graph showing how many defects were detected in each build.

A scan-build report showing an extremely unlikely path to a defect
Each defect listed has a detail link which reveals the full analysis and logic necessary to cause the defect to occur.

Unfortunately even NetSurf which is a relatively small piece of software (around 200,000 lines of code at time of writing) causes 107 defects to be emitted by scan-build.

All but 25 of the defects are however "Dead Store" where the code has a value assigned but is never checked. These errors are simply not interesting to the developers and are occurring in code generated by a tool.

Of the remaining defects identified the majority are false positives and several (like the example in the image above) are simply improbable requiring a large number of steps to reach.

This shows up the main problem with the scan-build tool in that there is no way to suppress certain checks, mark defects as erroneous or avoid false positives using a model file. This reduces the usefulness of these builds because the developers all need to remember that this list of defects is not relevant.

Most of the NetSurf developers know that the project currently has 107 outstanding issues and if a code change or tool improvement were to change that value we have to manually work through the defect list one by one to check what had changed.
CoverityThe coverity SAVE tool is a commercial offering from a company founded in the Computer Systems Laboratory at Stanford University in Palo Alto, California. The results of the original novel research has produced a good solution which improved on analysis tools previously available.

Coverity Interface showing summary of NetSurf analysis. Layout issues are a NetSurf bug
The company hosts a gratis service for open source projects, they even provide scans for the Linux kernel so project size does not appear to be an issue.

The challenges faced integrating the coverity tool into the build process differed from clang however the issue of execution time remained and the CI service was used.

The coverity scanning tool is a binary executable which collects data on the build which is then submitted to the coverity service to be analysed. This tool obviously relies upon the developer running the executable to trust coverity to some degree.

A basic examination of the binary was performed and determined the executable was not establishing network connections or performing and observably undesirable behaviour. From this investigation the decision was made that running the tool inside a sandbox environment on a CI build slave was safe. The CI system also submits the collected results in a compressed form directly to the coverity scan service.

Care must be taken to only submit builds according to the services Acceptable Use Policy which limits the submission frequency of NetSurf scans to every other day. To ensure the project stays within the rules the build performed by the CI system is manually controlled and confined to a subset of NetSurf developers.

Coverity connect defect management console for NetSurfThe results are presented using the coverity connect web technology based defect management tool. Access to the coverity connect interface is controlled by a user management system which precludes publicly publishing the results within the CI system.

Unfortunately NetSurf itself does not currently have good enough JavaScript DOM bindings to support this interface so another browser must be used to view it.

Despite the drawbacks the quality of the analysis results is greatly superior to the clang solution. The false positive rate is very low while finding many real issues which had not been previously detected.

The analysis can be enhanced by use of collection configuration and modelling files which remove intended constructions from consideration reducing the false positive rate to very low levels. The ability to easily and persistently suppress false positives through the web interface is also available.

The false positive management capabilities coupled with a user interface that makes understanding the defect path simple make this solution very practical and indeed the NetSurf developers have removed over 50 actual issues within a relatively short period since the introduction of the tool.

Not all of those defects could be considered serious but they had the effect of encouraging deeper inspection of some very dubious smelling source.
ConclusionsThe principle conclusions of implementing and using static analysis have been:

When I started looking at this technology I was somewhat dubious about its usefulness but I have definitely changed my mind. It is a useful addition to any non-trivial project and the return on time and effort should be repaid handsomely in all but already perfect code (if you believe you have such code I have a bridge to sell you).

19 November 2013

Carl Chenet: Yarn: a scenario testing tool

Follow me also on Twitter Those who follow me on Twitter may have read my recent tweet about Yarn, a really nice software by Lars Wirzenius I discovered the last week-end while attending the Mini-Debconf UK in Cambridge. To make a long story short, Yarn allows the user to write scenarios of the executions of softwares. This scenario will be automatically played by Yarn which also returns the overview of the execution, with the goal to test the behaviour of these softwares. 1. Small example of how to use Yarn Let s have a look of how to use Yarn with a real example. The scenario will test if the software Brebis, a fully automatized backup checker, executes in an expected way. We can proceed as follow :
  1. set up the test environment
  2. launch a command with the software under test in the test environment
  3. verify the execution of the test
This is an example of a Yarn scenario file, the file is formatted using the Markdown syntax:
    SCENARIO basic brebis execution
    GIVEN setting up brebis
    AND generating backup configuration with brebis
    WHEN brebis verifies a backup
    THEN verify brebis output
    IMPLEMENTS GIVEN setting up brebis
    hg clone http://hg.brebisproject.org $DATADIR/brebis
    mkdir -p $DATADIR/brebis/yarn-test
    cp $DATADIR/brebis/functional-tests/expected-generated-list-for-tar-archive/expected-generated-list-for-tar-archive.tar.gz $DATADIR/brebis/yarn-test
    IMPLEMENTS GIVEN generating backup configuration with brebis
    $DATADIR/brebis/brebis.py -G $DATADIR/brebis/yarn-test/expected-generated-list-for-tar-archive.tar.gz
    IMPLEMENTS WHEN brebis verifies a backup
    $DATADIR/brebis/brebis.py -c $DATADIR/brebis/yarn-test -l $DATADIR/brebis/yarn-test/brebis.log
    IMPLEMENTS THEN verify brebis output
    if [ -s $DATADIR/brebis/yarn-test/brebis.log ]; then return 1; else return 0; fi
2. Result of the execution of the Yarn scenario Yarn provides at the end of the scenario an overview of the execution, as show below: $ yarn brebis-scenario
Scenario test suite PASS, with 1 scenarios (4 total steps), in 16.4 seconds 3. Details of the Yarn scenarios The five first lines are basically your scenario. Each step takes a keyword and a sentence to identify a step. You define a name with the SCENARIO keyword. The next step, GIVEN, allows generally the setup of the test environment. It is possible of course to define different steps with the keywork AND. Once the environment setup has been completed, we launch the test with the keyword WHEN. At last we check the result of the execution with THEN. It is worth noting that if you need further steps after checking the result of the execution, you can use FINALLY. Very helpful while setting up the test environment is the variable $DATADIR, automatically initialized by Yarn. It provides the path to a temporary directory where you can store files during the execution of your scenario. This directory is by default removed at the end of the scenario. Now the real trick: at each step I described above is associated shell commands to execute thanks to the IMPLEMENTS keyword.It s in my opinion a great way to increase the understanding of what the scenario really does. In a short glance of the first lines, if the names of the steps are carefully chosen, you will understand what the scenario does without needing to read the actual shell code. Simple and efficient. 4. More about Yarn Yarn is written in Python and now available in Debian in the package cmdtest. The sources are available here. In my opinion, if you were looking for this kind of tool, Yarn is in a really good direction. This project is still young but the core features do the job and the upstream is eager to receive feedbacks (and patches) :) Some links: Interested in Yarn? What do you think about it? Let us know in the comments of this post.

13 November 2013

Julien Danjou: OpenStack Design Summit Icehouse, from a Ceilometer point of view

Last week was the OpenStack Design Summit Icehouse in Hong-Kong where we, OpenStack developers, discussed and designed the new OpenStack release (Icehouse) that is coming up. The week has been wonderful. It was my second OpenStack design summit, and I loved it. Bumping into various people I've never met so far and worked with online was a real pleasure. As it was to meet again with fellow OpenStack developers! The event organisation was great, as were the parties. :-) On the last day, I had the chance to present a talk with Eoghan Glynn and Nick Barcet how we built the auto-scaling feature in Heat, implementing the "alarming" feature needed in Ceilometer.
The slides and video of the talk is online on my talks page. Design sessions This time, Ceilometer design sessions were spread on 3 days. Everything we talked about has its Etherpad instance. The discussions were interesting, and the amount of feedback gathered is big and is going to be very useful. There's a lot of people and companies using Ceilometer now, and the project is getting more and more traction in general. There's a lot of different way to use it and to bend it to one's needs. Considering the amount of features and options that is provided, building functionality with a genericized approach it making Ceilometer useful for a lot of different and interesting use-cases. Icehouse roadmap The list of blueprints targeting Icehouse is available, but not yet complete. I expect people to start filling this list in the next days.If you want to propose blueprints, you're free to do so and inform us about it so we can validate it. The same applies if you wish to implement one of them! Thereafter, I try to guess what the roadmap will look like in the upcoming weeks for Ceilometer based on the discussion we had last week during the summit. Events management A lot of work is going to be put into event management. Ceilometer plans to store notifications sent using oslo.messaging by OpenStack projects. Some work already got merge for Havana, but the API part and future improvements and ideas will continue to flow into the Icehouse release. Agents and group management A lot has been discussed around the polling agents and around the alarm evaluator agent. The current state of the ceilometer-central-agent disallows any kind of high-availability and load-balancing, as the polling task are kept and scheduled on only one node. The high-availability part is already covered by a custom mechanism built into ceilometer-alarm-evaluator, but it came clear to us that a more generic approach is needed. A lot of other projects needs this kind of functionality, and a pattern have been pointed out. A blueprint about group membership has been discussed in an Oslo session, and will end into a new Python library written to solve this in Ceilometer and in other projects. TaskFlow will also probably be leveraged to solve the task distribution issue. Documentation Since a few weeks, Ceilometer auto-generates its API reference documentation using sphinxcontrib-docbookrestapi that parses our API code that uses WSME. We also want to start writing a user guide, and we'll do that inside our own repository. That way, I hope that we will be the first project in OpenStack to require documentation to be incorporated into every patch that's being sent to Ceilometer. This is the best way to assure that nothing can be changed nor added without being accompanied with a documentation update. Tempest testing Testing of Ceilometer already has been a subject during the previous design summit about testing. We already put a large effort on Tempest testing in this last cycle, but we encountered a lot of small issues that we had to tackle to achieve something. Some Ceilometer basic tests are already on their way into Tempest, so this is something that is going to be achieved very soon. Ultimately, I would also want Ceilometer moving towards providing its own set of Tempest tests as part of the code base. That way, it'd be as easy for core reviewers to refuse a patch if it doesn't provide functional tests as it is to refuse it if it doesn't provide unit tests. As we'll do for the documentation. API improvements Once again, a few API improvements will probably be implemented, like aggregation or the ability to specify multiple queries with OR and AND operators. Roll-up, archiving of data There seems to be interest in archiving and rolling-up the data stored by Ceilometer, so work in this area is to be expected. Supporting multiple data storage driver in parallel seems to be something that needs to be done for this and other aspects of Ceilometer feature set. Alarming The alarming feature set is already big, and the work that has been accomplished pretty amazing. A few improvements will be made, as retrieving better metrics and building better statistics (exclusion of low quality data points).

11 November 2013

Simon Josefsson: Using Replicant on Samsung Galaxy S III

For the last half-year I have used CyanogenMod on an Nexus 4 as my main phone. Recently the touch functionality stopped working on parts of the display, and the glass on the back has started to crack. It seems modern phones are not built to last. For comparison, before the N4 I used a Nokia N900 for around 3 years without any hardware damages (in my drawer now, still working). A few weeks ago I started looking for a replacement. My experience with CyanogenMod had been good, but the number of proprietary blobs on the N4 concerned me. Finding something better wasn t easy though, so I m documenting my experience here. My requirements were, briefly, that I wanted a phone that I could buy locally that had a free software community around it that produced a stable environment. I have modest requirements for things I wouldn t give up on: telephony, data connection (3G), email (IMAP+SMTP), chat (XMPP), and a web browser. I like the philosophy and openness around the Firefox OS but the more I have read about it, it seems unlikely that it would deliver what I need today. In particular none of the devices capable of running Firefox OS appealed to me, and the state of email reading seemed unclear. I m sure I ll revisit Firefox OS as an alternative for me in the future.

As I had been happy with CyanogenMod, but concerned about its freeness, it felt natural to move on and test the more free software friendly project Replicant. Replicant only supports a small number of devices. After talking with people in the #replicant IRC channel, it seemed the Samsung S3 would be a decent choice for me. The Samsung S2 would have worked as well, but it cost almost as much as the S3 where I looked. Despite the large number of Samsung S3 devices out there, it seems the prices even for used devices are high (around 2500 SEK in Sweden, ~380 USD). I ended up buying a brand new one for 3200 SEK (~500 USD) which felt expensive, especially after recalling the recent $199 sale for Nexus 4. Noticing that brand new Nexus 4 devices are still over 3000 SEK in Sweden comforted me a bit. I would have preferred a more robust phone, like the CAT B15, but the state of free software OSes on them seem unclear and I wanted something stable. So, enough about the background, let s get started. Building and installing Replicant on the device was straight forward. I followed the Replicant Samsung S3 Build instructions to build my own images. The only issue I had was that I had not set JAVA_HOME and the defaults were bad; make sure to set JAVA_HOME before building. I built everything on my Lenovo X201 running Debian Wheezy, with OpenJDK 6 as the Java implementation. Installing the newly built firmware was easy, I just followed the installation process documentation. I made sure to take a clockworkmod backup to an external SD card before wiping the old system. To get a really clean new device, I also re-formated /sdcard inside clockworkmod; I noticed there were some traces left of the old system there. I spent about one week testing various configurations before settling on something I could use daily. A fair amount of time was spent looking into backup and restore options for Android devices. My idea was that I would take a backup of the apps I ran on the N4 and transfer them to the S3. The Android Debug Bridge (adb) has a backup/restore command, however it (intentionally) ignores apps marked as allowBackup=false which a number of apps has. It doesn t seem possible to override that settings so much for the freedom to backup your own device. I then discovered oandbackup. It can backup your entire system, saving each app (together with associated data) into a separate directory, for simple review and inspection before restore. You can do batch backups and batch restore. I couldn t get it to automatically restore things, though, which would be neat for really automated re-installations (there is an open issue about this feature). After noticing that some apps did not like being moved from the N4 (running Android 4.2) to the S3 (running 4.0), I ended up installing most apps from scratch on a freshly installed Replicant. I use oandbackup to the external SD card so that I can quickly restore my phone. For backup/restore of SMS/MMS and Call Log, I use SMS Backup+ against my own IMAP server. Camera pictures are synced manually using adb when I am connected to my laptop. There is a number of apps that deserve to be mentioned because they are what I use on a daily basis. All of them come via the free software market F-Droid. For email (IMAP/SMTP), I use K-9 Mail which is feature rich but still easy to use. For chat, I use Xabber. I use NewsBlur s free software app to read RSS flows. For two-factor authentication, I use Google Authenticator. I haven t evaluated different PDF viewers, but the first one I tried (APV PDF Viewer) has worked fine so far. Handling a a synchronized address book and calendar deserve its own blog post because it is a challenging topic, but briefly, I m currently using a combination of aCal and DAVdroid. Finally, since Replicant is still work in progress, some words about stability and notes on what doesn t work. This is probably the most interesting part if you are considering running Replicant on an S3 yourself. Overall system stability is flawless, I hadn t had any crash or problem with the fundamental functionality (telephony, 3G, Camera). People have said graphics feels a bit laggy, but I cannot compare with the stock ROM and it doesn t get in the way of daily use. First some notes about non-free aspects:
  • Bluetooth doesn t work by default. After installing /system/vendor/firmware/bcm4334.hcd (MD5 b6207104da0ca4a0b1da66448af7ed4c) pairing and testing with a Bluetooth headset worked fine.
  • Front camera doesn t work by default. After installing /system/vendor/firmware/fimc_is_fw.bin (MD5 52eeaf0889bc9206860075cd9b7f80b9and) and /system/vendor/firmware/setfile.bin (MD5 0e6fdeb378fb154c39fd508ae28eaf2a) it works. The extensions are *.bin but I don t believe this code is executed on the main CPU.
  • GPS doesn t work by default. After installing /system/bin/gpsd (MD5 6757ed2e2a283259e67c62e6b2b9cfef), /system/lib/libsecril-client.so (MD5 a836df0f947d2aa2ef20dcb213317517), /system/lib/hw/gps.exynos4.so (MD5 1ea1d67f297dd52d59d40dbad9b02a0a) it works. This is code that runs on the main CPU, and even more alarming, it embeds a copy of OpenSSL and talks to various online servers for A-GPS, and is thus a likely (and probably not too challenging) attack vector for anyone wanting to remotely exploit any phone.
  • Wifi doesn t work, and I haven t gotten this to work. There is a list of non-free S3 firmware on the Replicant wiki however my stock ROM did not ship with those files. I don t believe any of the blobs run on the main CPU.
  • NFC doesn t work, and I haven t gotten it to work. It seems the infrastructure for NFC is missing in Replicant 4.0, it doesn t even expose the NFC hardware permission capability. This is quite unfortunate for me, since I daily work with YubiKey NEO and would have preferred to replace Google Authenticator with the YubiOATH that uses the NEO for OATH secret storage.
Some other observations:
  • Panorama mode in the Camera crashes; see issue about it.
  • There is a number of smaller graphical issues. I believe these are related to the EGL but haven t understood the details. What I ve noticed are the following issues. The task switcher doesn t show mini screenshots of all running apps (the screenshots are just black). ZXing is not able to QR decode images (I m told this is because Replicant uses a RGB color plane instead of the required YUV color plane). Video playback in the gallery is laggy to the point of being unusable. Video playback on Youtube in the default web browser works in full screen (not laggy), but not embedded in the webpage.
  • MTP has been a bit unreliable, my main laptop is able to import photos, but another laptop (also running Debian Wheezy) just stalls when talking to it. This may be a host issue, I have experience similar issues with a Nexus 7 2nd generation device.
I am quite happy with the setup so far, and I will continue to use it as my primary phone. flattr this!

Vincent Bernat: Snimpy: SNMP & Python

While quite old fashioned, SNMP is still an ubiquitous protocol supported by most network equipments. It comes handy to expose various metrics, like network interface counters, to be gathered for the purpose of monitoring. It can also be used to retrieve and modify equipments configuration. Variables exposed by SNMP agents (servers) are organized inside a Management Information Base (MIB) which is a hierarchical database1. Each entry is identified by an OID. Querying a specific OID allows a manager (client) to get the value of an associated variable. For example, one common MIB-module is IF-MIB defined in RFC 2863. It contains objects used to manage network interfaces. One of them is ifTable whose rows are representing agent s logical network interfaces. Each row will expose the interface name, characteristics and various associated counters.
ifIndex ifDescr ifPhysAddress ifOperStatus ifOutOctets
1 lo up 545721741
2 eth0 0:18:f3:3:4e:4 up 78875421
3 eth1 0:18:f3:3:4e:5 down 0
ifTable is indexed by its first column ifIndex. If you want to get the operational status of the second interface, you need to query IF-MIB::ifOperStatus.2 which is translated to OID .1.3.6.1.2.1.2.2.1.8.2 using information provided by the MIB definition.

Scripting SNMP An SNMP agent can deliver a lot of interesting information: You can gather those information manually with tools like snmpget and snmpwalk:
$ snmpwalk -v 2c -c public localhost IF-MIB::ifDescr      
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
However, building robust scripts with them is quite challenging. For example, if you wanted to get the descriptions of all active interfaces as well as the total number of octets transmitted, you could do something like that:
#!/bin/sh
set -e
host="$ 1:-localhost "
community="$ 2:-public "
args="-v2c -c $community $host"
for idx in $(snmpwalk -Ov -OQ $args IF-MIB::ifIndex); do
    descr=$(snmpget -Ov -OQ $args IF-MIB::ifDescr.$idx)
    oper=$(snmpget -Ov -OQ $args IF-MIB::ifOperStatus.$idx)
    in=$(snmpget -Ov -OQ $args IF-MIB::ifInOctets.$idx)
    out=$(snmpget -Ov -OQ $args IF-MIB::ifOutOctets.$idx)
    [ x"$descr" != x"lo" ]   continue
    [ x"$oper" = x"up" ]   continue
    echo $descr $in $out
done
Hopefully, SNMP bindings in various languages are pretty common. For example, Net-SNMP ships with a Python binding:
import argparse
import netsnmp
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
args =  
    "Version": 2,
    "DestHost": options.host,
    "Community": options.community
 
for idx in netsnmp.snmpwalk(netsnmp.Varbind("IF-MIB::ifIndex"),
                            **args):
    descr, oper, cin, cout = netsnmp.snmpget(
        netsnmp.Varbind("IF-MIB::ifDescr", idx),
        netsnmp.Varbind("IF-MIB::ifOperStatus", idx),
        netsnmp.Varbind("IF-MIB::ifInOctets", idx),
        netsnmp.Varbind("IF-MIB::ifOutOctets", idx),
        **args)
    assert(descr is not None and
           cin is not None and
           cout is not None) #  
    if descr == "lo":
        continue
    if oper != "1": #  
        continue
    print("     ".format(descr, cin, cout))
This binding is quite primitive and has several drawbacks:
  1. It exports everything as strings. See .
  2. Error handling is just deficient. If you mispell something, like a variable name, you ll get snmp_build: unknown failure on the standard error. No exception. If a variable does not exist, you ll get None instead. See .
This inability to sanely handle failures makes this binding quite dangerous to use in scripts. Imagine making important modifications on the basis of the returned values. If you forget to check against None, your script may cause havoc!

Snimpy Because I didn t find any reliable Python binding for SNMP, I decided to write Snimpy with two goals in mind:
  1. Leverage information contained in MIBs to provide a pythonic interface.
  2. Any error condition should raise an exception.
Here is how the previous script could be written:
#!/usr/bin/env snimpy
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
m = M(options.host, options.community, 2)
load("IF-MIB")
for idx in m.ifDescr:
    if m.ifDescr[idx] == "lo":
        continue
    if m.ifOperStatus[idx] != "up":
        continue
    print("     ".format(m.ifDescr[idx],
                            m.ifInOctets[idx],
                            m.ifOutOctets[idx]))
You can also use a list comprehension:
load("IF-MIB")
print("\n".join([ "     ".format(m.ifDescr[idx],
                                    m.ifInOctets[idx],
                                    m.ifOutOctets[idx])
                  for idx in m.ifDescr
                  if m.ifDescr[idx] != "lo"
                  and m.ifOperStatus[idx] == "up" ]))
Here is another simple example to get the routing database from the agent:
load("IP-FORWARD-MIB")
m=M("localhost", "public", 2)
routes = m.ipCidrRouteNextHop
for x in routes:
    net, netmask, tos, src = x
    print(" :>15s / :<15s  via  :<15s  src  :<15s ".format(
        net, netmask, routes[x], src))
IP-FORWARD-MIB::ipCidrRouteNextHop is a more complex table with a compound index. Despite this, querying the table still seems natural. Have a look at Snimpy s documentation for more information. Behind the hood, SNMP requests are handled by PySNMP and MIB parsing is done with libsmi2. Snimpy supports both Python 2, Python 3 and Pypy.

  1. A MIB is defined using a subset of ASN.1 called SMI. However, it is not uncommon to refer to the definition as a MIB too.
  2. Unfortunately, there is currently no robust SMI parser written in pure Python. For example, PySNMP relies on smidump which comes with libsmi. Snimpy uses a custom CFFI wrapper around libsmi.

Gunnar Wolf: Talking in Paran : Free Software philosophy / the Debian project. PS: Want to contribute?

I'm very happy: I was finally able to present a talk at a Free Software conference in Paran , Argentina Regina's hometown. Not only in Paran , but at the Vieja Usina culture center, half a block away from her parents' house. So, I must doubly thank Laura: First, for letting us know there would be a Free Software conference there, and second, for taking some pictures :- What was this conference? Conferencia Regional de Software Libre, organized by Grupo de Usuarios de GNU/Linux de Entre R os (GUGLER). Of course, flying to Argentina (and more specifically, to Paran , which is ~500Km away from the international airport) just for a one day conference was out of the question So I gave the talk by videoconference. Of course, given we will be travelling for the December vacations to Argentina, I expect to meet in person the GUGLER guys soon. I gave a single talk, mixing together two different topics: (my very personal take on) the Free Software philosophy and Debian's place in the Free Software universe. I had a very good time giving the talk, and while I was unable to look at my audience, I got reports saying they were happy and interested. I even got some mails from them, which makes me quite happy ;-) Now, one of the recurring points whenever I talk about Debian: I often tell people that I cannot tell them why they should use Debian instead of other distributions. My years testing every distribution I come across are long gone, and I nowadays am familiar with Debian only. But I also tell them that personally I gain nothing by having more Debian users in the world What I want to achieve is the next logical step: To have more people contributing to Debian. So, here is a great opportunity for interested people, specifically a group that often has a hard time finding a way to collaborate with Free Software projects. Today, Paul Tagliamonte published a call for proposals for Debian 8 (Jessie)'s artwork. So, given many people always want to find a way to contribute to Free Software without being a coder, here's a golden opportunity. You can look at the themes sent for Debian 7 as a reference; look also at the technical requirements for your artwork, and... Well, you have until early February to work on it!

10 November 2013

Gregor Herrmann: RC bugs 2013/45

here's the list of RC bugs I've worked on during the last week.
  • #713582 src:malaga: "malaga: FTBFS: ld: canvas.o: undefined reference to symbol 'ceil@@GLIBC_2.2.5'"
    add patches from Ubuntu / Matthias Klose, upload to DELAYED/2
  • #713595 src:goocanvas: "goocanvas: FTBFS: ld: demo.o: undefined reference to symbol 'sincos@@GLIBC_2.2.5'"
    add patch from Ubuntu / Matthias Klose, upload to DELAYED/2
  • #713635 src:osm-gps-map: "osm-gps-map: FTBFS: osm-gps-map-widget.c:778:21: error: format '%lld' expects argument of type 'long long int', but argument 4 has type 'goffset' [-Werror=format=]"
    add patch from upstream git, package FTBFS again later
  • #713666 src:gnustep-base: "gnustep-base: FTBFS: GSXML.m:983:38: error: dereferencing pointer to incomplete type"
    add patch from Ubuntu / Benjamin Drung, upload to DELAYED/2
  • #713700 src:gsnmp: "gsnmp: FTBFS: gsnmp.h:27:18: fatal error: gnet.h: No such file or directory"
    apply patch from Ubuntu / Daniel T Chen, upload to DELAYED/2
  • #713701 src:libmatchbox: "libmatchbox: FTBFS: ld: dump-image.o: undefined reference to symbol 'XOpenDisplay'"
    add patches from Ubuntu / Daniel T Chen, upload to DELAYED/2
  • #713708 src:gtkgl2: "gtkgl2: FTBFS: ld: zktor.o: undefined reference to symbol 'atan2@@GLIBC_2.2.5'"
    apply patch from Ubuntu / Daniel T Chen, upload to DELAYED/2
  • #713727 src:xmlroff: "xmlroff: FTBFS: ld: ../libfo/.libs/libfo-0.6.a(fo-datatype.o): undefined reference to symbol 'floor@@GLIBC_2.2.5'"
    apply patch from Ubuntu / Daniel T Chen, upload to DELAYED/2
  • #713728 src:matchbox-panel: "matchbox-panel: FTBFS: ld: mb-applet-menu-launcher.o: undefined reference to symbol 'XInternAtom'"
    update patch to link against more libs, upload to DELAYED/2
  • #724080 src:libgraph-perl: "libgraph-perl: FTBFS: make: *** [clean] Error 1"
    fix clean target in debian/rules, upload to DELAYED/2, then moved to 0-day with maintainer's consent
  • #724085 src:myspell: "myspell: FTBFS: POD errors"
    prepare a patch
  • #724115 src:hunspell: "hunspell: FTBFS: POD error"
    prepare a patch
  • #724142 src:xmltv: "xmltv: FTBFS: Tests failed"
    investigate error
  • #724750 electrum: "electrum fails to launch from terminal"
    add missing dependency, upload to DELAYED/2
  • #726801 biomaj: "biomaj: fails to install with Recommends enabled"
    tag unreproducible, later further analyzed as a java dependency problem by follow DD

Jelmer Vernooij: Quantified Self

Dear lazyweb, I've been reading about what the rest of the world seems to be calling "quantified self". In essence, it is tracking of personal data like activity, usually with the goal of data-driven decision making. Or to take a less abstract common example: counting the number of steps you take each day to motivate yourself to take more. I wish it'd been given a less annoying woolly name but this one seems to have stuck. There are a couple of interesting devices available that track sleep, activity and overall health. Probably best known are the FitBit and the jazzed-up armband pedometers like the Jawbone UP and the Nike Fuelband. Unfortunately all existing devices seem to integrate with cloud services somehow, rather than giving the user direct access to their data. Apart from the usual privacy concerns, this means that it is hard to do your own data crunching or create a dashboard that contains data from multiple sources. Has anybody found any devices that don't integrate with the cloud and just provide raw data access?

Luke Faraone: Why I use my bank's mobile site on my desktop

(or, cutting out bloat by using a platform where bloat won't fly)

Let me start off by saying I'm generally a huge fan of my bank, USAA. Their offerings are free of hidden fees, their phone support excellent, and the perks they provide are competitive. They don't have the best savings interest rates, but you can always find a better deal online to park money not actively in your checking account.

However, USAA's website is a behemoth. My account page took about 8 seconds to fully load, downloading 1.4MiB of content.

The "My Accounts" page you're redirected to after logging in.
It is frequently buggy; whenever I log in via Google Chrome on Ubuntu 12.04 I land on a page with a URL beginning with "https://www.usaa.com/inet/gas_bank/AccountBannerAjax" and a bunch of GET parameters like "currentaccountkey" and "accnumber" with values like "encrypted12a1f4dd1[ ]". The server returns a 200 OK, promises a Content-Length of 20, but then actually returns zero bytes. After navigating to the homepage and clicking a button, I end up getting logged in, but I wonder what percentage of their userbase are experiencing this problem?

For some strange reason, I get a lot of checks. It appears that nobody else informed the banking system that it's 2013, and the easiest mechanism for people to send money without paying fees is still on paper. To its credit, USAA made remote deposit of checks available to all customers in 2006, when it was mostly an offering limited to businesses. However, it seems like they haven't updated their web workflow since then.


Using it on the web still requires using a signed Java applet (itself discouraged by CMU's CERT) that does the incredibly complex task of letting you select a file from your computer and upload it to their servers. At least, that's what I think it does, because any time I chose "Run", my browser complained a few minutes later that the tab had stopped responding. Regardless of functionality, you can accomplish almost anything their site could currently be doing with HTML5 and a third party service if they want to crop images locally.

Spinning after logging
in on Android
USAA's mobile app for Android has another host of problems; I haven't been able to log into it for 2 weeks, and when I chatted with someone today I was told they were "doing some maintenance this weekend", so I should try again in a few hours once that's finished.

I googled around a bit for some way to perhaps make the applet work in Ubuntu (which admittedly is not a supported platform), and came upon a Facebook thread where a rep suggested using the mobile web site.

A breath of fresh air
I loaded it in my browser, and was amazed at how well it functioned. Obviously designed for higher-end devices (It didn't even load in one WAP emulator I tried), the mobile web interface was a refreshing breath of fresh air. It scaled well to a full-screen device (see below), loaded quickly, and gave me all the information I would have wanted out of the normal web interface.

Most notably: remember the whole "upload a check" workflow that required a buggy Java applet on the main website? We get bog-standard HTML form fields, no additional magic. There goes any theories about the Java client doing some magic validation or prep of the image; here, all they're getting is the images and my session cookie.


I'm still shocked at whoever thought a My Yahoo!-style homepage was the best layout for a bank, but props to the web developers who managed to make a mobile interface that was both pretty and allowed me to work around broken functionality in their implementations on every other platform I had access to.

But why was the mobile web interface the least bloated? Easy. On the desktop, you generally have a nice pipe, or if not, the user knows it and won't be too upset if your site is just as slow as other sites similarly situated. On mobile, the user downloaded all the code already, so the only latency should be the API requests against the server, right?

On the mobile web users have come to expect relatively speedy mobile-optimised sites and there's less screen real estate to do fancy things that get in the way of content. For many sites, that's a huge improvement. Of course, it would be really nice if more banks supported open protocols for interactions (USAA has a read-only, limited-duration OFX feed), but I would settle for a better web interface.

So tl;dr: USAA, please make www.usaa.com redirect to m.usaa.com, kthxbai.

7 November 2013

Matt Zimmerman: Management: a rant

You keep using that word. I do not think it means what you think it means. -Inigo Montoya
Having worked in full-time management positions for some years now, I am increasingly convinced that management is widely misunderstood as a role, as a discipline and as a field, and that this makes a lot of lives more difficult and stressful than necessary. It is the subject of much speculation and misbelief, and I ve chosen a few of my favorite examples to deconstruct here. Misbelief #1: management is what managers do I ve noticed a trend among a certain class of companies, whose employees will tell anyone who will listen that there is no management in their organization, they never plan to have any, and neither should you. I think these statements are respectively a lie, a na ve belief, and a piece of bad advice. Usually, these companies are just a few years old and relatively small, most of the people in the company have been there for less than a year, and the speaker is trying to persuade us what a unique and innovative company they work for because nobody there is a manager . They invariably have not read The Tyranny of Structurelessness. Management is the practice of enabling people to effectively cooperate. A manager is someone whose job is to do that. It s that simple. It usually involves tasks such as sharing information, agreeing on a course of action, dividing up work, and figuring out what to do when there s a problem. They re things that every team needs to do, whether anyone is designated a manager or not. Teams can function without managers, but they can t function properly without management. Someone (or everyone) has to do the work to make cooperation possible. Modern management is a specialized discipline, which draws on a broad range of skills in communication, psychology, empathy, problem-solving, leadership, and more. These skills aren t unique to managers, but it often makes sense to designate certain people to do more of the management work, on behalf of the team. By devoting more of their time and attention to it, they free other members of the team to focus on other tasks. They can act as a coordinator to help the team stay in sync, and by focusing on this job, they may be able to do a better job of it, and acquire a higher level of skill through practice and study. But it remains an inherently collaborative practice. Misbelief #2: management is about telling people what to do There are many different varieties of management, each of which is oriented toward a particular type of team or organization. Factories are managed differently from design studios, large companies are managed differently from small companies, and every team has its own distinct management style which arises from the unique group of people involved. Some managers are specialists in a particular type of management, while others are more generalists. The telling people what to do style of management is called command and control . It s characterized by authority, hierarchy, and strict adherence to protocol. It s widely employed by military organizations, and by the managers we see in television and film. It has some advantages and disadvantages, which I won t discuss here. My point is that it is just one example, but this example is used to represent the general concept of management. Self-organization, where no one in particular is responsible for group decisions, is another, quite different, style of management. Small, self-organizing teams are capable of amazing feats of productivity. They re less difficult to manage because they re comparatively simple, and so simple tools and techniques work well. Everyone can be fully aware of what everyone else is doing, and new information propagates quickly throughout the team. But as the team or organization grows, it will often outgrow this way of working, and needs to adapt. There is no single management approach which works universally well. Misbelief #3: management is a promotion You know the story. When an employee is successful within their area of expertise, someone will eventually offer them a management role as a reward for their good work. This is utter nonsense. Management is not a promotion: it s a career change. It means starting over as a beginner in a new discipline and learning from the ground up. Domain expertise is important, as a manager needs to understand the work of the other people on their team, but it is no longer paramount. The team, the human system, becomes their focus. When organizations fail to provide career advancement within a discipline, people may turn to management as the only way to get promoted , only to discover that they are completely unprepared for this new field, and often their new job when they realize what they ve gotten into. If someone were promoted from a position as a financial analyst to a new job as a biochemist with no training or expertise, we would probably find this bizarre. But this is analogous to what happens to new managers all the time, and has become almost standard practice in many organizations and industries. So what? Management is misunderstood. So are science, engineering, and many other fields. What does it matter?
People leave managers not companies in the end, turnover is mostly a manager issue, - Marcus Buckingham & Curt Coffman, First, Break All the Rules
This mythology leads to massive organizational dysfunction, making it harder for everyone to do their jobs. It virtually guarantees incompetent management, which is a scourge on anyone who is exposed to it. It ruins days, weeks, jobs and careers. It leads talented people to leave companies, and it drives them out of their chosen professions. I recommend that we stop denigrating and ignoring management, and start doing a better job of it.

5 November 2013

Johannes Schauer: announcing bootstrap.debian.net

The following post is a verbatim copy of my message to the debian-devel list. While botch produces loads of valuable data to help maintainers modifying the right source packages with build profiles and thus make Debian bootstrappable, it has so far failed at producing this data in a format which is:
  • human readable (nobody wants to manually go through 12 MB of JSON data)
  • generated automatically periodically and published somewhere (nobody wants to run botch on his own machine or update periodically update the TODO wiki page)
  • available on a per-source-package-basis (no maintainer wants to know about the 500 source packages he does NOT maintain)
While human readability is probably still lacking (it's hard to write in a manner understandable by everybody about a complicated topic you are very familiar with), the bootstrapping results are now generated automatically (on a daily basis) and published in a per-source-package-basis as well. Thus let me introduce to you: bootstrap.debian.net Paul Wise encouraged me to set this up and also donated the debian.net CNAME to my server. Thanks a lot! The data is generated daily from the midnight packages/sources records of snapshot.debian.org (I hope it's okay to grab the data from there on a daily basis). The resulting data can be viewed in HTML format (with some javascript for to allow table sorting and paging in case you use javascript) per architecture (here for amd64). In addition it also produces HTML pages per source package for all source packages which are involved in a dependency cycle in any architecture (for example src:avahi or src:python2.7). Currently there are 518 source packages involved in a dependency cycle. While this number seems high, remember that estimations by calculating a feedback arc set suggest that only 50-60 of these source packages have to be modified with build profiles to make the whole graph acyclic. For now it is funny to see that the main architectures do not bootstrap (since July, for different reasons) while less popular ones like ia64 and s390x bootstrap right now without problems (at least in theory). According to the logs (also accessible at above link, here for amd64) this is because gcc-4.6 currently fails compiling due to a build-conflict (this has been reported in bug#724865). Once this bug is fixed, all arches should be bootstrappable again. Let me remind you here that the whole analysis is done on the dependency relationships only (not a single source package is actually compiled at any point) and compilation might fail for many other reasons in practice. It has been the idea of Paul Wise to integrate this data into the pts so that maintainers of affected source packages can react to the heuristics suggested by botch. For this purpose, the website also publishes the raw JSON data from which the HTML pages are generated (here for amd64). The bugreport against the bts can be found in bug#728298. I'm sure that a couple of things regarding understandability of the results are not yet sufficiently explained or outright missing. If you see any such instance, please drop me a mail, suggesting what to change in the textual description or presentation of the results. I also created the following two wiki pages to give an overview of the utilized terminology: Feel also free to tell me if anything in these pages is unclear. Direct patches against the python code producing the HTML from the raw JSON data are also always welcome.

4 November 2013

Jakub Wilk: ~/.netrc security

TL;DR: don't put valuable passwords in ~/.netrc In the olden days, the ~/.netrc file was used for storing FTP usernames and passwords. These days we have clients of other protocols that use said file. Perhaps your IMAP or SMTP client use it. So you put your e-mail accounts password into ~/.netrc, and then meticulously configured the clients to always connect via TLS and to verify server certificates. You feel secure. But you shouldn't. Here's how an attacker capable of MiTM can exploit wget to steal ~/.netrc passwords: 1) Alice tries to download a file over HTTP:
$ wget http://xkcd.com/538/
2) Eve takes over the HTTP connection, sending a redirection response:
HTTP/1.1 303 See Other
Location: http://supersecuremail.example.net/
3) Alice's wget follows the redirection. 4) Eve takes over the connection to supersecuremail.example.net, requesting password authentication:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="moo"
5) Alice's wget sends the supersecuremail.example.net password straight to Eve.

30 October 2013

Joey Hess: license monads

This could have been a blog post about toothbrushes for monkeys, which I seem to have dreamed about this morning. But then I was vaguely listening to the FAIFCast in the car, and I came up with something even more esoteric to write about! License monads would allow separating parts of code that are under different licenses, in a rigorous fashion. For example, we might have two functions in different license monads:
foo :: String -> GPL String
bar :: Char -> BSD Char
Perhaps foo needs to use bar to calculate its value. It can only do so if there's a way to lift code in the BSD monad into the GPL monad. Which we can legally write, since the BSD license is upwards-compatable with the GPL:
liftGPL :: BSD a -> GPL a
On the other hand, there should be no way provided to lift the GPL monad into the BSD monad. So bar cannot be written using code from foo, which would violate foo's GPL license. Perhaps the reason I am thinking about this is that the other day I found myself refactoring some code out of the git-annex webapp (which is AGPL licensed) and into the git-annex assistant (which is GPL licensed). Which meant I had to relicense that code. Luckily that was easy to do legally speaking, since I am the only author of the git-annex webapp so far, and own the whole license of it. (Actually, it's only around 3 thousand lines of code, and another thousand of html.) It also turned out to be easy to do the refactoring, technically speaking because looking at the code, I realized I had accidentially written it in the wrong monad; all the functions were in the webapp's Handler monad, but all of them used liftAnnex to actually do their work in the Annex monad. If that had not been the case, I would not have been able to refactor the code, at least not without entirely rewriting it. It's as if I had accidentially written:
foo :: String -> GPL String
foo = mapM (liftGPL . bar)
Which can be generalized to:
foo :: String -> BSD String
foo = mapM bar
I don't think that license monads can be realistically used in the current world, because lawyers and math often don't mix well. Lawyers have, after all, in the past written laws re-defining pi to be 3. Still, license monads are an interesting way to think about things for myself. They capture how my code and licenses are structured and allow me to reason about it on a more granular level than the licences of individual files. (I have a vague feeling someone else may have written about this idea before? Perhaps I should have been blogging about monkey toothbrushes after all...)

Next.

Previous.