I ve spent the last few hours working on a tool to take snapshots of all the html files in a .deb using phantomjs. It works great, but renders text all funny on my machine. I ve got two builds going with slight tweaks to check for the issue, but some advice before I file a bug would be helpful. (or see the full size image) This will hopefully be integrated into debuild.me, or debexpo.

Some of my Romanian readers might know that for the last two years I've got involved in the skeptical movement to such a degree that I am a co-producer of a bi-weekly podcast on science and skepticism (in Romanian) called Skeptics in Romania . Some might even be regular listeners of the show.

(There isn't much to see now visually on the site, but me and the other people behind the project have some ongoing plans to change that.)

In spite of our modest site, up until now we had some successes, one of them being the publication of an article on us in a known Romanian printed publication and another being the invitation to a live show face to face with Oreste Teodorescu, a well known Romanian mysticist and woo promoter.

During that live show we managed to show a demonstration (video below, in Romanian) of how astrology gives the impression of working, without actually working, and, taking into account we had no prior TV camera experience and that it was a live show, I think we managed an honourable presence.


We also have a series of interviews in English with some really interesting people: Dr. Eugenie Scott, Prof. Christopher French, Prof. Edzard Ernst, Samantha Stein and others. We did these interviews at Denkfest 2011, in Zurich, and we integrated the translated (voice over) interviews in our podcast. The conlusion is that most of our activities revolve around the podcast, so let me tell you more about that.

The podcast has a somewhat fixed structure, it starts with a conversation between ourselves, then we have a segment on the history of science, technology, skepticism and woo, and then we have a segment called The dangers of not being skeptical . In this segment we present cases of people who lost their lives, their health, their money or any combination of the former because they were duped into some scam, science-y sounding non-science, unfounded claim or some other woo.

Having lost recently my brother-in-law to a form of cancer known as Hodgkin lymphoma, I became especially sensitive about miracle-cure claims for cancer, and this section of the show has lately seen its fair share of such cases. Honestly, if there could be a way to prosecute the irresponsible, ignorant and/or cynical people promoting all sorts of quack "therapies", especially for cancer*, I would really like to see it happen. But there isn't, and we're trying the best accessible approach, informing the public.

During my brother-in-law's last two years of his life, he went through lots of chemotherapy and radiotherapy sessions, repeated periods of hospitalisation, and lots of drugs. This is the best of what we currently have for treating and curing most forms of cancer, and too many times this isn't enough. I can't even imagine how stressful and discouraging it must feel when the best of what we have doesn't help.

Here is where the desperation and hopes of patients and their families meet the purely irresponsible cynical or ignorant promoters of woo and quack therapies. Because it takes either an ignorant or a really cynical (I really feel this word isn't enough) person to prey on the suffering of other people to make easy money under the false pretence of offering a cure.

It almost happened to my brother-in-law and his family, because they almost went for some herbal concoction promoted as a cancer cure on some forum, blog or page of a seller of this fake therapy. It was really hard for me to make them understand why using such a product it not advisable, not even in parallel with the medical treatment due its possible counter effect or interactions with the real medical treatment, without them getting the wrong idea that I wasn't trying to help. While trying to be brief and informative not to lose their attention, I told them how "natural" doesn't necessarily mean "good" (uranium, lead and Irukandji's venom are all natural), and how plants are drugs because they all contain chemical substances (and no, "chemical" does not mean "human made" or "artificial") which could interact with the medical treatment.


But most people don't even have the chance of having close by a person with a more science-leaning thought process and a skeptical mind. Those unfortunate people are the most vulnerable people and constitute the biggest chunk of the victims of baseless pseudo-cures or pseudo-treatments.

On our last show, I presented the case of Yvonne Main, a cancer suffering patient who mistook an invasive carcinoma for a cyst, and irdologist Ruth Nelson for a real healthcare giver.

Yvonne Main, died from an invasive carcinoma after seeking help from a iridologist, and delaying real medical treament for 18 months

Yvonne, after seeking medical advice from a person that essentially promotes the dead idea of guessing diseases by looking at the eyes**, used natural treatments for about 18 months and, after all this time, her carcinoma grew to a size of 10 to 11 cm, eating through her skull and causing damage which was later attempted to be countered through bone transplant from her ribs.

Ruth Nelson wasn't prosecuted in any way and continues her practice of quackery unharmed.

This is not the only case, nor even one case from a select few where woo and quackery lead to grave consequences for patients. There are many, many more; they're so many that even after splitting them in categories they seem too many per category, especially when you realise these are only the findings of, essentially, a single man:




This is part of what I have been doing in the last few years, instead of working on Debian. Is it a good thing? Is it a bad thing? Maybe it's good. I want to know what do you think?


* you will, most likely, never hear such a promoter of non-therapies say that there isn't just one cancer, and that, in fact, cancer is a name for a certain family of diseases which are all called cancer - that's a first sign that you might be dealing with quack
** probably in the line of thought that the eyes are the gates to the soul so they must tell something significant about health

I have been toying around with Erlang since quiet some time and finally made something that could be considered the de facto Hello World of Erlang. Unlike most of the other programming languages where concurrency is an added feature, Erlang was built around the concept of concurrent programming. If you love functional programming and can get over the weird Erlang syntax, learning Erlang will not be very difficult and is definitely worth the effort. I have been reading Joe Armstrong s Programming Erlang: Software for a Concurrent World and will definitely recommend it to anyone starting out with Erlang. I am yet to dive into OTP, but so far, so good. :) I have taken the naive recursive factorial algorithm and parallelized it to take advantage of multiple cores available on average system these days. Lets assume that we need to distribute the task of calculating N factorial over K processes. To do that, I have divided the set of integer 1 N to K subsets, say S , S , ., S_k, such that S S S_k = 1 N and S S S_k = Where, S = [ 1, K+1, 2K+1, , (x )*K+1 ];
x : (x )*K + 1 <= N and (x +1)*K + 1 > N and x S = [ 2, K+2, 2K+2, , (x )*K+2 ];
x : (x )*K + 2 <= N and (x +1)*K + 1 > N and x

S_k = [ K, K+K, 2K+K, , (x_k)*K+2 ];
x_k : (x_k)*K + 2 <= N and (x_k+1)*K + 1 > N and x_k Product of all the elements in each of the sets in calculated in a separate thread and all these calculations, theoretically, can be done in parallel. These sub-products are then multiplied together to get the final factorial. *Update*: The algorithm described above works but is not the best way to find the factorial of a number and after some time on Google, I am convinced that Luschny s is the resource on the internet if you are looking for something faster and has a factorial somewhere in it. I am an Erlang beginner, and this could be very well a sub-optimal implementation. Comment and let me know, what you think of it and if you enjoyed reading this, you should follow me on twitter.

Start the Erlang shell
$ erl
Compile the code
1> c(factorial).
Call the calculate function
2> factorial:calculate(10000).

<script src="https://gist.github.com/3715155.js"> </script>

My first morning in India I woke up very late in the house of Pavithrans parents. Pavithran was out organizing events for us, his father was at work, leaving me with his mother. She spoke little english - unsure if it was lack of skills or she was humble, or perhaps she just felt as shy as me in the situation. How to, ahem, do your thing at a restroom when there is no toilet paper but instead a bucket of water? I decided to not yell for help but use my imagination. How to eat breakfast local style? I knew from dinner previous night with Pavithran that local custom was to eat with my hands and that it (lucky for me) was tolerated in India to use left hand, so turned down the kind offer for a spoon. But were I supposed to mix rice with all the curries? Or one at a time? In which order? What if a curry was too hot to dip my fingers into? Was the liquid stuff to drink or to mix with the rice or eat afterwards? I asked for help, and first she just smiled at my alien useless language and then when I persisted, she patiently demonstrated with her hands in my food how to do. The only natural thing to do, really, and I dearly appreciate her help and patiency. But wauw, it was mind-blowing to me! Since early childhood, fingers in food is a forbidden thing: "Don't play with the food!". On top of that, having someone else handle my dish while at the table is something I associate with being very old and needing to be spoon-fed. In the afternoon we went for a small hike to an old stone fortress in the middle of Khammam - with a great view of the sun setting. Next day we went to Sarada Institute of Technology & Science where I gave my first talk to about 100 students. I had intended to provide concrete facts on Debian generally and on my pet project, Debian Pure Blends, but then the night before decided to radically shift focus to their situation in the early twenties - as best as I could imagine it. Pavithran had clearly expected a different style talk, but I liked it and believe it was received well by the audience as well. The teacher responsible for the event, Bhukya Jabber, afterwards asked for hints on running Debian at their computer lab. I suggested to not lock down access but instead make it easy to reinstall, and he explained how he was quite interested in a larger degree of learning-by-doing (which I had also promoted in my talk) but was constrained by curriculum dictated higher up in the educational system. Late that day Pavithrans father introduced me to CPM Khammam - the local offices and community center of the Communist Party - and to a colleague of his at the place, N. N. Rao. I instantly fell in love with the place and its atmosphere, and now have an open invitation to come back to spend 1-2 months to study and to collaborate with other users of the place (including some kids hosted there) on Free Software. Next morning we headed back by train to Hyderabad I am still amazed how radical it feels sticking my fingers into food. Not the physical feeling (I am not that disconnected from my body) but similar to a discovery I had as a teenager: After 7 years of piano lessons (and numerous other instruments less patiently) I gave up because I felt it was too difficult expressing personality through the instrument. After that I exclusively sang - as I had always done, but only now did I recognize my voice as a serious musical instrument. Similarly I now realized that eating with the fingers is not just yet another eating style like fork+knife or sticks - it is the natural one. Obviously, in hinsight. This text is part of my Asia 2011 scriblings.

A while back mdz blogged about challenges facing Ubuntu and other Linux distributions. He raises the point that runtime libraries for Python / Ruby etc have a unique set of issues because they tend to have their own packaging systems. Merely a month later he attended Debconf 2010 where a presentation was given on the issues that Java packages have on Dpkg based systems. Since then the conversation seems to have dried up. I ve been reminded of it recently in discussions within Canonical looking at how we deploy web services. Matt suggested some ways forward, including:

• Decouple applications from the core
• Treat data as a service (rather than packages) get data live from the web rather than going web -> distro-package -> user machines.
• Simplify integration between packaging systems (including non-packaged things)

I think its time we revisit and expand on those points. Nothing much has changed in how Ubuntu or other distributions approach integration with other packaging systems but the world has kept evolving. Internet access is growing ever more ubiquitous, more platforms are building packaging systems clojure, scala, node.js, to name but three, and a substantial and ever growing number of products expect to operate in a hybrid fashion with an evolving web service plus a local client which is kept up to date via package updates. Twitter, Facebook and Google Plus are three such products. Android has demonstrated a large scale app store on top of Linux, with its own custom packaging format. In order to expand them, we need some background context on the use cases that these different packaging systems need to support. Platforms such as antivirus scanners, node.js, Python, Clojure and so forth care a great deal about getting their software out to their users. They care about making it extremely easy to get the latest and greatest versions of their libraries. I say this because the evidence is all around us: every successful development community / product has built a targeted package management system which layers on top of Windows, and Mac OSX, and *nux. The only rational explanation I can come up for this behaviour is that the lower level operating system package management tools don t deliver what they need. E.g. this isn t as shallow as wanting a packaging system written in their own language, which would be easy to write off as parochialism rather than a thoughtful solution to their problems. In general packaging systems provide a language for shipping source or binary form, from one or more repositories, to users machines. They may support replications, and they may support multiple operating systems. They generally end up as graph traversal engines, pulling in dependencies of various sorts you can see the DOAP specification for an attempt at generic modelling of this. One problem that turns up rapidly when dealing with Linux distribution package managers is that the versions upstream packages have, and the versions a package has in e.g. Debian, differ. They differ because at some stage, someone will need to do a new package for the distribution when no upstream change has been made. This might be to apply a local patch, or it might be to correct a defect caused by a broken build server. Whatever the cause, there is a many to one relationship between the package versions that end users see via dpkg / rpm etc, and those that upstream ship. It is a near certainty that once this happens to a library package, that comparing package versions across different distribution packages becomes hard. You cannot reliably infer whether a given package version is sufficient as a dependency or not, when comparing binary packages between Red Hat and Debian. Or Debian and Ubuntu. The result of this is that even when the software (e.g. rpm) is available on multiple distributions (say Ubuntu and RHEL), or even on multiple operating systems (say Ubuntu and Windows), that many packages will /have/ to be targeted specifically to build and execute properly. (Obviously, compilation has to proceed separately for different architectures, its more the depedency metadata that says and build with version X of dependency Y that has to be customised). The result of this is that there is to the best of my knowledge no distribution of binary packages that targets Debian/Ubuntu and RHEL and Suse and Windows and Mac OS X, although there are vibrant communities building distributions of and for each in isolation. Some of the ports systems come close, but they are still focused on delivering to a small number of platforms. There s nothing that gives 99% coverage of users. And that means that to reach all their users, they have to write or adopt a new system. For any platform X, there is a strong pressure to have the platform be maintainable by folk that primarily work with X itself, or with the language that X is written in. Consider Python there is strong pressure to use C, or Python, and nothing else, for any tools that is somewhat parochial, but also just good engineering reducing variables and making the system more likely to be well maintained. The closest system I know of Steam is just now porting to Ubuntu (and perhaps Linux in general), and has reached its massive popularity by focusing entirely on applications for Windows, with Mac OSX a recent addition. Systems like pypi which have multi platform eggs do target the wide range of platforms I listed above, but they do so both narrowly and haphazardly: whether a binary or source package is available for a given platform is up to the maintainer of the package, and the packages themselves are dealing with a very narrow subset of the platforms complexity: Python provides compilation logic, they don t create generic C libraries with stable ABI s for use by other programs, they don t have turing complete scripts for dealing with configuration file management and so forth. Anti virus updaters similarly narrow the problem they deal with, and add constraints on latency- updates of anti virus signatures are time sensitive when a new rapidly spreading threat is detected. A minor point, but it adds to the friction of considering a single packaging tool for all needs is the different use cases of low level package management tools like dpkg or rpm vs the use cases that e.g. pypi has. A primary use case for packages on pypi is for them to be used by people that are not machine administrators. They don t have root, and don t want it. Contrast that with dpkg or rpm where the primary use case (to date) is the installation of system wide libraries and tools. Things like man page installation don t make any sense for non-system-wide package systems, whereas they are a primary feature for e.g. dpkg. In short, the per-platform/language tools are (generally):

1. Written in languages that are familiar to the consumers of the tools.
2. Targeted at use on top of existing platforms, by non-privileged users, and where temporary breakage is fine.
3. Intended to get the software packaged in them onto widely disparate operating systems.
4. Very narrow they make huge assumptions about how things can fit together, which their specific language/toolchain permits, and don t generalise beyond that.
5. Don t provide for security updates in any specific form: that is left up to folk that ship individual things within the manager.

operating system package managers:

1. Are written in languages which are very easy to bootstrap onto an architecture, and to deploy onto bare metal (as part of installation).
2. Designed for delivering system components, and to avoid be able to upgrade the toolchain itself safely.
3. Originally built to install onto one operating system, ports to other operating systems are usually fragile and only adopted in niche.
4. Are hugely broad they install data, scripts, binaries, and need to know about late binding, system caches etc for every binary and runtime format the operating system supports
5. Make special provision to allow security updates to be installed in a low latency fashion, without requiring anything consuming the package that is updated to change [but usually force-uninstalling anything that is super-tightly coupled to a library version].

Anti virus package managers:

1. Exist to update daemons that run with system wide escalated privileges, or even file system layer drivers.
2. Update datasets in realtime.
3. Without permitting updates that are produced by third parties.

Given that, lets look at the routes Matt suggested Decoupling applications from the core as a strategy makes an assumption that the core and applications are partitionable. If they are not, then applications and the core will share common elements that need to be updated together. Consider, for instance, a Python application. If you run with a system installed Python, and it is built without zlib for some reason, but the Python application requires zlib, you have a problem. A classic example of this problem is facing Ubuntu today, with all the system provided tools moving to Python 3, but vast swathes of Python applications still being unported to Python 3 at all. Currently, the Python packaging system virtualenv/buildout + distribute don t provide a way to install the Python runtime itself, but will happily install their own components for everything up the stack from the runtime. Ubuntu makes extensive use of Python for its own tools, so the system Python has a lot of packages installed which buildout etc cannot ignore this often leads to issues with e.g. buildout, when the bootstrap environment has (say) zope.interfaces, but its then not accessible from the built-out environment that disables the standard sys.path (to achieve more robust separation). If we want to pursue decoupling, whether we build a new package manager or use e.g. virtualenv (or gem or npm or ), we ll need to be aware of this issue and perhaps offer, for an extended time, a dedicated no-frills, no-distro-packages install, to avoid it, and to allow an extended supported period for application authors without committing to a massive, distro sponsored porting effort. While its tempting to say we should install pip/npm/lein/maven and other external package systems, this is actually risky: they often evolve sufficiently fast that Ubuntu will be delivering an old, incompatible version of the tool to users well before Ubuntu goes out of support, or even befor the next release of Ubuntu. Treating data as a service. All the cases I ve seen so far of applications grabbing datasets from the web have depended on web infrastructure for validating the dataset. E.g. SSL certificates, or SSL + content checksums. Basically, small self-rolled distribution systems. I m likely ignorant of details here, and I depend on you, dear reader, to edumacate me. There is potential value in having data repackaged, when our packaging system has behind-firewall support, and the adhoc system that (for instance) a virus scanner system has does not. In this case, I specifically mean the problem of updated a machine which has no internet access, not even via a proxy. The challenge I see it is again the cross platform issue: The vendor will be supporting Ubuntu + Debian + RHEL + Suse, and from their perspective its probably cheaper to roll their own solution than to directly support dpkg + rpm + whatever Apple offer + Windows the skills to roll an adhoc distribution tool are more common than the skills to integrate closely with dpkg or rpm What about creating a set of interfaces for talking to dpkg / rpm / the system packagers on Windows and Mac OSX ? Here I think there is some promise, but it needs as Matt said careful thought. PackageKit isn t sufficient, at least today. There are, I think, two specific cases to cater to:

1. The anti-virus / fresh data set case.
2. The egg/gem/npm/ specific case.

For the egg/gem/npm case, we would need to support a pretty large set of common functionality, on Windows/Mac OSX / *nux (because otherwise upstream won t adopt what we create: losing 90% of their users (windows) or 5% (mac) isn t going to be well accepted . We d need to support multiple installations (because of mutually incompatible dependencies between applications), and we d need to support multiple language bindings in some fashion some approachable fashion where the upstream will feel capable of fixing and tweaking what we offer. We re going to need to support offline updates, replication, local builds, local repositories, and various signing strategies to match the various tradeoffs made by the upstream tools. For the anti-virus / fresh data case, we d need to support a similar set of operating systems, though I strongly suspect that there would be more tolerance for limited support in that most things in that space either have very platform specific code, or they are just a large-scale form of the egg/gem/npm problem, which also wants easy updates. What next? We should validate this discussion with at least two or three upstreams. Find out whats missing I suspect a lot and whats wrong I hope not much . Then we ll be in a position to decide if there is a tractable, widespread solution *possible*. Separately, we should stop fighting with upstreams that have their own packaging systems. They are satisfying different use cases than our core distro packaging systems are designed to solve. We should stop mindlessly repackaging things from e.g. eggs to debs, unless we need that specific thing as part of the transitive runtime or buildtime dependencies for the distribution itself. In particular, if us folk that build system packaging tools adopt and use the upstream application packaging tools, we can learn in a deep way the (real) advantages they have, and become more able to reason about how to unify the various engineering efforts going into them and perhaps even eventually satisfy them using dpkg/rpm on our machines.

Do you remember at school, when the teachers mentioned that something would be on your "permanent record"? The big secret is that there isn't one. Apologies to all teachers, but it's not likely this blog's demographic will reach your students. However, since the advent of the internet, a permanent record does exist. Everywhere from the Wayback Machine to Google's cache contains copies of all public activity you post online. At Collabora, we hire some of the best open source software engineers to work for our clients. It should be no surprise that a quick Ohloh and search for someone's name is a standard part of the hiring process. This normally comes up with all sorts of exciting things which people have done, both with software and further abroad. A tricky situation will occur if you have lost your temper online, as Patrick Ryan has done recently. He made a comment on #node.js on the Freenode network that:

[20.16.34] < TheEmpath> anyone else have a hot programmer girl in their office? o____O

This was quite rightly called out as being inappropriate for the channel, and quickly decided to have a large and abusive rant about this. This was then continued with him opening a number of bugs against the projects owned by the person who called him out, with titles such as "Hyperactive Leftist Power Trip not included" and "Not anti-Republican enough". This then gets worse. Once the log is posted (link above) there is yet more abuse and unfounded accusations by Mr Ryan in the comments. So, what happens when you search for his name? You get a log with someone being entirely inappropriate and then large rants. Not the best move for a software consultant.

In this choc-filled post, I ve got a few things I d love to share. debian.js First, a quick side-hack called debian.js. It s not entirely a serious project, but it s interesting. Here s some example code (and output from the terminal) Here s some of the client code (but slightly modified for clarity)

debian.on('bug', function(bug)  
    console.log(bug);
 );

which produces the following:

  sender: 'John Johansen <john.johansen@canonical.com>',
  package: 'vidalia',
  id: '<503577E5.2080502@canonical.com>',
  bugno: '682431',
  action: 'followup',
  class: 'bug',
  description: 'Bug#682431: [apparmor] Factorize Qt and GTK themes abstractions?'  

(and later, after I filed for unblock of desktop-base (more on that later))

  sender: 'Paul Tagliamonte <paultag@debian.org>',
  package: 'release.debian.org',
  id: '<20120823011639.GA8561@leliel.localdomain>',
  bugno: '685657',
  action: 'report',
  class: 'bug',
  description: 'Bug#685657: unblock: desktop-base/7.0.3'  

Legit. Anyway, hopefully more hax on this later. I d love to see Debian and Node.js get along better. dpu Nextly, I ve been working on something called dpu , with the (epic) Niels Thykier which will be a replacement for Lintian s test-runner. Progress is good, but not feature-complete yet. The end goal is to allow all projects which deal with the process of generating a debian package to start testing their code. If you have requirements, let me know! desktop-base I ve uploaded version 7.0.3, which features a switch to xz, and slight changes to existing theme elements. Mad props to Eshat Cakar for some slick KDE updates, and to Jo Carter for the Plymouth updates. Props!

In the process of installing and configuring wanna-build, I had lots of problems. Because there was almost no documentation on the subject (and some of the existing was outdated), I had to figure out how everything works by reading the source code and asking people on IRC and mailing lists. I use the most recent version (from git) and since it s not stable, there were some problems and bugs. I fixed several bugs in order to set the system up. I also documented every step so now there s documentation on everything I have done. I uploaded the documentation on installing and configuring wanna-build [0]. I installed and configured reprepro on a local machine in order to use it with wanna-build. In the process I made step-by step setup instructions [1]. I installed and configured sbuild. The lastest version of sbuild contains filters but they don t work. Their purpose is to keep only the necessary variables in the environment, but apparenly the configuration file which should set them is never read. That leaves the filter list empty which causes the environment to be empty. I had to disable them to keep my work going. I installed buildd. I had some problems setting it up there was a bug in the scripts related to parsing the configuration file. I was pointed to a helpful bug report on the topic and with some additional changes I eventually got it working. In the process I had to read some of the source code of buildd. I made a patch that fixes a bug in the manual page of reprepro and got it approved. I contacted the Alioth admin team via the web site, requesting to use their server for my project Clang for build services . I got approved and I created a git repository where I m now uploading my work. The repository that I use to upload changes to sbuild and buildd is [2]. I add the name of the used compiler to the control file and to the name of the deb package. I wrote a function for parsing the debian/rules file looking for hard-coded compiler there. If there is such compiler, which is different from the one chosen as a default, the building proccess exits with a clear error. I made changes to wanna-build, dpkg, sbuild and buildd to fix bugs in them and make it possible to use a compiler different from gcc for the build process.
After my work a build compiler can be set easily. If a package can not be built with that compiler, the build process fails with a clear error. I also modified the pgstatus web interface to display build logs and create reports on user request. I wrote complete documentation about setting up wanna-build, reprepro and buildd ([0] and [1]). This makes a fully functional buildd infrastructure. Now we have a full step by step tutorial. Me and Sylvestre Ledru who was my great mentor, modified the buildd daemon so it will be possible to be run several instances of it for different architectures on a single physical server. I made a lot of changes to pgstatus this is a web interface to the wanna-build database. I had to add some features and fix several problems. In the process I read some of the code of pgstatus and buildd-tools. Now I have a better understanding of how they work. I added a new feature in pgstatus that allows the user to create a full report of the state of all builds. When a designated reports page is opened, a table with the current state of the builds is shown, together with a list of all the reports saved to files for the current distribution and architecture. The reports can be downloaded directly from the
interface. They are stored in JSON format. The log files were not accessible from pgstatus because of lack of implementation and bugs. I created a script that reads emails sent by wanna-build and extracts log files from them. I made changes to wanna-build so that it feeds the database with the necessary information about package history. I updated the wanna-build patch to include these changes. As a result the buildd logs can now be opened from the web interface of pgstatus. I applied a patch that fixes bug [3] so that config_schroot.sh (this script installs and sets the default compiler) starts with the necessary privileges. I added a new feature to sbuild that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building. Now anybody can configure sbuild to run such a script by modifying the appropriate configuration file. By adding the following lines: $external_commands =
chroot-pre-build-commands => [
['/root/purge-gcc.sh'],
],
; The script /root/purge-gcc.sh is run by sbuild in the chroot. This script can be necessary to set everything in the environment so that packages are built exactly as needed. Me and my mentor used this to replace gcc with a script that removes all gcc-specific packages so that we can be sure whether a package uses clang or not. I have a fully functional buildd infrastructure installed on my development system. In order to get it running, I made several patches: dpkg.patch The main patch with a lot of changes related to compiler support for building.
wanna_build.patch Fixes bugs and makes wanna-build feed the database with package history information.
sbuild.patch Fixes bugs and work-arounds the problem that some sbuild scripts are called without reading the configuration file of sbuild. Adds configuration option that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building.
pgstatus.patch Fixes bugs and adds reports page to pgstatus. Me and my mentor contacted the Debian wanna-build team to connect our new service with Debian. We explained what the current state of the project is and what we need from them. I thank my mentor Sylvestre Ledru for his presentation on DebConf 12 ([4], [5]). He showed our project and it was seen that people are interested in it. A nice resume of his talk has been made by Michael Larabel on Phoronix [6]. I tested everything thoroughly to make sure my changes don t break anything. I am happy to say that the project is now finished. ==== [0] http://wiki.debian.org/DebianWannaBuildInfrastructureOnOneServer
[1] http://wiki.debian.org/SetupBuildServiceForWanna-build
[2] http://anonscm.debian.org/gitweb/?p=pkg-llvm/clang-build.git;a=summary
[3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840
[4] http://sylvestre.ledru.info/blog/media/5/rebuild-debian-presentation-clang-July-10.pdf
[5] http://meetings-archive.debian.net/pub/debian-meetings/2012/debconf12/high/884_Build_Debian_with_another_compiler.ogv
[6] http://www.phoronix.com/scan.php?page=news_item&px=MTEzODQ

github has become the place where a lot of code is made available. It turns out that not everyone has the time or inclination to answer queries / questions or apply pull requests. So, whenever I am investigating using something I've found, I always look to see what the forks are; if there are common issues, etc. In the past, I've done this by hand, for small repositories it isn't too time consuming. Today, though, I sat down and wrote find-forks.py which will do this automatically. Now, I find a project, clone it, run ./find-forks.py and settle down with gitk to examine the history and various other forks. <script src="http://gist.github.com/3405534.js"></script> This works well for me, feel free to use or improve as you like. I hereby place this in the Public Domain.

UK Readers Can Now Purchase DRM-Free Books From Tor UK

We believe that making our Tor ebooks DRM-free is the best for our readers, allowing you to use legitimately-purchased ebooks in perfectly legal ways, like moving your library from one ereader to another, says Jeremy Trevathan, Publisher at Pan Macmillan. We understand that DRM can make your ebooks less easy to read. It also makes building and maintaining your digital library more complicated. For these reasons, we are committed to remaining DRM-free.

I'm really happy to see this and I hope more ebook sellers follow suit. Some of the others already doing DRM-free can be found on our list at Defective by Design. Also see StoryBundle, which just launched and offers a set of DRM-free ebooks on a pay-what-you-want basis, with some of the money going to charity.

The recent update by S ren H jsgaard's to his gRbase package for graphical models made it the 75th package to depend on our Rcpp package for R and C++ integration. So in a lighthearted weekend moment, I tweeted about gRbase being number 75 for Rcpp to which Hadley replied, asking if Rcpp was in fact number one among R package Depends. Far from it, and I immediately replied listing lattice and Matrix as packages with way more other packages depending upon them. But as the question seemed deserving of a bit more analysis, I spent a few minutes on this and prepared three charts listing package in order of reverse Depends, reverse Imports and reverse LinkingTo. First off, the reverse Depends:. This is the standard means of declaring a dependence of one package upon another. Unsurprisingly, the MASS package from the classic Venables and Ripley book comes first, with Deepayan Sarkar's powerful lattice package (also covered in a book) coming second. These are both recommended packages which are commonly distributed with R itself. Next are mvtnorm and survival. Our Rcpp is up there in the top-ten, but not a frontrunner. With the advent of namespaces a few R releases ago, it became possible to import functions from other packages. So the Imports: statement now provides an alternative to the (older) Depends:. The next chart displays the same relationship for Imports:: Now lattice still leads, but Hadleys's plyr package grabbed the second spot just before MASS and Matrix. It is interesting to see that the sheer number of Imports: is still not where the Depends: are. On the other hand, we see a number of more recent packages popping up in the second chart. This may reflect more recent coding practices. It will be interesting to see how this stacks up over time when we revisit this chart. Lastly, we can also look at LinkingTo:, a declaration used to provide a C/C++-level dependency at the source code level. We use this in the Rcpp family to provide automatic resolution of the header files needed to compile against our packages. And unsurprisingly, because packages using Rcpp actually use its API (rather than R functions), the package is a little ahead of others. In the package we find three more packages of the Rcpp family, but only a limited number of other packages as C/C++-level dependencies are still somewhat rare in the R universe. There are also fewer packages overall making use of this mechanism. One could of course take this one level further and sum up dependencies in a recursive manner, or visualize the relationship differently. But these dotchart graphs provide a first visual description of the magnitude of Depends, Imports and LinkingTo among CRAN packages for R.

Monthly DPL bits, fresh from the oven.

---

Tip to feel good about the release #476: before reading this, grab and fix one of the RC bugs affecting Wheezy. Done? Now you're ready for a slightly less exciting report of DPL activities. Highlights

• I've wrapped up a couple of years of discussions with FSF representatives, proposing a joint working group on FSF' assessment of Debian Free-ness. Some news coverage ensued. The discussion list that has been set up welcomes participants and, in particular, could use some more DDs, fond of Debian principles but also interested in external review of our free/non-free separation.
• DebConf12 has happened!, and many of us attended it. Due to family reasons I've been able to attend only DebCamp and DebConf day-0, just in time to deliver the traditional "bits from the DPL" speech. Among other topics, in the speech I've discussed governance scaling issues in our governance, and in particular about the role of the DPL --- issue which I'm partly trying to address with the "DPL helpers" initiatives. Slides and videos of the talk are available. As many others DebConf people, I took part in Nicaraguan TV shows about DebConf, showing a good deal of interest by the local media in our presence. I don't have links to the recordings yet, but I suspect DebConf orga people will provide them as soon as they're available.
• Many others DebConf12 events are relevant for future Project directions. I went through many of the events I've missed thanks to the work of the video team and of the gobby note takers. I recommend doing the same. Some events are particularly relevant to recent project-wide discussions, such as EFI, hijacking, and the release process and I'm sure you'll find more If you're a DebConf12 speaker and produced slides, please upload them to penta, so that they could be used as future reference.

Assets

• On the hardware front, we bought another server according to DSA's "5-year plan". It will be used as a ganeti node at the man-da hosting location. We bought it thanks to donated money, for a total of ~6'600 EUR.
• (After ~8 months of pestering,) we got from SPI a data dump of Debian financial transactions there. Thanks for this result goes to the SPI Board and in particular to Robert Brockway and Joerg Jaspert. This was the last blocker to produce detailed financial reports for the past couple of years of Debian activities. Unfortunately, it still is a blocker: Debian auditors found out that the data dump is not complete. SPI will send us remaining data soon , but I don't have an ETA, unfortunately.
• An ftpmasters sprint is being organized and I've been happy to approve its budget. Help is still needed on the -sprints list to keep up with the organization part of sprints, including activities like documenting sprints to show donors what we do with their help.

Logo & trademark

• to stop the absurdity of not using the official logo ("with Debian") as part of our official theme, I advanced a bit on the topic of relicensing the logo under a DFSG-free license. I sought a second legal advice to SFLC (as requested by the SPI board). They confirm we can safely do the relicensing (without undermining our rights on contained marks) under a license like dual LGPLv3+ / CC-BY-SA, provided that a suitable trademark policy is in place
• which is why I've posted to -project a trademark policy draft, based on legal advice from SFLC. I've gather useful feedback already, but more is welcome. Note that, as Russ very eloquently put it, we do not particularly enjoy thinking about which restrictions we put on the usage of our marks. But given we do have (and use) those marks, we should better empower the legal owners (SPI in this case) to properly defend them. Our specification for the policy to SFLC has been "as free as possible", and that is what we're trying to achieve.

Note that the actual logo relicensing should be done by SPI, via their board, upon request of mine (as Debian Project liaison). We won't make it for the next SPI board meeting, as it is on 4 days away. But we can aim for the subsequent meeting, on September 13th. If all goes well (big "if"), we will enjoy a DFSG-free logo after that date. Internal organization Recent flurry of re-organization in the tech-ctte --- which I somewhat triggered pushing for periodic meetings --- seems to be proceeding well. I'm very happy about it, as we all need to trust that tech-ctte decisions will be not only sound, but also prompt. If you're interested into this topic, some recent evidence of the ongoing reorganization and its result can be found in their DebConf12 BoF, minutes of the last meeting, a set of forthcoming GRs, and the recent great decision of posting decisions results to d-d-a (as it happened for node/nodejs). Kudos to tech-ctte members for the recent activism! I haven't worked on it myself directly, but I highlight Enrico's work on server side archival of NM conversations. It has the potential of enabling automatic detection of stuck NM processes. I'm a bit rusty as AM now, but I think missing that ability is one of the main remaining causes of frustration when joining Debian. So, if you are an AM, please opt-in and use this feature with your appicants. If you are not an AM why not? Miscellanea Some misc legal stuff:

• I got the legal advice we asked about the possibility of turning mentors.debian.net into an official service. The worry there was about the potential copyright infringements by mentees. In the end, we don't need to worry about that, provided we've clear contact points (and "reasonable" timing ) for removing infringing material once it's been reported, and that we declare our public policy about that. I've a draft policy document to start with. I'll soon contact mentors.d.n to adapt it where needed and have it vetted by SPI before going live.
• I've worked with GNOME maintainers (thanks in particular to Jordi Mallach) to rework the mixed Debian/GNOME logo on their alioth project page, to stay on the safe side and avoid any ambiguity wrt GNOME trademark policy.

and some misc "political" stuff:

• we've been happy to accept an invitation to attend the GNOME Advisory Board meeting at GUADEC: Jord (thanks, again!) kindly accepted to represent Debian there. An early report has been posted on his blog, a more detailed one is forthcoming to -project.
• OSI has discussed with their affiliates the possibility of signing the Declaration of Internet Freedom. On behalf of Debian, I've vouched for it. By the way, I'm serving pro tempore as Debian liaison at OSI, but if there are people interested in doing that, I'll be happy to consider handing over the responsibility to other (it's a quite lightweight one, at present).

Happy RC bug squashing,
Cheers.

---

PS the boring day-to-day activity log for July 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201207

As a introduction on what is SILPA you can read my previous post This application was written by Santhosh about 6-7 years ago in May 2010 2009 ( Correct me if I'm wrong Santhosh ;-) Thanks to Santhosh for correcting me ) in Python without using any web framework. He crafted whole system single handed and with very nice ideas. He wrote all the modules available in it for all possible Indian languages even though he is native Malayalam speaker. I joined him 3 years back and I was providing some or other improvements and he ultimately made me as the Co-Developer of SILPA. Recently I came across the Flask microframework and thought of rewriting SILPA to use Flask. You might ask me why Flask? So here are my reasonings

 1. Why reinvent Python webframework when we can use existing? Besides maintaining a not so well known framework code which is not written by you will be difficult    and time consuming.
 2. All Python modules are useful even outside SILPA project why make them dependent on SILPA code?
 3. Templating system used is very old and can't be ported to new Python version lets use something which is new and portable
 4. If new contributor wanted to add a module he need to understand SILPA framework to write and integrate it. This might be discouraging new
    contributions so lets use pure python module and make integration with SILPA straight forward.
 5. I was overwhelmed by Flask :P

I had previously tried bottle microframework but for some reason I couldn't crack it so choosing of Flask is not because it was better among other available microframework or something like that. I choose it because I could easily understand it and it suited my needs better. So I proposed about this to Santhosh and he asked me for a POC code. I started working on April 8 2012 and today by August 5 I can say I've first stable version of SILPA rewritten using Flask deployable over WSGI on Apache2. It can also be deployed on other servers supporting WSGI interface. It took me quite a bit of time but to be frank in the middle I completely had stopped coding because of some other works. Here is the commit graph and code frequency graphs from Github.

Changes in New Version Here is some major changes in current version.

1. Newly developed framework provides plug and play features for new modules.
2. You need to install the modules which you want to run on it separately using PIP and only minimal code change is required. Only configuration and webbridge.py 
   will be changed. Ofcourse you need to add html file to *templates* folder.
3. This version uses decent routing logic from Flask for page serving.
4. It also provides JSONRPC interface which is compatible with previous version but only with method name changed.
5. This version separates out run time module loading and routing the request to modules.
6. Dynamic module loading now happens only at the beginning of application, so if I want to implement caching for particular services it will be straight forward
   now.
7. Routing requests to proper module is adapted from old version with few modifications
8. All previous pages are now converted to use *Jinja2* template system.

Things Remaining TODO Below is some stuffs still pending before this version can completely replace http://silpa.org.in

1. Separate all modules from original SILPA and get it uploaded to pypi
2. Provide RESTful interface
3. Provide proper LICENSE, AUTHORS file for all modules 
4. Prepare a Nice UI
5. Find a better name than *SILPA*
...

Update

Forgot to provide the github link and demo link for the new version of SILPA. Thanks to Aravinda for pointing
it out. Here it goes
Github: http://github.com/copyninja/Silpa-Flask
Demo: http://flasksilpa-indic.rhcloud.com

during the last weeks I was quite busy with other things (like DebCamp, DebConf, & vacations), so this is a report covering 4 weeks. at least I managed to catch up during the last days a bit

• #639565 liboauth-dev: "liboauth-dev: please make loose libcurl dependency"
  send a patch to the BTS
• #642165 src:libmime-lite-html-perl: "libmime-lite-html-perl: FTBFS: test failures"
  update test reference files, upload to DELAYED/2
• #656505 taxbird: "taxbird: build-dependency on libgeier-dev needs to be versioned"
  propose a different patch in the BTS
• #662606 php-horde-core: "php-horde-core: FTBFS: "This package contains role "horde" and requires package "horde/Role" to be used""
  add missing build dependency, upload to DELAYED/2
• #664724 rrdcached: "rrdcached: fails to install, postinst, invoke-rc.d rrdcached start, start-stop-daemon, segfault"
  (re-)create directories needed by daemon in init script, upload to DELAYED/2
• #665018 src:python-fixtures: "python-fixtures: FTBFS: AttributeError: 'BrokenFixture' object has no attribute 'items'"
  add patch from Emanuele Aina, upload to DELAYED/2
• #666231 src:byzanz: "byzanz: FTBFS with gtk 3.4"
  sponsor NMU from Etienne Millon, upload to DELAYED/2
• #678108 httpcomponents-core: "httpcomponents-core: FTBFS twice in a row - broken clean rule"
  call mh_clean as suggested by Niels Thykier, upload to DELAYED/2
• #678859 libmozjs185-dev: "libmozjs185-dev depends on libffi-dev"
  add missing dependency as proposed by Nicolas Boulenguez, upload to DELAYED/2
• #679669 src:underscore: "underscore: build-dependency on uglifyjs not satisfiable in wheezy"
  propose a patch in the BTS, maintainer prefers alternative ways of fixing it
• #680423 trac-wikiprint: "trac-wikiprint: depends on nonexisting python-pil if rebuilt"
  propose a patch in the BTS
• #680424 trac-odtexport: "trac-odtexport: depends on nonexisting python-pil if rebuilt"
  propose a patch in the BTS, then fixed & closed by maintainer
• #681146 bitlbee-common: "bitlbee-common: fails to install, purge, and install again"
  propose a patch in the BTS, then upload to DELAYED/2 with maintainer's permission
• #681567 opennebula-common: "opennebula-common: fails to install, purge, and install again"
  propose a patch in the BTS, then upload to DELAYED/2 with maintainer's permission

Tonight I upgraded my personal machine to run the recently released 3.5[.0] kernel. On my personal machine(s) I'm usually loathe to change a running kernel, but this one was a good step forward because it allows me to experiment with seccomp filters. I've tested the trivial "no new privileges" pctl and I followed along with the nice seccomp tutorial which gave me simple working code which I married to my javascript interpreter. On top of that I upgraded node.js, which meant I had to clean up a little depreciated code in my node reverse proxy - which is the public face of the websites I run upon my box. (The proxy tunnels to about 10 different thttpd instances, each running upon 127.0.0.1:xx). Happily however my weekend was not full of code, it was brightened by the opportunity to take pictures of Aurora and her long hair - more to come as I've still got about 350 images to wade through.. ObQuote: "Don't you think I make a remarkable queen? " - St. Trinian's (2007)

Three weeks ago, I slightly modified the comment system on this blog for an experiment. This blog is a standard wordpress installation. Comments are normally directed to the wp-comments-post.php script by the HTML form. What I did is:

• Create a comments-post.php wrapper script that just includes wp-comments-post.php (this allows things to still work properly after wordpress upgrades),
• Make the HTML form direct to a comments-post.php script,
• Add a usedForm=1 parameter to the HTML form action, such that comments-post.php is supposed to always be called with it,
• Add a simple javascript that adds a hasJS=1 parameter to the HTML form action when the page is loaded, and a Submit=1 parameter when the form is submitted.

During the past three weeks, on this blog, there were 7170 comments, 8 of which were actual comments. 7162 were spam (~99.9%).

• 3165 spams (~44.1%) were sent to the original WordPress comment handler (wp-comments-post.php) from 1589 unique IP addresses.
• 0 spam were sent to the new comment handler without a query string (comments-post.php), but 1 was sent with an empty query string (comments-post.php?).
• 18 spams were sent to the new comment handler with a lowercased query string (comments-post.php?usedform=1) from 6 unique IP addresses.
• 3971 spams (~55.4%) were sent to the new comment handler with the form query string (comments-post.php?usedForm=1) from 1153 unique IP addresses.
• 7 spams (~0.1%) were sent to the new comment handler with the full query string, including what is added through javascript (comments-post.php?usedForm=1&hasJS=1&Submit=1) from 5 unique IP addresses.

This means a large portion of spammers didn t care about actually checking the comment forms and used the standard wordpress url, and another large portion don t run javascript on their bots, although a very few do.

I s pose a better name would be How to preform automated deep-source checks I ve pushed the code to paultag/qlight, and it s just for personal use at the moment. It s built to use paultag/sb around sbuild. Basically, it preforms a crudload of tests such as licensecheck, suspicious-source, pyflakes, bashisms, cppcheck, php in linter mode, and a ton of other checks. You can see a large majority of them in the source file. The batch runner is a singleton , so it s safely called more then once. It s running on a 5-minute cron, and I ve configured a dput target to push it into it s queue & run the enqueue script. Here s some example output from a random package I had lying around:

  ===== build status =====
sbo-run-clang  successful
sbo-run-gcc  successful
  ===== global status =====
Ran plugin extract, exit code 0
Ran plugin watch, exit code 0
Ran plugin licensecheck, exit code 0
Ran plugin validate-source, exit code 0
Ran plugin build, exit code 0
Ran plugin descr, exit code 0
Ran plugin lintian, exit code 0
  ===== descr =====
I: PT: Does this sound right? "gbemol is/are a(n) Graphical frontend for the Music Player Daemon (MPD)"
  ===== piuparts =====
   [last 20 lines]
  ===== licenses =====
BSD (3 clause) 
GPL (v2 or later) 
GPL (v2 or later) GENERATED FILE
GPL (v2 or later) (with incorrect FSF address) 
LGPL (v2 or later) (with incorrect FSF address) 
*No copyright* GPL (v2 or later) 
*No copyright* UNKNOWN
  ===== lintian =====
I: gbemol: spelling-error-in-binary usr/bin/gbemol Wheter Whether
W: gbemol: hardening-no-relro usr/bin/gbemol
I: gbemol: spelling-error-in-manpage usr/share/man/man1/gbemol.1.gz allows to allows one to
  ===== watch =====
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   http://sf.net/gbemol/gbemol-(.*)\.tar.gz
-- Found the following matching hrefs:
     gbemol-0.3.tar.gz
     gbemol-0.3.1.tar.gz
     gbemol-0.3.2.tar.gz
Newest version on remote site is 0.3.2, local version is 0.3.2
 => Package is up to date
-- Scan finished
  ===== source issues =====
suspicious-source:
bashisms:
script ./ltmain.sh does not appear to have a #! interpreter line;
you may get strange results
possible bashism in ./ltmain.sh line 368 ($RANDOM):
        my_tmpdir="$ my_template -$ RANDOM-0 $$"
error: ./ltmain.sh: Unterminated quoted string found, EOF reached. Wanted: <">
pyflakes:
cppcheck:
[src/gbemol-utils.c:167]: (error) Memory leak: path
[src/libmpdclient.c:1654]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1655]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1656]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1664]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
phplint:
desktop validate:
.//.pc/desktop_file.patch/gbemol.desktop: warning: key "Encoding" in group "Desktop Entry" is deprecated
.//.pc/desktop_file.patch/gbemol.desktop: error: (will be fatal in the future): value "gbemol.png" for key "Icon" in group "Desktop Entry" is an icon name with an extension, but there should be no extension as described in the Icon Theme Specification if the value is not an absolute path
.//.pc/desktop_file.patch/gbemol.desktop: warning: value "Application;AudioVideo;Player;" for key "Categories" in group "Desktop Entry" contains a deprecated value "Application"
mp3check:
mp3val:
fontlint:
ftvalid:
msgfmt:
POFileChecker:
POFileSpell:
pngcheck:
jpeginfo:
ogginfo:
oggz-validate:
xmllint:
lacheck:
jslint:
check for environ calls to home:
check for use of /tmp:
./intltool-extract.in:    $OUTFILE     = "./tmp/$OUTFILE.h"
check for use of system():
./aclocal.m4:    lt_ld_extra= awk '/^include /   system(sprintf("cd /etc; cat %s 2>/dev/null", \[$]2)); skip = 1;     if (!skip) print \[$]0; skip = 0;  ' < /etc/ld.so.conf   $SED -e 's/#.*//;/^[  ]*hwcap[    ]/d;s/[:,  ]/ /g;s/=[^=]*$//;s/=[^= ]* / /g;/^$/d'   tr '\n' ' ' 
./intltool-merge.in:    return 1 if ((-x $file) or (system("$file -l >$devnull") == 0));
./intltool-merge.in:    system ("$iconv -f $encoding -t UTF-8 <$devnull 2>$devnull");
./intltool-update.in:    return 1 if ((-x $file) or (system("$file --version >$devnull") == 0));
./intltool-update.in:          system ($EXTRACT, "--update", "--srcdir=$SRCDIR",
./intltool-update.in:          system ($EXTRACT, "--update", "--type=$gettext_type", 
./intltool-update.in:    system ("$MSGMERGE", "-o", $outfile, $infile, "$MODULE.pot");
./intltool-update.in:    system ("$MSGFMT", "-o", "$devnull", "--verbose", $output_file);
./intltool-update.in:   system ("$MSGFMT", "-o", "$devnull", "--verbose", "$SRCDIR/$lang.po");
./configure:    lt_ld_extra= awk '/^include /   system(sprintf("cd /etc; cat %s 2>/dev/null", \$2)); skip = 1;     if (!skip) print \$0; skip = 0;  ' < /etc/ld.so.conf   $SED -e 's/#.*//;/^[   ]*hwcap[    ]/d;s/[:,  ]/ /g;s/=[^=]*$//;s/=[^= ]* / /g;/^$/d'   tr '\n' ' ' 
./intltool-extract.in:        system ("mkdir -p $dirname");
./intltool-extract.in:        system("mkdir tmp/"); 
./debian/gbemol.1:the \fBDebian\fP system (but may be used by others).  Permission is 
./debian/manpage.sgml:      the &debian; system (but may be used by others).  Permission is
  ===== sbo-run-clang.log warnings =====
update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: Invalid argument
update-binfmts: warning: unable to enable binary format python2.6
dpkg-source: warning: extracting unsigned source package (gbemol_0.3.2-2.dsc)
dpkg-buildpackage: warning: using a gain-root-command while being root
gbemol-preferences.c:483:13: warning: 'gbemol_preferences_load_spin' defined but not used [-Wunused-function]
gbemol-net.c:60:11: warning: variable 'res' set but not used [-Wunused-but-set-variable]
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfontconfig.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libatk-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against librt.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libgio-2.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libcairo.so.2 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpango-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangocairo-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfreetype.so.6 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangoft2-1.0.so.0 (it uses none of the library's symbols).
dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe
     + add misc-depends to fix lintian warning.
  ===== sbo-run-gcc.log warnings =====
dpkg-source: warning: extracting unsigned source package (gbemol_0.3.2-2.dsc)
dpkg-buildpackage: warning: using a gain-root-command while being root
gbemol-preferences.c:483:13: warning: 'gbemol_preferences_load_spin' defined but not used [-Wunused-function]
gbemol-net.c:60:11: warning: variable 'res' set but not used [-Wunused-but-set-variable]
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfontconfig.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libatk-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against librt.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libgio-2.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libcairo.so.2 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpango-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangocairo-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfreetype.so.6 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangoft2-1.0.so.0 (it uses none of the library's symbols).
dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe
     + add misc-depends to fix lintian warning.

Not as interesting as it can get, but still pretty cool. Andrea, I swear I m not picking on you :) More to come later (hopefully) :)

I got a few emails about the status panel I'd both toyed with and posted. The end result is that the live load graphs now have documentation, look prettier, and contain a link to the source code. Apart from that this week has mostly involved photographing cute cats, hairy dogs, and women in corsets. In Debian-related news njam: Insecure usage of environmental variable was closed after about 7 months, and I reported a failure of omega-rpg to drop group(games) privileges prior to saving game-state. That leads to things like this:

skx@precious:~$ ls -l   grep games
-rw-r--r--   1 skx games   14506 Jul  8 15:20 Omega1000

Not the end of the world, but it does mean you can write to directories owned by root.games, and potentially over-write level/high-score files in other packages leading to compromises. ObQuote: "Your suffering will be legendary, even in hell! " - Hellraiser II (Did you know there were eight HellRaiser sequels?)

Forward Error Correction with the TI CC1120 Sub-GHz transceiver We re building a new, fancier flight computer and decided to give the new, fancier TI CC1120 chip a try. It has significantly more transmit power (+16dBm) and receive sensitivity (-109dBm) than the transciever built in to the TI CC1111 RF SoC that we ve built our other boards with. Given that we d already decided to use a fancier STM32L processor, using the nicest stand-alone radio chip we could find seemed to make good sense. One of the nice features found in the CC1111 is support for FEC using a 1/2 rate constraint length 4 convolutional code. Both the encoder and the soft-decision Viterbi decoder are built right into the chip; all we had to do was flip a couple of bits and we got error correction, interleaving, data whitening and CRC checking all for free. We assumed that, as the CC1120 was advertised as a better replacement for our CC1111 radio, that it would also include all of these fine features. After we got the chip all soldered down to our first prototype boards, we read through the reference manual looking for the right register values to flip for forward error correction. We found support for CRC computation and data whitening, but no mention of interleaving or forward error correction at all. Thanks, TI. Of course, without the FEC pieces, the other packet handling hardware is worthless you can t exactly do a CRC on the encoded data and expect it to be useful on the other end of the link, and data whitening happens after the CRC. Basic CC1120 driver I assumed I d eventually figure out something to do about the FEC pieces and started writing a basic CC1120 driver AltOS, the operating system used in all of our projects. I started with the transmit side, as that seemed easier as I could use the existing packet hardware by uploading the desired bit sequence using whatever fancy encoding desired and the hardware would happily transmit it. I first tested that using low data-rate bits (2kbps), transmitting an alternating sequence of zeros and ones to generate a 1kHz tone on a regular 70cm receiver. That seemed to work just fine, demonstrating that I had some grasp of the basic operation of the chip. Convolutional Encoding in Software Not to be daunted by a small challenge in software, I set out to replicate the encoding scheme used in the CC1111 chips for use in our CC1120-based design. We obviously want our CC1120 boards to interoperate with the CC1111 boards. Fortunately, TI fully documents the CRC, data whitening, convolutional encoding and interleaving done in the hardware. I created my own implementation of the encoder (licensed, as usual, under the GPLv2) which you can see in the file ao_fec_tx.c It s not quite as flexible as the hardware, it always whitens the data and always appends the CRC bytes to the end of the packet, along with the necessary trellis termination bytes. The core of that code is surprisingly simple:

static const uint8_t ao_fec_encode_table[16] =  
/* next 0  1      state */
    0, 3,   /* 000 */
    1, 2,   /* 001 */
    3, 0,   /* 010 */
    2, 1,   /* 011 */
    3, 0,   /* 100 */
    2, 1,   /* 101 */
    0, 3,   /* 110 */
    1, 2    /* 111 */
 ;
uint8_t
ao_fec_encode(const uint8_t *in, uint8_t len, uint8_t *out)
 
    uint8_t     extra[AO_FEC_PREPARE_EXTRA];
    uint8_t     extra_len;
    uint32_t    encode, interleave;
    uint8_t     pair, byte, bit;
    uint16_t    fec = 0;
    const uint8_t   *whiten = ao_fec_whiten_table;
    extra_len = ao_fec_prepare(in, len, extra);
    for (pair = 0; pair < len + extra_len; pair += 2)  
        encode = 0;
        for (byte = 0; byte < 2; byte++)  
            if (pair + byte == len)
                in = extra;
            fec  = *in++ ^ *whiten++;
            for (bit = 0; bit < 8; bit++)  
                encode = encode << 2   ao_fec_encode_table[fec >> 7];
                fec = (fec << 1) & 0x7ff;
             
         
        interleave = 0;
        for (bit = 0; bit < 4 * 4; bit++)  
            uint8_t byte_shift = (bit & 0x3) << 3;
            uint8_t bit_shift = (bit & 0xc) >> 1;
            interleave = (interleave << 2)   ((encode >> (byte_shift + bit_shift)) & 0x3);
         
        *out++ = interleave >> 24;
        *out++ = interleave >> 16;
        *out++ = interleave >> 8;
        *out++ = interleave >> 0;
     
    return (len + extra_len) * 2;
 

ao_fec_encode_table takes care of the convolutional piece, ao_fec_whiten_table (not shown) contains the data whitening codes while the little loop at the bottom does the interleaving. I didn t spend a lot of time optimizing this as it seemed to run plenty fast on the 32MHz processor. With the bits all nicely encoded, I passed them to my simple CC1120 driver and let it send them along to our CC1111 receiver. Much to my delight, with only a few bugs fixed, it worked just fine! I d managed to get one direction working, validating the hardware and making it clear that we d at least be able to use the board in flight for telemetry. Getting Soft-decision data out of the CC1120 In packet mode, the CC1120 receiver converts the incoming GFSK stream directly into bits, making its best guess as to whether the detected value was a one or zero. Of course, like any FM modulation, the receiver will be producing intermediate values, especially when the signal is weak or masked by interference. A Viterbi decoder can make good use of this information;these noisy intermediate values are less influential in the final resulting path cost which allows the strong 0/1 values to pull the solution to the right one. Soft decision decoding yields a theoretical 2dB gain over hard decision decoders, and given our desire for maximum packet reception over long distances, it seemed important to make this work. The CC1120 has a way to recover the soft decision values, but only one 8-bit value at a time. It does, however, helpfully provide a GPIO wire which can be programmed to interrupt the CPU when the soft decision data has arrived. So, I hooked up a GPIO line on the processor to the GPIO line on the CC1120 and let it interrupt once per bit time, or 38400 times per second, to read this valuable soft decision data out of the radio part. According to TI support, this is the only way to make this work, and that they generally recommend that developers use the packet mode stuff and throw away this valueable source of data. I was happily surprised to learn that the STM32L can in fact survive this interrupt rate, and by setting interrupt priorities appropriately, I manage to reliably capture an entire packets worth of data without dropping a single bit. I set up a simple packet dumping function to run on the prototype board and captured raw encoded bytes from our CC1111-based flight computer so that I d have some real data to work from. Soft-decision Viterbi Decoding The sample encoding implementation from TI made that piece fairly straightforward. Lacking a similar sample for the decoding piece meant that I d be starting from scratch. And, implementing something that seemed like magic to me when I started. I started by reading a few articles on Viterbi decoding: I found Chip Fleming s tutorial very useful as it included a bunch of worked examples using a convolutional encoders with constraint length (k) of 3 instead of 4; this makes all of the trellis diagrams half the size. I also sent my friend, Phil Karn, a query about this process, and he responded with a bunch of helpful suggestions, while also recommending that I just steal his DSP and FEC library implementation. Of course, I don t like using code that I can t understand, and his code only implements much longer codes than I needed, so I d have to understand it anyways to make it work for me. Phil s code is a marvel, but it s not exactly comprehensible to someone who doesn t even know what a Viterbi decoder is supposed to do. So, with Chip s tutorial on my screen, I wrote a primitive Viterbi decoder. This did only the Viterbi step, and split that into two phases. The first computed the cost for each state in the system for every received pair of symbols received. It also tracked the chain of states through the entire decode process:

    for (state = 0; state < 8; state++)  
        struct ao_soft_sym zero = ao_soft_sym(ao_fec_encode_table[state * 2 + 0]);
        struct ao_soft_sym one = ao_soft_sym(ao_fec_encode_table[state * 2 + 1]);
        uint8_t zero_state = ao_next_state(state, 0);
        uint8_t one_state = ao_next_state(state, 1);
        int zero_cost = ao_cost(s, zero);
        int one_cost = ao_cost(s, one);
        zero_cost += cost[b][state];
        one_cost += cost[b][state];
        if (zero_cost < cost[b+1][zero_state])  
            prev[b+1][zero_state] = state;
            cost[b+1][zero_state] = zero_cost;
         
        if (one_cost < cost[b+1][one_state])  
            prev[b+1][one_state] = state;
            cost[b+1][one_state] = one_cost;
         
     

At the end of the packet, it found the least costly path and walked that from back to front, generating output bits (yes, one bit per element of the bits array):

for (state = 1; state < 8; state++)  
    if (cost[b][state] < c)  
        c = cost[b][state];
        min_state = state;
     
 
for (b = len/2; b > 0; b--)  
    bits[b-1] = min_state & 1;
    min_state = prev[b][min_state];
 

Completely separate from this code were the interleaving and data whitening stages. Keeping those separate allowed them to be tested independently, which made debugging them far easier. None of this code ever ran on the STM32L processor, it was instead run in a test framework on my laptop. I first got this working with data generated by the software convolutional encoder, then I managed to get it to decode the raw packets that I had recorded directly from the receiver it self. This marked a significant milestone in my mind I d managed to replace, in software, the CC1111 FEC encoder and decoder. However, the decoder performance was not fast enough to manage back-to-back packets yet. Having something in hand that worked meant that further development would be directly comparable to something that did work, making optimization much easier. Yet another lesson in making it work, and then making it work fast. Optimizing the Viterbi Decoder The first step was to reduce memory usage within the decoder. While it s best to defer computing the path until the very last bit has been received, it s also true that the chances of a bit affecting old data reduces as the distance between the new and old data increases. With the rule of thumb being a distance of k 6 or k 7, I saved 32 output bits for each state and wrote the oldest of 8 each time the buffer filled up, making a distance of 24 bits between the newest saved data and the current decoding position. That meant a fixed amount of storage was needed for an arbitrarily long packet. I tested a variety of different history sizes from 8 to 56 bits (cooresponding to 16, 32 and 64 bits of saved data for each state). With 100000 random packets generated that had gaussian noise added to each bit, here are the error rates for the different history lengths:

History Length	Total Packets	Correct	Incorrect
8 bits	100000	90520 (90.52%)	9480 (9.48%)
24 bits	100000	93614 (93.61%)	6386 (6.39%)
56 bits	100000	93620 (93.62%)	6380 (6.38%)

The performance of the 8-bit history and 16-bit history versions turned out to be essentially identical (given the 32-bit nature of the STM32L, that shouldn t be too surprising). The performance of the 56-bit version, which manipulated uint64_t data types was sigificantly slower, and given the very modest benefit, deemed not worth it. If we move to a larger constraint length (k > 4), or started using a punctured convolutional encoding, we would want to re-measure this. The next step was to combine the de-interleaving, decoding, de-whitening and CRC computation into a single loop. This would eliminate a pile of data copying and extra memory usage. Nothing fancy here, but it made enough of a difference that I could decode most of the incoming packets now, dropping only about 25% of them. To get to being able to decode every packet, I needed to start decoding the packet while it was being received. Because of interleaving, I had to have 32 bits of data to be able to decode anything, so it was a simple matter of having my per-soft-value interrupt handler start the decoder going after each interleave block of data was received. With this in place, I was able to decode a 76-byte payload (160 transmitted bits) in 14.7 ms, or just about 50% of the time it took to transmit the packet. Unrolling one of the inner decoding loops eliminated a bunch of computation and sped this up even more, decoding packets in 9.5ms. Success at last, a solid 100% of packets received were being decoded, as long as the CPU was otherwise idle. Finally, I went and re-read Phil Karn s code and pondered over how it works. It was opaque until I wrote a message back to Phil asking how it worked and pointing out sections that were confusing. Fortunately, simply phrasing my confusion in words made me understand how the code works. Here s my original code for computing the per-state cost metric:

/* Metric for a zero bit */
uint32_t    bitcost = ((uint32_t) (s0 ^ ao_fec_decode_table[(state<<1)]) +
               (uint32_t) (s1 ^ ao_fec_decode_table[(state<<1)+1]));
 
    /* Total path metric and state for a zero bit */
    uint32_t    cost0 = cost[p][state] + bitcost;
    uint8_t     state0 = ao_next_state(state, 0);
    /* Compare the total path metric against all existing
     * metrics heading into the same new state, choosing
     * the least expensive and killing the others.  Record
     * the new lowest cost and output bit.
     */
    if (cost0 < cost[n][state0])  
        cost[n][state0] = cost0;
        bits[n][state0] = (bits[p][state] << 1)   (state & 1);
     
 
 
    /* Total path metric and state for a one bit.  Because
     * encoding a one is always exactly the opposite of
     * encoding a zero from any state, this cost is
     * 255*2 - bitcost
     */
    uint32_t    cost1 = cost[p][state] + 510 - bitcost;
    uint8_t     state1 = ao_next_state(state, 1);
    if (cost1 < cost[n][state1])  
        cost[n][state1] = cost1;
        bits[n][state1] = (bits[p][state] << 1)   (state & 1);
     
 

Before this loop is run, I have to initialize cost[n] to a large value so that the comparisons will be true for the first case to reach each state. Phil s code, in contrast, doesn t have to initialize cost[n] as we know which two states can reach any possible new state. His code looks like this:

/* Metric for a zero bit */
bitcost = ((uint32_t) (s0 ^ ao_fec_decode_table[(state<<1)]) +
       (uint32_t) (s1 ^ ao_fec_decode_table[(state<<1) 1]));
/* Only state and state+4 reach state<<1 with a zero bit */
/* Cost from state to state<<1 for a zero bit*/
m0 = cost[p][state] + bitcost;
/* Cost from state+4 to state<<1 for a zero bit */
m1 = cost[p][state+4] + (510 - bitcost);
/* Which source state is cheaper? */
bit = m0 > m1;
/* Record the cheaper cost and the add to the path of bits */
cost[n][state<<1] = bit ? m1 : m0;
bits[n][state<<1] = (bits[p][state + (bit<<2)] << 1)   (state&1);
/* State and state+4 reach (state<<1)+1 with a one bit */
/* Cost from state to (state<<1)+1 for a one bit */
m0 -= (bitcost+bitcost-510);
/* Cost from state+4 to (state<<1)+1 for a one bit */
m1 += (bitcost+bitcost-510);
/* Which source state is cheaper? */
bit = m0 > m1;
/* Record cheaper cost and add to the path of bits */
cost[n][(state<<1)+1] = bit ? m1 : m0;
bits[n][(state<<1)+1] = (bits[p][state + (bit<<2)] << 1)   (state&1);

This code does two states at a time, and so while it s slightly longer than the above, it only needs to run half as many times. The resulting code now decodes packets in 5.7ms. The final version of the code can be see in ao_fec_rx.c The whole AltOS operating system, with many drivers and support for CC1111, AVR and STM32L processirs is available, under the GPLv2 from git://git.gag.com/scm/git/fw/altos

I got an awfull bug with npm after installing Node.js 0.8.0 and trying to do an npm -g update. The solution was deleting npm from /usr/local and with nave reinstall 0.8.0. After this my npm was working perfectly but that made me loose like 12 hours. (I thought was a bug in an app I m writing, I totally forgot I ran npm -g update.)