Some Random WTFs The Daily WTF is an educational and amusing site that recounts anecdotes about failed computer projects. One of their stories titled Remotely Incompetent concerns someone who breaks networking on a server and is then granted administrative access to someone else s server by the Data Center staff [1]! In one of the discussions about that I saw people make various claims about Data Center security, such as claiming that having their own locked room helps. My experience indicates that such things don t do much good, I have often been granted access to server rooms without appropriate checks. My experience is that security guards on site generally don t directly do any good. I once had a guard hold a door for me when I was removing a server from a DC without even bothering to ask for ID! On another occasion in the Netherlands I had a security guard who didn t speak English unlock the wrong server room for me, I used hand gestures to inform him that I needed access to the room with the big computers and he gave me the access I needed! It seems that the benefit of security guards is solely based on scaring people who don t have the confidence needed to bluff their way in. Preventing children from thieving is a good thing, On another occasion I showed ID and signed in for access to a DC owned by my employer and I used my security key to go through a locked door with a sign that promised many bad consequences if I failed to lock the door behind me. Then I discovered that the back door was wide open for the benefit of some electricians who were working in the building. Presumably the electricians who had no security training were expected to act as ad-hoc security guards if someone tried to enter through the back door presumably they would not have been good at it. When a company uses part of their own office for a server room then many of these problems disappear. But a common issue in such ad-hoc DCs is the lack of planning and procedures, I have lost count of the number of times I ve seen doors (and even windows) propped open to allow ventilation because there were too many servers for the air-conditioning to cope. The most ironic example of this is the company that had a walk-in safe (think of a small bank vault with concrete walls and thick solid steel door) used for storing servers but with it s door propped open to allow cooling. The advantage of a serious hosting company is that they will have procedures for cooling etc and will be very unlikely to do strange and silly things. Having a locked room in a DC makes some sense, but if security guards have the master keys and are allowed to use them then it might not do much good. The one time I locked my keys in such a room I had a guard let me in without verifying my ID or the claim that there were actually keys locked in the room. Presumably anyone could just claim to have forgotten their keys and get the door unlocked just like a cheap hotel. Locking a rack sounds like a good idea, but the racks I ve seen have had locks which are quite easy to pick. On the one occasion when I had to pick a lock on a rack (due to keys being too difficult to manage for the relevant people) the security guards didn t investigate, so either the security cameras were not supervised or they just didn t care about people picking locks in a shared server room. Also if you allow people to do things freely in a shared server room they could install devices to monitor network traffic. A locked cage in a server room should work well. In the one case where I worked for a company that used such a cage I found it to mostly work well apart from the few weeks when the lock was broken. One company that I worked for had scales before the door between a server room and the car-park to prevent people from stealing heavy servers. Of course that wouldn t stop people stealing hard drives full of data which is worth more than the servers! Also an over-weight colleague had to have the scales disabled for him (as they were based on absolute mass not unexpected changes in an individual s mass) which presumably means that any skinny employee could steal a 2RU server and still be below the mass threshold. How to Solve some of these Problems Computers are subject to all manner of security problems. But they tend not to do arbitrary things for no apparent reason and they will never give in to someone who is charming, attractive, or aggressive unlike humans. I have servers running on Hetzner, Linode, and the Rackspace Cloud. I am always concerned about possible security compromises. But I am not worried about someone climbing in a window of a server room or convincing a security guard to let them in through the door. All three of those hosting companies have the vast majority of interactions automated. I can change many aspects of the servers without involving ANY human interaction. Out of the three of those companies I have had some human interaction with Hetzner (who provide managed servers) when a hard drive needed to be replaced obviously replacing a disk in the wrong server would have been a significant system integrity issue even though everyone would be running RAID-1 and if Hetzner improperly disposed of the broken disk then there could be security issues but this is an unlikely mistake in the face of a rare occurrence. With Linode and the Rackspace Cloud (and the previous Slicehost hosting that was purchased by Rackspace) the most common interactions I have with employees of those companies are when my clients don t pay their bills on time and that s an administrative not a technical issue. When I do have to contact the support people about a technical issue it s usually something that s not immediately connected to the virtual server (EG a loss of routing to the DC). It seems most likely that there are a fairly small number of people who are allowed in the DCs for companies like Hetzner, Linode, and Rackspace. Those people would probably be recognised by the security guards and their work would be restricted to replacing failing hardware and not involve granting access requests. There are some unusual requests that they can process (EG one of my clients recently transferred a virtual server between business units) but even in those cases the administrative software controls who gets access. This is much better than just handing hardware access to what seems to be the correct physical server to a client. If you have software running a few computers and operating correctly then you can probably scale it up to run thousands of computers and have it still work correctly. But if you have a team of people controlling access requests and want to scale it up significantly then there are huge problems in hiring skilled people and training them correctly. There is a real risk of security flaws in such administrative software, if someone managed to exploit the automated management system for one of those three companies then they could probably gain access to the private data of any of their customers. But the risk of this seems a lot less than the risk of general incompetence among humans who perform routine and boring tasks which have the potential for great errors.

• [1] http://thedailywtf.com/Articles/Remotely-Incompetent.aspx

Related posts:

1. The Security Benefits of Being Unimportant A recent news item is the hacking of the Yahoo...
2. Security Lessons from a Ferry On Saturday I traveled from Victoria to Tasmania via the...
3. Public Security Cameras There is ongoing debate about the issue of security cameras,...

Some bashisms are syntax sugar at first sight, such as the += concatenation syntax. Usually, they happen to be faster than their more portable counterparts. But only with bash itself.

Take the following script as an example:

#!/bin/sh
# script1-portable.sh
part="$(seq 1 100000)"

for i in $(seq 1 10); do
seq="$ part "
seq="$ seq $ part "
done

$ time bash script1-portable.sh
user 0m20.837s

Now, compare to the following script that uses += :

#!/bin/sh
# script1-bash.sh
part="$(seq 1 100000)"

for i in $(seq 1 10); do
seq="$ part "
seq+="$ part "
done

$ time bash script1-bash.sh
user 0m14.227s

Yes, it's faster. However, when the first script is run with dash:

$ time dash script1-portable.sh
user    0m0.609s

[[ is another example:

#!/bin/sh
# script2-portable.sh
a="$(seq 1 100000)"; b="$(seq 1 100)"

for i in $(seq 1 10); do
[ "$a" = "$b" ]
done

$ time bash script2-portable.sh
user    0m9.148s

And the version using the bashism:

#!/bin/sh
# script2-bash.sh
a="$(seq 1 100000)"; b="$(seq 1 100)"

for i in $(seq 1 10); do
[[ $a = $b ]]
done

$ time bash script2-bash.sh
user    0m4.223s

Then again, the bash way is faster, yet it doesn't compare to dash:

$ time dash script2-portable.sh
user    0m0.588s

One of features I wanted to introduce with Weblate is ability to do some quality checks while translating. We want contributors to have easy access to translating, but still we need to ensure some quality of translations. That's why Weblate comes with set of quality checks which are performed on every translation unit. The current set of checks (see documentation) is pretty basic and more might be easily added in the future pretty easily. There are two ways how checks are used in Weblate. During translating, if any of checks fires, translator stays on current translation with warning that something might be wrong. This is not a hard error as he still can be correct, but should at least prevent accidental mistakes like corrupted format strings or missing trailing colon. The other way is to do review of a translation. Failing checks are shown on each translation page or you can browse them by project to give you overview what can be wrong. For example with phpMyAdmin this helped me discover that there were few languages where almost half of translated strings are actually not translated but simply copied from English. There were also other problems like inconsistent translations across branches. All in all using these features helped me to improve quality of translations in phpMyAdmin quite a lot and I hope we will keep it for the future.

Filed under: English Phpmyadmin Weblate 2 comments Flattr this!

I have a few thoughts based on the yet another round of debate where marketing clashes with free software advocation and technical details. Nothing new in the debate itself, but I'm adding a couple of insights.

Marketing is not highly respected by many technical people, and neither by the people wanting more advocating than the messing up with facts and feelings that the marketing does. I'm all for advocating free software, but it's currently not something you can use for marketing to win big markets. If we advance to a world where free software is as wanted as the green values today are, it can be used in marketing as well similar to all the ecological (according to the market department at least) products today, but alas the benefits of free software are not yet as universally known. Since it doesn't say much that touches the masses, advocating has a negative marketing effect since it takes space away from the potentially "hitting" marketing moves, in those cases where you target the big masses in the first place. "Open" this and that has some marketing power in it nowadays, but it's a mess of different meanings that probably doesn't advance libre software freedoms as such. Wikipedia has probably been the biggest contributor to advancing general knowledge of software and culture libre. Disclaimer: I'm not a proper marketing person, some more professional might have better insights in this area. If I'd be a proper marketing person, I'd decorate this blog post with fancy pictures so that more people would actually read it.

Some of the marketing can be done without sacrificing any of the advocation. The fabulous Fedora campaigns, graphics, slogans and materials are a great example of those and do an important job, even though Fedora isn't reaching the big masses with it (and Fedora isn't targeted for OEMs to ship to millions either). But they do hopefully reach a lot of level people on the grassroots level. I hope as a Debian Developer that more people valuing the freedoms of free software would help also Debian as a project to reach more of its advocation potential and developers from the more proprietary world. But I'm happy that at least some free software projects have nowadays true graphical and marketing talent. Even though not as widely known, freedoms of the software users including for example privacy aspects are a potential good marketing tool toward a portion of the developer pool.

Let's not forget that Android conquered the mobile market without using the brand power of Linux. The 30+ million people who know Linux a bit deeper than just "I've heard it" already run a Linux distribution like Ubuntu, but we need hotter brands than a project name of a kernel to reach the bigger masses. Call it Ubuntu, Fedora, or something, but no matter do what it needs to ship it to millions. AFAIK mostly Ubuntu and SUSE are shipping currently via desktop/notebook OEMs, and more Ubuntu than SUSE. Others aren't concentrating on the market, which is a very difficult one. Ubuntu is doing a lot more mass population advocating for free software than Android has ever done. Note for example the time any user tries to play a video for the first time in a non-free format - Ubuntu will tell the user about the problems related to those formats and asking a permission to install a free software player for those (or buy licensed codecs), and the Ubuntu Help texts describe a lot of details about restricted formats, DRM et cetera. Not to mention what happens if the person actually wanders into the community, discovering Debian, other free software projects, free software licenses and so on. Meanwhile Android users never notice anything being wrong while watching H.264 videos or watching DRM Flash videos. Granted, like Boot2Gecko debate shows, it may be a partially similar situation on shipping desktop Linux variants as well, in order to actually ship them via partners.

Anyway, the best example of brands is MeeGo. Even the LWN editor does not get into dissecting the meaning of MeeGo on Nokia N9, because "there has been no real agreement on that in the past", but just uses the brand name as is. That is the power of brands. Technical people debate that it's not really MeeGo, it's maemo GNU/Linux with a special permission from Linux Foundation to use the MeeGo brand name, and then counter-argument with that the MeeGo is an API and maemo matches the MeeGo 1.2 API which is actually just Qt 4.7 API. And actually, as proven by the LWN example, it's not even "technical people". For most of technical people Nokia N9 is MeeGo as well. Only the people who have actually worked on it plus the couple of other people migrated from maemo and MeeGo.com communities to Mer project understand the legacy, history and the complete difference between the Maemo Harmattan platform and what MeeGo.com was. Yet at the same time like all GNU/FreeDesktop.org/Linux distributions, they are 95% same code, just all the infrastructure and packaging and polishing and history is different.

For 99.9% of people who know what MeeGo is, MeeGo is the cool Nokia smartphone, one of its kind and not sold in some of the major western markets, and/or the colorful sweet characters of meego.com shown at one time on a couple of netbooks. That is the brand, and the technical details do not matter even to the technical people unless they actually get into working on the projects directly.

To most people, the Linux brand is a mess of a lot of things, while other brands have the possibility at least to have a more differentiating and unique appearance.

One of the prerequisites for seccomp filter is the new PR_SET_NO_NEW_PRIVS prctl from Andy Lutomirski. If you re not interested in digging into creating a seccomp filter for your program, but you know your program should be effectively a leaf node in the process tree, you can call PR_SET_NO_NEW_PRIVS (nnp) to make sure that the current process and its children can not gain new privileges (like through running a setuid binary). This produces some fun results, since things like the ping tool expect to gain enough privileges to open a raw socket. If you set nnp to 1 , suddenly that can t happen any more. Here s a quick example that sets nnp, and tries to run the command line arguments:

#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
#ifndef PR_SET_NO_NEW_PRIVS
# define PR_SET_NO_NEW_PRIVS 38
#endif
int main(int argc, char * argv[])
 
        if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0))  
                perror("prctl(NO_NEW_PRIVS)");
                return 1;
         
        return execvp(argv[1], &argv[1]);
 

When it tries to run ping, the setuid-ness just gets ignored:

$ gcc -Wall nnp.c -o nnp
$ ./nnp ping -c1 localhost
ping: icmp open socket: Operation not permitted

So, if your program has all the privs its going to need, consider using nnp to keep it from being a potential gateway to more trouble. Hopefully we can ship something like this trivial nnp helper as part of coreutils or similar, like nohup, nice, etc.

2012, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Hm, I realized that I announced this on Google+ (via Rcpp) as well as on Twitter, on the r-packages list, wrote a new and simple web page for it, but had not put it on my blog. So here is some catching up. Sequential Monte Carlo / Particle Filter is a (to quote the Wikipedia page I just linked to) sophisticated model estimation technique based on simulation. They are related to both Kalman Filters, and Markov Chain Monte Carlo methods. Adam Johansen has a rather nice set of C++ classes documentated in his 2009 paper in the Journal of Statistical Software (JSS). I started to play with these classes and realized that, once again, this would make perfect sense in an R extension built with the Rcpp package by Romain and myself (and in JSS too). So I put a first prototype onto R-Forge and emailed Adam who, to my pleasant surprise, was quite interested. And a couple of emails, and commits later, we are happy to present a very first release 0.1.0. I wrote a few words on a RcppSMC page on my website where you can find a few more details. But in short, we already have example functions demonstrating the backend classes by reproducing examples from

Johansen (2009)

and his example 5.1 via pfLineartBS() for a linear bootstrap example;

Doucet, Briers and Senecal (2006)

and their (optimal) block-sampling particle filter for a linear Gaussian model (serving as an illustration as the setup does of course have an analytical solution) via the function blockpfGaussianOpt()

Gordon, Salmond and Smith (1993)

and their ubiqitous nonlinear state space model via the function pfNonlinBS().

And to illustrate just why Rcpp is so cool for this, here is a little animation of a callback from the C++ code when doing the filtering on Adam's example 5.1. By passing a simple plotting function, written in R, to the C++ code, we can get a plot updated on every iteration. Here I cheated a little and used our old plot function with fixed ranges, the package now uses a more general function: The animation is of course due to ImageMagick glueing one hundred files into a single animated gif. More information about RcppSMC is on its page, and we intend to add more examples and extensions over time.

If you're in the habit of doing sudo debi at the end of a build, it's worth noting a complication with Multiarch.

Sometimes, it's tempting to rebuild (and install) a shared library with a few untested changes, however, if (like me), you have multiarch packages installed, this can cause a surprise:


dpkg: error processing ../libqof2_0.8.4-2_amd64.deb (--unpack):
trying to overwrite shared '/usr/share/doc/libqof2/changelog.Debian.gz', which is different from other instances of package libqof2:amd64


Bumping the changelog doesn't help:


dpkg: error processing libqof2:amd64 (--install):
package libqof2:amd64 0.8.4-3 cannot be configured because libqof2:armel is at a different version (0.8.4-2)
dpkg: error processing libqof2:armel (--install):
package libqof2:armel 0.8.4-2 cannot be configured because libqof2:amd64 is at a different version (0.8.4-3)


The fix for this is, of course, to remove libqof2:armel and all it's reverse dependencies and the packages can't be put back until you've built an armel version of your changes.

The way to backout the test change is a bit longwinded, depending on the complexity of your library:


sudo apt-get --reinstall install libqof2:amd64=0.8.4-2 libqof-dev:amd64=0.8.4-2 libqof2-backend-qsf:amd64=0.8.4-2 libqof2-backend-sqlite:amd64=0.8.4-2 libqof2-dbg:amd64=0.8.4-2 libqof2:armel=0.8.4-2


Hmm. I think a helper tool could be needed in the medium term here.

This also means that cross-building Emdebian Crush packages from a Wheezy / Wheezy+1 base is again looking as if it will need to happen inside a chroot, albeit with problems for the packages which are dependencies of the build tools (or the cross toolchain) itself, as the Crush packages will inevitably contain modified files. (For example, Emdebian Policy differs from Debian Policy by requiring - not forbidding - compression of debian/copyright.)

Cross-building packages which are intended to be binary compatible with Debian without conversion (including cross-built packages to use on top of Emdebian Grip) shouldn't be too affected. You just need to ensure that where the package exists in two or more versions, all installed architectures are built before any architecture is installed.

This is an additional burden compared to the world of dpkg-cross but it is key to how Multiarch allows cross-building to use sensible dependency resolution.

So, what I think we'll need is an enhancement to debi which can be passed an architecture (possibly an architecture list), debi then looks for both the native arch and the requested architecture(s), fails> if the .changes file for the extra architecture(s) isn't found or proceeds to install all packages for the native arch and arch-dependent packages for the second architecture in the same dpkg operation. As a further enhancement, debi might be able to check dpkg --print-foreign-architectures, check the output of $package:$arch against dpkg -l and either complain or automatically use the Multiarch support to look for the relevant .changes files. debi may even want to check the package (as it's working in the package top directory usually) for Multiarch support before accepting the architecture list option.

Think I need to do some reading of the debi source and see if a patch is workable...

It took a while to get it out, but finally, 3 years after the latest release (1.2), the version of 1.3 of xpyb (the XCB Python bindngs) is out. This version has a lot of improvement, and major bug fixes (memory corruption and memory leak were tracked down and fixed). One amazing feature that is now shipped with that release, is my code to export the xpyb API to other Python modules, allowing to draw with Pycairo in Python using XCB. Here is an example of a Python program that draws a spiral in a window using xpyb and Pycairo. You need xpyb >= 1.3 and Pycairo >= 1.10 to make this works.

import cairo
import xcb
from xcb.xproto import *
WIDTH, HEIGHT = 600, 600
def draw_spiral(ctx, width, height):
    """Draw a spiral with lines!"""
    wd = .02 * width
    hd = .02 * height
    width -= 2
    height -= 2
    ctx.move_to (width + 1, 1-hd)
    for i in range(9):
    ctx.rel_line_to (0, height - hd * (2 * i - 1))
    ctx.rel_line_to (- (width - wd * (2 *i)), 0)
    ctx.rel_line_to (0, - (height - hd * (2*i)))
    ctx.rel_line_to (width - wd * (2 * i + 1), 0)
    ctx.set_source_rgb (0, 0, 1)
    ctx.stroke()
# Connect to the X server
conn = xcb.connect()
# Get the X server setup
setup = conn.get_setup()
# Generate X ID for our X "objects"
window = conn.generate_id()
pixmap = conn.generate_id()
gc = conn.generate_id()
# Create a new window
conn.core.CreateWindow(setup.roots[0].root_depth, window,
                       # Parent is the root window
                       setup.roots[0].root,
                       0, 0, WIDTH, HEIGHT, 0, WindowClass.InputOutput,
                       setup.roots[0].root_visual,
                       CW.BackPixel   CW.EventMask,
                       [ setup.roots[0].white_pixel, EventMask.ButtonPress   EventMask.EnterWindow   EventMask.LeaveWindow   EventMask.Exposure ])
# Create a pixmap: it will be used to draw with cairo
conn.core.CreatePixmap(setup.roots[0].root_depth, pixmap, setup.roots[0].root,
                       WIDTH, HEIGHT)
# We just need a GC to copy later the pixmap on the window, so create one
# very simple
conn.core.CreateGC(gc, setup.roots[0].root, GC.Foreground   GC.Background,
                   [ setup.roots[0].black_pixel, setup.roots[0].white_pixel ])
# Create a cairo surface
surface = cairo.XCBSurface (conn, pixmap,
                            setup.roots[0].allowed_depths[0].visuals[0], WIDTH, HEIGHT)
# Create a cairo context with that surface
ctx = cairo.Context(surface)
# Paint everything in white
ctx.set_source_rgb (1, 1, 1)
ctx.set_operator (cairo.OPERATOR_SOURCE)
ctx.paint()
# Draw our spiral
draw_spiral (ctx, WIDTH, HEIGHT)
# Map the window on the screen so it gets visible
conn.core.MapWindow(window)
# Flush all X requests to the X server
conn.flush()
while True:
    try:
        event = conn.wait_for_event()
    except xcb.ProtocolException, error:
        print "Protocol error %s received!" % error.__class__.__name__
        break
    except:
        break
    # ExposeEvent are received when we need to refresh the content of the
    # window, so we copy the content of the pixmap (where cairo drew) in the
    # window
    if isinstance(event, ExposeEvent):
        conn.core.CopyArea(pixmap, window, gc, 0, 0, 0, 0, WIDTH, HEIGHT)
    # You click, I quit.
    elif isinstance(event, ButtonPressEvent):
        break
    conn.flush()

Seeing the complexity it is to draw something simple with this technology, I somehow understand why nobody bothered to release or use the code during the last 3 years. But hey, now that it's out, you can build the next Python based desktop environment with bleeding edge technologies. :-)

I use R a lot. It's one of the primary tools I use in my day job as a scientist analyzing large datasets. If you use LaTeX with R (as I often do), you probably use Sweave to interleave R output and figures with your text describing those figures using the noweb method of literate programming. Sweavealike is a plugin for IkiWiki that tries to do some of the useful things for IkiWiki that sweave does for R and LaTeX. You use it like the following:

[[!sweavealike  echo=1 code="""
a <- 1
a <- a + 10
print(a)
"""]]

which produces this result when run:

> a <- 1
> a <- a + 10
> print(a)
[1] 11

You can also generate figures with it:

[[!sweavealike  fig=1 echo=1 results="hide" code="""
plot(1:10,(1:10)^2,xlab="x",ylab=expression(x^2),main="Example Figure")
"""]]
> plot(1:10,(1:10)^2,xlab="x",ylab=expression(x^2),main="Example Figure")

The plugin itself uses the neat Statistics::R perl module to handle all of the heavy lifting. I personally plan on using this plugin to help write some more entries in my learning R series of posts that I'm beginning to work on. Hopefully I'll find and fix most of the bugs as I embark on that process so anyone else who uses the plugin won't, but feel free to e-mail me if something isn't working as it should. Finally, you shouldn't run this plugin on a publicly editable IkiWiki instance, because that would be a trivial local user exploit as R can run arbitrary code, read and write to arbitrary files, exhaust all memory, etc.

Actually, that's not true - I've produced plenty of open source software over the years. However, in a sense, it is true: only the very best actually get paid to work on open source software full time, and I'm not one of them. People like Linus Torvalds. People like Guido van Rossum, although even he supposedly divides his time, and does not work on Python full-time. Think about that. Python is a hugely popular programming language used by many companies and individuals who get a lot of value out of it. But the creator doesn't even work on it full-time. Now, that's just one example - perhaps Guido enjoys the things that Google sends his way to work on outside of Python in any event - but I think it's representative of open source in general. Back to me: I've produced some small bits of open source code that other people find useful. Several people have even built products on Hecl that make money. But I'm not good enough to work on open source full time - I'm not one of those famous, brilliant coders who is so good that someone will find a way to pay them to work on stuff that gets given away for free. I am, however, a good enough programmer to work on people's proprietary code, and have never had too much trouble finding someone with a project they're happy to pay me for. Why is that? Because it's so much easier to funnel money back into a proprietary project. If people like the product and buy it, the company gets money, which can be used to pay the developers. With open source, millions of people might use it and get a lot of value from it, but the developer has no right to receive any of that back as cash, which he or she can use to pay for things like food and rent. So, I can code tolerably well, and I could conceivably contribute more open source code, but instead I spend my time working on proprietary code because it pays the bills. Clearly, when I can, I use open source software in these endeavors, and contribute back whenever I can, but the "secret sauce" remains closed. That's a pair of hands lost to the creation of more open source. I know I'm not alone in this, either - tons of people work on mostly proprietary projects the world over, but relatively few people get paid to work on open source code full time. So when I read about people debating the utility of copyright bring up the existence of open source as some sort of counterexample, it irritates me a bit. The right level of protection and enforcement of copyright is a complex debate that I'm not going to get into here. What I want to point out is "that which is not seen". Sure, open source exists. But how much more of it would exist it there were more money to fund work on it? How much open source software remains an idea in the developers head that does not get realized for lack of time? People often criticize the "Linux desktop" despite its extraordinary strides in recent years. Well, how much farther along would it be if there were more people paid to work on the 'boring stuff', like usability testing? Ubuntu and Redhat pay a few people to do that kind of stuff, but how many more people do Microsoft and Apple have for that kind of work? That's not to say that open source "doesn't work" or some such nonsense. It obviously works quite well, but it really shines where the currency is code, not money. Developers can and do give back lots, in terms of code, bug reports, suggestions, documentation, and so on to open source projects, which make them better for all involved. Where open source doesn't seem to work quite as well are in small, fast-moving, consumer-oriented products. My guess is that 99% of iPhone users could care less about the source code for their apps, but on the other hand, a large portion of the Emacs user base more than likely has written at least a few lines of Elisp. In any event, the point isn't to beat up on open source software, but to counter this idea that "intellectual property" is in no way shape or form necessary because the existence of open source software somehow "proves" that "things will get made just the same". Yes, maybe they will, but in lower quantities than consumers might find desirable. After all, most of us aren't good enough to work on open source software.

I gave a brief introductory talk about Android at this month s LUV meeting. Here are the slides with a brief description. All the screen-shots were made on a Samsung Galaxy S running Cyanogenmod version 7.1 [1] (Android version 2.3.7). With that build of Cyanogenmod you can press the power button for about 1.5 seconds to get a menu which gives an option to take a screen shot. The aim of the talk was to give an overview of what Android can do. I also gave some random commentary about Android such as explaining why it doesn t make a good phone. Most of the pictures in this post have links to the Android applications in question. Essential and Important Apps I started by explaining why having root access to your system is really important, including the issue of backing up an Android phone [2]. Cyanogenmod includes a terminal program which allows you to run su - . Running a shell as root isn t generally that useful, what you really want is to be able to run programs such as Titanium Backup which can only work properly if given root access. When you run an OS that allows root access you can run su - at a terminal prompt and you can also have an application use a GUI to request root access. I recommend rooting and modding an Android phone immediately after buying it. However that takes some time which is somewhat equivalent to money and is a significant hidden cost to purchasing an Android phone. The business models of telephone companies seem to involve hidding users for unexpected fees and extra fees for excess bandwidth can be really expensive. 3G Watchdog is one app that can monitor bandwidth and disable data transfers if too much is used. Onavo is an alternative that allows tracking data use on a per-application basis, but it only runs on Android 2.3.x while 3G Watchdog works on Android 2.1. EBay has an official app which is handy for searching for items. So far I ve only used it to get price estimates and have used a PC for buying. K9 seems to be the best MUA for Android. The MUA that ships with Android 2.1 isn t nearly as good and K9 is good enough that I didn t even bother testing the MUA from Cyanogenmod. The above picture shows a list of mail in my SE Linux folder. The Opticron Grapher is a good graphing calculator. I won t claim it s the best because I didn t seriously test such programs, but for the basic tests I ve done it has worked well. The Google Maps client comes with every Android system, the above shows the location of the LUV meeting. Osmand is an Android client for the Open Street Map project. Here is the web site for the Open Street Map project [3]. One significant advantage of OSM over the Google Maps is that OSM is free, the data is all contributed by users like Wikipedia. Another significant advantage is that you can download as much data as you need to your phone, for example the entire dataset for Australia is about 200M. Storing 200M of data on your phone is no big deal when you consider the availability of phones with more than 16G of storage and the ability to use a map when offline is a real benefit. So far I ve used Osmand while waiting for a train at an underground station and I plan to use it to track my progress the next time I m on a cruise. Serval is a mesh networking application for Android that supports VOIP phone calls and distributing messages and files. It s designed to be used in disaster areas, but there are lots of other potential uses of the technology. The Serval Project blog has an article about the presentation they gave at LCA 2012 [4].
Periodic Droid is my favorite Periodic Table viewing program. The Tomfusion AU Weather forecast app seemed to be better than the one from the BoM last time I checked. It s probably the best weather app for Australia.
The Opera Mini browser is often faster than other browsers because it uses a compressing proxy run by Opera. It s not so good for privacy though I have been using Wapedia for browsing Wikipedia. Since giving the talk I discovered the Official Wikipedia browser from the Wikimedia foundation [5] which is a better fit for my needs and I ve uninstalled Wapedia. As an aside modern phones have 16G of storage or more and could easily have a copy of the entire English text of Wikipedia on internal storage. It would be good if someone like Jason King (who is known for work on stand-alone DVD images for Wikipedia) was to write an Android program to do this. Handy Apps
Androsensor is a program to display output from most (all?) of the sensors on your phone. The results aren t as accurate as one would hope, for example Earth s gravity is 9.81m/s^2 not the 10.26m/s^2 my phone registered. But they are a useful indication.

The Zxing Barcode scanner is one of many programs that will scan barcodes with the camera in an Android phone. It can launch a Google search on a product code or open a URL from a
QR Code. The above pictures show it scanning a Coke can (the can and other background was displayed on the full screen before the screen capture program activated), doing a Google search on the can barcode, and looking up a QR code that was on an advertisment outside the LUV venue. Cruise Cams allows you to download pictures from cruise ships. Some cruise ships have several cameras on different parts of the ship uploading pictures regularly so that people around the world can see what s happening. The c:geo opensource program allows you to get information on Geocaches and see a compass or map showing the location. This program has been getting an increasing number of features to do everything you might want to do related to Geocaching. The above picture shows some caches that are close to where I made the screen-shot, in a park a couple of Km from the meeting location. The Google Sky map uses augmented reality techniques to display stars in the direction that your phone is pointing along with their names and the names of the constellations.


The Marine Traffic program shows the locations of ships as well as lots of information about them. The above pictures show me discovering that the Pacific Sun was nearby, viewing it s location on Google Maps, seeing the details, and then viewing fan pictures. The developer s web site allows viewing all the same data without an Android phone [6]. Anyone can join the project by buying an Automatic Identification System (AIS) receiver and configuring a PC to take data from AIS and send it to the MarineTraffic.com servers. As an aside they seem to be missing coverage in western Victoria, so it would be good if someone near Apollo Bay or Warrnambool could install an AIS receiver and help out. Satellite AR uses augmented reality to show the location of satellites and other things in spare. Unfortunately the screen capture process turned off the camera as I had a sign advertising fast food positioned in an amusing location in the background.
Shipmate publishes a set of programs giving information on cruise ships, they have one program for each cruise line. Above is the program for the Princess cruise line, the above pictures give information on the Dawn Princess. Unfortunately the program wasn t usable without net access when I tried to use it on a cruise ship. Games Air Attack HD is an entertaining game that demonstrates the capability of Android phones to run action games. Like many Android games it has a free version and paid versions if you want more. Angry Birds is one of the most well known games for touch-screen devices. It has also spawned a huge line of merchandise.
Labyrinth Lite is one of the many Android games based on the old mechanical game where you tilt a toy to roll a ball-bearing through a maze. It s free and is better than most of the free games in that genre. Minecraft Pocket Edition allows you to play Minecraft on your phone. The demo version doesn t allow saving the game, you have to buy the game for about $5 if you want to do that. It also lacked the full features of the game last time I checked, it didn t have monsters.
Paradise Island is one of many business simulation games for Android. It s more playable than most and has very detailed graphics, but the down-side is that it s a memory hog and will crash if you don t have enough RAM. It s one of the games that are free to download but encourage you to pay money to level up, for some players it s probably a very expensive game. Tower Raiders 2 is one of the better tower defense games for Android.

• [1] http://www.cyanogenmod.com/
• [2] http://etbe.coker.com.au/2012/02/07/5-principles-backup/
• [3] http://www.openstreetmap.org/
• [4] http://www.servalproject.org/archives/862
• [5] https://play.google.com/store/apps/details?id=org.wikipedia
• [6] http://www.marinetraffic.com/

Related posts:

1. Choosing an Android Phone My phone contract ends in a few months, so I m...
2. Galaxy S vs Xperia X10 and Android Network Access Galaxy S Review I ve just been given an indefinite loan...
3. Standardising Android Don Marti wrote an amusing post about the lack of...

SHOGUN has been accepted for Google Summer of Code 2012. SHOGUN is a machine learning toolbox, which is designed for unified large-scale learning for a broad range of feature types and learning settings. It offers a considerable number of machine learning models such as support vector machines for classification and regression, hidden Markov models, multiple kernel learning, linear discriminant analysis, linear programming machines, and perceptrons. Most of the specific algorithms are able to deal with several different data classes, including dense and sparse vectors and sequences using floating point or discrete data types. We have used this toolbox in several applications from computational biology, some of them coming with no less than 10 million training examples and others with 7 billion test examples. With more than a thousand installations worldwide, SHOGUN is already widely adopted in the machine learning community and beyond. SHOGUN is implemented in C++ and interfaces to all important languages like MATLAB, R, Octave, Python, Lua, Java, C#, Ruby and has a stand-alone command line interface. The source code is freely available under the GNU General Public License, Version 3 at http://www.shogun-toolbox.org. During Summer of Code 2012 we are looking to extend the library in three different ways:

1. Improving accessibility to shogun by developing improving i/o support (more file formats) and mloss.org/mldata.org integration.
2. Framework improvements (frameworks for regression, multiclass, structured output problems, quadratic progamming solvers).
3. Integration of existing and new machine algorithms.

Check out our ideas list and if you are a talented student, consider applying!

Some of my recent time has been devoted to making our boot media more Mac friendly, which has entailed rather a lot of rebooting. This would have been fine, if tedious, except that some number of boots would fall over with either a clearly impossible kernel panic or userspace segfaulting in places that made no sense. Something was clearly wrong. Crashes that shouldn't happen are generally an indication of memory corruption. The question is how that corruption is being triggered. Hunting that down wasn't terribly easy.

My first thought was that we were possibly managing to load the kernel over a region used by UEFI code. UEFI defines two types of code - boot services and runtime services. While runtime services code and data must be preserved by the OS, in theory boot services code and data is available to the OS once the firmware has exited. In practice, that's not true. It seemed entirely possible that the kernel might be ending up on top of some of that boot services code or data and getting trodden on. Grub now has code to avoid putting the kernel on boot services, so testing the latest code seemed like a good plan. But no, crashes still happened.

That pretty much ruled out the bootloader. My next thought was that executing some of the firmware code was triggering a write to some other memory that contained the kernel. Josh Boyer suggested the next trick, which was to try marking the kernel read-only to see whether anything was hitting it. x86 lets you mark pages as read-only - any attempt to write to them should take a fault. UEFI functions are executed in the context of the kernel, so share the same page tables. That let me rule this out, since everything still went just as wrong and I wasn't taking an extra fault first.

However, at this point I was reasonably happy that it wasn't the kernel itself being overwritten - faults were occurring in userspace code as well. That was a pretty strong indication that what was happening was continuing to happen once userspace had started, so it wasn't a direct response to a firmware call. I made sure of that by stubbing out all the calls that could be triggered after kernel initialisation, and saw the same failures. Once all attempts to be clever have failed, it's time to just start using brute force. The kernel lets you reserve areas of RAM by passing arguments like memmap=0xlength$0xstart to block length bytes starting at start from being used. It took a while, but I finally found a 256MB range that made a difference - reserving it resulted in the machine booting reliably, letting the OS use it resulted in occasional crashes.

Definite progress. Comparing that memory range to the EFI memory map was helpful. There were several blocks of UEFI boot services data present there, which really seemed like too much of a coincidence. By reserving each of them in turn, I'd traced it down to a single 31MB region of boot service data - that is, memory reserved by the firmware for use by the UEFI boot services. Per spec, this is available to the OS once the boot environment has been exited. Nothing other than the OS should be touching this after boot, but something clearly was. Tracking down what was far easier than I expected, although the first attempt was a failure. Setting it read-only should have triggered a fault, but didn't. That was rather confusing. But, rather than give up, I patched the kernel to fill the region with 0xff at kernel init. Then I booted the system, read it back and looked for values that weren't 0xff. I got this:

00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff   ................ 
*
001568a0  ff ff ff ff ff ff ff ff  ff ff ff ff 84 00 00 00   ................ 
001568b0  00 20 a7 ac 46 00 00 00  00 00 00 00 00 00 06 01   . ..F........... 
001568c0  c2 0b 0c 00 ff ff ff ff  ff ff ff ff ff ff ff ff   ................ 
001568d0  ff ff 0a 04 f0 03 82 0d  40 00 00 00 ff ff ff ff   ........@....... 
001568e0  ff ff 00 21 00 36 9a 80  ff ff ff ff ff ff 00 7e   ...!.6.........~ 
001568f0  00 09 43 48 41 2d 47 75  65 73 74 01 04 02 04 0b   ..CHA-Guest..... 
00156900  16 32 08 0c 12 18 24 30  48 60 6c 2d 1a 0e 18 1a   .2....$0H l-.... 
00156910  ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................ 
00156920  00 00 00 00 00 00 00 dd  09 00 10 18 02 00 10 01   ................ 
00156930  00 00 dd 1e 00 90 4c 33  0e 18 1a ff ff 00 00 00   ......L3........ 
00156940  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................ 
00156950  00 00 bd ea f8 b3 ff ff  ff ff ff ff ff ff ff ff   ................ 
00156960  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff   ................ 
*
022fb000

That's a lot of 0xffs (around 31MB of them) with one small section that contains an 802.11 probe packet with the SSID of the hospital across the road from my house. Apple support network booting off wireless networks. It seems that the firmware brought up the wireless card, associated with this network (it's the only public one nearby) and then left the card DMAing packets into RAM. The read-only page attribute only applies to CPU-initiated accesses, so it could do this without triggering a page fault. It also explained why it was so random - whether memory corruption occurred would depend on whether a packet appeared between that memory being used by the OS and the kernel reinitialising the wireless card. It certainly explains why I couldn't reproduce it when I left the machine repeatedly rebooting on the bus home.

How do we fix this? Unsure. With luck disconnecting the UEFI driver in the bootloader should quiesce the hardware, but without testing I'm not sure of that yet. For now it's just another example of firmware managing to break expectations in deeply strange ways.

comments

Recently, I got to try out working in a couple different environments. Working in these environments helped me formalize some of the things that I like and some of the things I don't like. With these new environments I've come to realize my style of development. It's called penance driven development. It's called this because there are two important things:
1) It's better to ask forgiveness than for permissions
2) You need to be deserving of forgiveness.

So the concept is simple, whenever you break something, you first fix the symptom, then you improve the infrastructure so similar breaks should be harder to do. The cost of the penance doesn't have to be huge, but the penance has to be done, otherwise technical debt accumulates and people believe there is no downside to breaking things.

What needs to be improved depends on what is missing. For example, if an engineer broke a feature, but that feature didn't have a test, the penance should be to write a test. Even if the one that broke the feature isn't the owner or the one that wrote the feature. If someone breaks the build and no one noticed because the continuous build didn't email, the fix could be setting up the continuous build to email out build failures, or even better, run each change on a build bot before allowing commits.

I think I'll have to start a penance of the day blog/twitter stream.

I've been practicing this style of development for most of my career, and it's similar if less precise than the 5 whys . As mentioned in the The Lean Startup, the cost of the penance should be proportional to the cost of the break.

Documentation in Debian Edu is provided in several languages, and it is important to make it both easy to contribute and to keep the translated versions in sync. To do this we have come up with what we believe is a very efficient work flow.

1. The documentation is written in a moinmoin wiki (see for example the Squeeze release manual) with support for exporting the content as docbook XML.
2. This docbook document is given to po4a to extract a gettext style .pot file with the content, which in turn is used to create .po files with the translated text.
3. The .po files are given to translators, and they can always tell which part of the original wiki document is new or changed. They can use their normal translation tools like lokalize or poedit to write the translation. There is even a system in place to handle translated images.
4. The translated .po files are combined with the original docbook XML document using po4a to create a translated docbook document.
5. The final step is to use all the generated docbook files and create PDF and HTML version of the original and translated documents.

This setup work very well, but have a few issues. The biggest issue is that the docbook support we use in moinmoin is not actively maintained. The docbook support is also buggy, and our build system contain workarounds to make sure the generated docbook is usable despite these bugs. If you want to have a look at our setup, it is all there in the debian-edu-doc package.

So, since Wednesday, we have no more teenagers among our children. Indeed, our youngest, Magali, turned 20 and we happily celebrated that in a japanese restaurant in Paris : Magali is living in a (very) small flat in Paris as she attends preparation courses for exams meant to enter physical therapist schools. She's currently working very hard to prepare these and is left with only one week before the first exam. We're 100% confident that she'll succeed. Really, the days where our little shrimp was crawling across her bed, turning clockwise and finally sleeping in the width of the bed....do not seem so far. And still, these folks we call "our kids" are now grown-up persons living their own lives : Jean-Baptiste (some of my friends reading this blog have met him at DebConf 10) will soon enter profesionnal life after graduating an engineering school in electronics, at 24. Sophie, now 22, will resume her social worker school in Bordeaux in September and will probably start working as social worker ("assistante sociale", in French) afterwards. And we hope that our little shrimp (which her sister calls "mini-poop") starts a physical therapist school next year. So, well, more and more, Elisabeth and I are old parents and weeks where we're indeed both alone at home are now common (not to mention weeks where only one of us is alone either because she has professionnal commitments on some other side of the planet or I have some free software event on another side of the planet). It is indeed really likely that this year's holidays will be the first ones where we have a distant travel...which we'll be doing alone. Holidays in Nicaragua (you probably guess why Nicaragua) are much awaited and will be enjoyed a lot, but still...:-) So, well, to all those of you who read this and are parents, or plan to become parents, be happy. The wonder that it is lasts for years and it's very hard to describe precisely the link that connects parents and their children. At least the one we have with our children, which both us and them are proud of. And that pride and this link have no price. I really wish you guys to develop the same with your newborns (particularly those of you who very very very recently became parents anbd, yes I mean you who know who you are...).

---

---

Hi,

in this bits:

1. Debian Med Bug Squashing Advent Calendar 2011
2. Anniversary of Debian Med
3. Second Debian Med sprint (Southport, 27th-29th January 2012)
4. Mentoring of Month (MoM)
5. DDs who came to Debian because of Debian Med
6. Future plans
7. General lessons learned
   

Debian Med Bug Squashing Advent Calendar 2011In December last year Thorsten Alteholz has started a nice QA initiative which might be interesting for other teams next Advent. The Debian Med team was able to fix about 70 bugs in this time. Thanks to Thorsten for this nice piece of motivation and thanks to everybody who took part in the bug squashing.

Anniversary of Debian MedOn Mon, 7 Jan 2002 the Debian Med project was first officially announced. I submitted a short blog posting about this and perhaps you might like to see a long sequence of talks about this topic.

Second Debian Med sprint (Southport, 27th-29th January 2012)In end of January 2012 the Debian Med team has met to the second sprint. As last year I would call this a very successfull event and I would recommend other teams to instanciate such meetings as well. Feel free to read my more verbose report.

Mentoring of Month (MoM)I have started this project for the following reasons:

• gather more manpower to the team
• strengthen connections to upstream (which might become MoM students)
• help shy people to become more verbose
• to learn myself about potential problems of people who do not feel fit for packaging tasks

I made a short summary how the first MoM project worked (see at bottom).

DDs who came to Debian because of Debian MedAfter realising that several members of the Debian Med team finally became DDs I made a little survey to find out about their reasons to become DD / DM. I came to the conclusion that a Blend could be a nice entry point for people to join Debian because newcomers can identify themselves with a known topic (the scope of the Blend - in this case medicine and bioinformatics) first and learn Debian rules in a team with common interest. This perfectly fits my expectation which I had from the beginning 10 years ago and I would be very happy if other Blends would follow this example to be nice, inviting and try to *actively* ask people for cooperation (see some simple rules which I learned in this process below). In the teammetrics GSOC project some graphs were created where you can see the level of contribution of these people (and other team members).

Future plansCurrently some heavy work regarding bringing bibliographic references about packages straight into package information is going on. This topic is specifically interesting in Biology because programs are frequently connected to some publication about the methods used inside the code. This topic is as well relevant to Debian Science and DebiChem. Thanks to the patient work done by Charles Plessy we now have about 70 packages featuring debian/upstream files featuring bibliographic references and there is ongoing work to move these data to UDD to enable further usage. We are in the process of final polishing the format and finishing scripts for the import. If people are interested to join this effort this would be the right moment to raise their hand.

General lessons learned

1. Do not let wait anybody who wants to do work.
2. Newcomers are frequently shy - try to invite them kindly and patiently.
3. Tell people verbosely about your project - it is astonishing how less people know and what wrong assumptions they make about your project.

Kind regards
Andreas.

I've been experimenting with a new packaging tool/workflow based on marking certain commits on my integration branch for export as quilt patches. In this post I'll walk though converting the package nauty to this workflow.

1. Add a control file for the gitpkg export hook, and enable the hook: (the package is already 3.0 (quilt))

   % echo ':debpatch: upstream..master' > debian/source/git-patches
   % git add debian/source/git-patches && git commit -m'add control file for gitpkg quilt export'
   % git config gitpkg.deb-export-hook /usr/share/gitpkg/hooks/quilt-patches-deb-export-hook
   

   This says that all commits reachable from master but not from upstream should be checked for possible export as quilt patches.
2. This package was previously maintained in the "recommend topgit style" with the patches checked in on a seperate branch, so grab a copy.

    % git archive --prefix=nauty/ build   (cd /tmp ; tar xvf -)
   

   More conventional git-buildpackage style packaging would not need this step.
3. Import the patches. If everything is perfect, you can use qit quiltimport, but I have several patches not listed in "series", and quiltimport ignores series, so I have to do things by hand.

   % git am  /tmp/nauty/debian/patches/feature/shlib.diff
   

4. Mark my imported patch for export.

   % git debpatch +export HEAD
   

5. git debpatch list outputs the following

   afb2c20 feature/shlib
   Export: true
   makefile.in    241 +++++++++++++++++++++++++++++++++--------------------------
   1 files changed, 136 insertions(+), 105 deletions(-)
   

   The first line is the subject line of the patch, followed by any notes from debpatch (in this case, just 'Export: true'), followed by a diffstat. If more patches were marked, this would be repeated for each one. In this case I notice subject line is kindof cryptic and decide to amend.

    git commit --amend
   

6. git debpatch list still shows the same thing, which highlights a fundemental aspect of git notes: they attach to commits. And I just made a new commit, so

   git debpatch -export afb2c20
   git debpatch +export HEAD
   

7. Now git debpatch list looks ok, so we try git debpatch export as a dry run. In debian/patches we have 0001-makefile.in-Support-building-a-shared-library-and-st.patch series That looks good. Now we are not going to commit this, since one of our overall goal is to avoid commiting patches. To clean up the export, rm -rf debian/patches
8. gitpkg master exports a source package, and because I enabled the appropriate hook, I have the following

    % tar tvf ../deb-packages/nauty/nauty_2.4r2-1.debian.tar.gz   grep debian/patches
    drwxr-xr-x 0/0               0 2012-03-13 23:08 debian/patches/
    -rw-r--r-- 0/0             143 2012-03-13 23:08 debian/patches/series
    -rw-r--r-- 0/0           14399 2012-03-13 23:08 debian/patches/0001-makefile.in-Support-building-a-shared-library-and-st.patch
   

   Note that these patches are exported straight from git.
9. I'm done for now so

   git push 
   git debpatch push
   

the second command is needed to push the debpatch notes metadata to the origin. There is a corresponding fetch, merge, and pull commands. More info

• Gitpkg bug
• Git Repo with git-debpatch (WIP)
• Example package: nauty
• Example package: bibutils In this package, I was already maintaining the upstream patches merged into my master branch; I retroactively added the quilt export.