We have customers, which require a report about what we've done during maintenance windows. Usually this includes a report about upgrades, newly installed packages etc. and obviously everything we've done apart from that.
Till now we've prepared them manually. For a greater bunch of systems this is a big PITA, because to be somehow useful you have to collect the data for all systems and after that, prepare a report where you have:

• The upgrades done on all systems (e.g. a libc or kernel update)
  
  seperated from
  
• the updates specific to a certain system or system class

Its also error-prone because humans make mistakes.

Perl to the rescue!
At least the part about generating a report about installed/upgraded/removed packages could be automated, because dpkg writes a well-formed logfile in /var/log/dpkg.log. But I noticed that there appearently is no library specialised at parsing that file. Its not a big deal, because the format of that file is really simple, but a proper library would be nice anyway.
And so I wrote such a library.
It basically takes a logfile, reads it line by line and stores each line parameterized into a generic object.

Its features include:

• Parsing the logfile (obviously)
• Storing each line in a logfile as a DPKG::Log::Entry object that holds informations about the entry type (e.g. action or status update), the package associated (if any), timestamp, verbatim line, line number etc.

• Limiting the number of lines parsed to a time range specified by DateTime objects

Based on that, I wrote another library DPKG::Log::Analyse, which takes a log, parses it with DPKG::Log and then extracts the more relevant information such as installed packages, upgraded packages, removed packages etc.

This, in turn, features:
- Info about newly installed packages
- Info about removed packages
- Info about upgraded packages
- Info about packages which got installed and removed again
- Info about packages which stayed halfinstalled or halfconfigured at the end of the logfile (or defined reporting period)


These libraries are already uploaded to CPAN and packaged for Debian.
They passed the NEW queue very quickly and are therefore available for Sid:

http://packages.debian.org/sid/libdpkg-log-perl

As an example use (and for my own use case, as stated above), I wrote dpkg-report, which uses the module and a Template::Toolkit based template to generate a report about what happened in the given logfile.
It currently misses some documentation, but it works somehow like this:

Report for a single host over the full log:
dpkg-report

Report for a single host for the last two days:
dpkg-report --last-two-days

Report for multiple hosts (and logfiles):
The script expects that each log file has the name <systemname>.dpkg.log so that it can guess the hostname from the system and can grab all such log files from a directory if a directory is specified as log-file arguments:</systemname>

dpkg-report --log-file /path/to/logs

This will generate a report about all systems without any merging.

Report for multiple hosts with merging:
dpkg-report --log-file /path/to/logs --merge

This will do the following:

• Report packages which where installed/removed/etc. on all systems seperate from the upgrades only done on the specific systems
• If systems start with a common name and end on a number (e.g. mail1, mail2, mail3 etc.), report packages which where installed on all systems with that common name seperate from the upgrades only done on the specific systems
• For a specific system list only the changes on that particular system and nothing else.

A (fictive) report could look some what like this:

dpkg-Report for all:
------------------------
Newly Installed:
abc
Removed:
foo
Upgraded:
bar (0.99-12 - 1.00-1)

dpkg-Report for test*:
------------------------
Newly Installed:
blub

dpkg-Report for test1:
------------------------
Upgraded:
baz (1.0.0-1 -> 1.0.0-2)

dpkg-Report for test2:
------------------------
Upgraded:
zab (0.9.7 -> 0.9.8)

Currently this report generator is only included in the source package or in the git(hub) repository of the library. I wonder if it makes sense to let the source package build another binary package for it.
But its only a 238 lines perl script with a dependency on the perl library so I'm unsure if it warrants a new binary package. What do others think?

...For the cases where, yes, you actually do want it. For the impatient assuming you're in a clean checkout on the tip of your current branch:

git merge -s ours ref-to-be-merged
git diff ref-to-be-merged   git apply -R --index
git commit -F .git/COMMIT_EDITMSG --amend

you can then double check that everything is as you expect with git diff ref-to-be-merged, which should be empty. The longer story It's noted from time to time that git does not have a -s theirs option to complement the -s ours strategy when merging (or, it did, but it was removed long ago, see below). For those not savvy the idea with git merge -s ours is to pretend as though a merge from some other branch has occurred, though the result remains the current branch's contents, unchanged. This is a neat trick that can be used to show that all changes in some branch have been superceded by the current branch, which can help ease merges further down the line as parallel devlopment continues. However, there is no corresponding -s theirs option, which would conversely say that "everything developed up to here is superceded by this other branch". This feature was discussed and discarded by one of the git authors, who instead advised a developer to throw away the previous work and start on a fresh branch:

One big problem "-s theirs" has, compared to the above "reset to origin, discarding or setting aside the failed history" is that your 'master' history that your further development is based on will keep your failed crap in it forever if you did "-s theirs". Hopefully you will become a better programmer over time, and you may eventually have something worth sharing with the world near the tip of your master branch. When that happens, however, you cannot offer your master branch to be pulled by the upstream, as the wider world will not be interested in your earlier mistakes at all.

In what would probably qualify as "most cases", he's right, but sometimes you do want to do this, and not because you have "crap" in your history, but perhaps because you want to change the baseline for development in a public repository where rebasing should be avoided. In debian, for example, it's pretty common for maintainers who use git-buildpackage to build debian packages from git to have two branches: an "upstream" branch and a "debianized" branch. The upstream branch will follow the changes from the upstream project (either by importing tarballs, or by pulling directly from the project's git repo if they also use git), while the debianized branch contains all the packaging related changes as well as any debian-specific changes to the code. In this context, if the upstream project uses git and rebases their history (out of your control), or if they shift development to a different branch, you might want to have a way to merge with their latest changes without rebasing/abandoning your current debian branch. Hence, a "theirs" merge. The author is totally right that the upstream project may be less interested in pulling from such a repository (since they would drag in the entire previous history), and if that's a problem you might need to splinter off the old branch and start a fresh branch instead. But at least for the projects with which I work, I find that it's more likely that pulling is pretty much one way upstream to debian, and that the patches flow back to the upstream project by way of bug trackers, git format-patch sendmail, or quilt patches sitting in the patch-tracker. And if it becomes a problem later you're no worse off, you can still split off a new branch or take some other action. So anyway, I asked Teh Internetz and found a few hits in the right direction, such as this guy at stack overflow, or this guy here. But from an aesthetic point of view I thought their solutions still left some room for improvement--they were both more complicated, involved setting up temporary branches, and I don't know, just weren't pretty on the eyes. So, ending where I started, this is the relatively clean and easy to follow 3 lines I conjured up, which I will give to teh internetz for posterity.

git merge -s ours ref-to-be-merged
git diff ref-to-be-merged   git apply -R --index
git commit -F .git/COMMIT_EDITMSG --amend

Alternatively, if you want to keep the local upstream branches fast-forwardable, a potential compromise is to work with the understanding that for sid/unstable, the upstream branch can from time to time be reset/rebased (based on events that are ultimately out of your control on the upstream project's side). This isn't a big deal and working with that assumption means that it's easy to keep the local upstream branch in a state where it only takes fast-forward updates. However, on the debian branch your less interested in the clean upstream development, and instead want to keep a sane history of the debianization work, so on this branch you still do something like a merge -s theirs (though in this case, you probably want to amend the previous version's ./debian dir back in post-merge). So applying the same approach as above with the slight modification, in practice (assuming you're on the clean tip of the debian unstable branch, that'd be):

git branch -m upstream-unstable upstream-unstable-save
git branch upstream-unstable upstream-remote/master
git merge -s ours upstream-unstable
git diff ref-to-be-merged   git apply -R --index --exclude="debian/*"
git commit -F .git/COMMIT_EDITMSG --amend

PS In spirit, this is a similar approach to what's done by the XSF team, though I would argue that with what I'm describing both the approach and the resulting history are a bit cleaner and easier to follow. Also, thanks to ron, mrvn, jcristau, and nthykier on #debian-devel for some quick reviewing of this.

Last week, I was at this event in Bristol where we heard some interesting stories from other creative co-ops. I think the oldest had been running for about the same length as time as our co-op, but with a bit of a smoother history. The organisers have posted their report on the event. What did surprise me is that it seemed that the other tech co-ops weren t as strong about free and open source software. I felt that the links between co-op values of self-help, self-responsibility, democracy, equality, equity, solidarity, honesty, openness, social responsibility and caring for others and the freedoms to use, study, adapt and share were obvious, but maybe I should try to write them out very clearly?

A bug-fix release RQuantLib 0.3.6 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. There are only two changes to two files where an explicit conversion as per Rcpp::as<double> was called for. A Debian archive rebuild had triggered one of those fails to build from source bug reports as the compiler version seems to be more finicky now than when version 0.3.5 was uploaded in November. No other changes were made. Thanks to CRANberries, there is also a diff to the previous release 0.3.5. Full changelog details, examples and more details about this package are at my RQuantLib page.

git-annex is a recent tool allowing to manage files with git without having theirs contents checked into git. I've been looking at it with some interest in the past few weeks in order to use it to handle my music collection on the various computers I use. My laptop just got a new SSD drive which is quite small and cannot handle the whole collection. By using git-annex, I'm now able to only check out part of my music collection and keeping it synchronized with all my others computers. Using git-annex is pretty trivial once you understand the concepts. I've set up a a git repository in my Music directory, like that:

% git init
Initialized empty Git repository in /home/jd/Music

Then I initialized git annex:

% git annex init "Music on keller"

So now I'm able to import all my date into git annex. git-annex offers several backends to store data. WORM is the default, but is not a good choice for file that could be modified, and music files can be, at least when editing the metadata. Therefore, I chose the slow but more reliable SHA1 backend.

% git annex add --backend=SHA1 .

That can be long, so you should be patient. Then you should commit the fact you annexed files:

% git commit -a -m "Imported my music collection"
[master 82d060f] Imported my music collection

Now, it's done. You can go on another computer and clone the repo:

% git clone ssh://keller/~/Music
% cd Music
% git annex get "Incubus - Light Grenades"

And then let the magic happens. It will retrieve the whole directory for you. Do not forgot to commit the fact that you got those album on that machine! The concept used by git-annex is quite simple. All files are replaced by symlinks pointing to .git-annex/SHA1:filechecksum.log. This file contains lines, whose format is:

retrieval-timestamp is-file-present uuid-of-the-repository

Indicating if the file is present in a repository. This logs file are commited into the repository, whereas the real content is stored in the .git/annex directory and therefore not commited. I recommend to take a look to the walkthrough page to get more idea and what you can and cannot do with git-annex.

My server, which is among other things hosting this weblog, is a dedicated server originally hosted by the German webhoster webperoni. It was a bargain at that time, and still a good price, and I was satisfied with webperoni s performance and reliability. Unfortunately, they were taken over by another web hosting company, best known for their server4you brand. They moved my server to a different data center and continued to run it there. Unfortunately, server4you was less reliable and less flexible than my original hoster: For example, when one sector on the hard drive turned bad, they were not able to put in a new drive in my system, let me mirror the content, and then remove the old drive. In another, even worse, instance, my second IP address, which I bought from webperoni, was given to another customer. When I finally noticed and complained, they refunded me the charges that I paid for the IP. But understandably, I wanted to get away from that hoster. The contract is automatically prolonged for one year if not canceled in time, and I missed the deadline last year. This year, I finally got around to rent a slightly better and slightly more expensive server from Hetzner, which whom I already made good experience. I would have preferred to switch when I am back from India, but server4you would not even let me prolong the contract by just two months only a whole year or nothing. So they got nothing, and I started to work on the move yesterday. At first I set off installing the new system from scratch, setting up all the various services and copying over data and configuration. But quickly, I gave up and moved the system by completely by copying programs, configuration and data from the old server. The new system has its root partition on a LVM share on software RAID, and now uses grub2 to boot. I had a hard time to get it booting, and a hard time to debug the issues without access to the physical console. I finally understood the problem by running kvm -hda /dev/sda -hdb /dev/sdb -curses from the recovery system and observing that the initial ramdisk is having problems finding the root partition. The problem was that I thought addressing partitions via file system labels is the way to go now and manually added root=LABEL=root to the Linux command line in /etc/default/grub. But then the initial ramdisk would not activate the logical volume. If I had just left it to the grub package, which inserted root=/dev/mapper/vg-root, everything would work fine. The new IP address (88.198.48.143) has not propagated throughout the Internet yet. Running the following command for a variety of ports on the old machine (80.86.81.20) makes sure nobody notices:

port=443
iptables -t nat -A POSTROUTING -p tcp --dst 88.198.48.143 --dport $port -j SNAT --to-source 80.86.81.20
iptables -t nat -A PREROUTING -p tcp --dst 80.86.81.20 --dport $port -j DNAT --to-destination 88.198.48.143:$port

The only question remaining is: What to do with the old server for the remaining six weeks? I am considering to erase the system, set up a plain Debian system and run a tor/freenet/whatelse node at full speed.

I've been very quiet on here for a few months, and the reason is (of course) because I've been doing other things. No surprise there, I guess! What I've been doing, first of all, is writing aCal, an Android CalDAV client with a lot of help from Chris Noldus. It's probably fair to say that Chris did most of the heavy lifting while I learned Java, did most of the UI & graphic design, and the low-level CalDAV synchronisation and geeky calendaring code. This is now available in the Android market for NZD$2.00 to help us fund further development and in a few days (or a couple of weeks) I'll get the source code up somewhere public (probably Gitorious) and it will be licensed GPLv3... When I get a moment - although refer to title :-) Of course on top of that I'd already committed to a second year of handling the sponsorship job for Linux.Conf.AU, so a fair amount of January was taken up with preparing for, and attending LCA in Brisbane where I took a lot of photos. LCA was, as always, a blast, and all credit to the fantastic team over in Brisbane who really came together when it seemed it was all going to fall apart after the flooding. This seems to be six long months of disasters around this part of the world, although until Tuesday I guess we thought we'd escaped the worst of it in New Zealand. After LCA I snuck home for a few days before flying off in the other direction, to UC Berkeley, for CalConnect XX where I managed to get in some great testing of the new Android CalDAV client, some excellent standards discussions, more photos and (even) a few days sightseeing. I've been home for a week from that now - just barely long enough to hug the kids, really - and today I am in the air once more, off to CeBIT in Hannover where I have been lucky to be sponsored a free booth in the Open Source Lounge by Linux New Media AG. My original travel sponsor slipped through the cracks, but I was fortunate that InternetNZ agreed to help fund me instead. If you're coming to CeBIT please do stop by the Morphoss table in the Open Source Lounge (Hall 2, F40) for a chat about DAViCal and Android - and anything else you want to talk about! I'll be there all week :-) And if I have some quiet time in that week (yeah, right!) I'll be trying to push out a new DAViCal release, which is overdue really, since the last one was two months ago.

I ve recently got a new Xperia X10 Android phone for my with with Virgin mobile, it s generally been working OK although I am having some issues [1], I ll write another blog post soon about other problems I ve discovered with the phone and how I ve solved some of the previous ones.I upgraded my wife s phone first because I can t be without Nagios SMS messages if things don t work. So now that things are generally working I want to get myself an Xperia through Virgin (and have my wife s phone get the Nagios SMS in the mean-time). But since last Friday the entire Virgin sales infrastructure has apparently been down. It started with just declining my attempts to purchase a new phone on a separate account, but when I decided to add a second phone to my wife s account the web site told me that they are upgrading their CRM system and it should be fixed on the 22nd of Feb (yesterday). The web site is now saying that I should check back in 2 hours for an update , it s been saying that for a couple of days now.So for most of a week potential Virgin customers have been turned away. It could be that Virgin stores are processing sales on paper, but they offer some significant discounts for web sales the plan I want is a $39 per month plan and I ll get 3 months free for buying on the web. I m not about to visit a store and lose $117! I m sure that many people are losing confidence in Virgin and taking their business elsewhere. I have only just installed the 3G Watchdog free Android app that monitors bandwidth use and automatically turns off 3G when the quota is reached. For the first few days of using the phone which were more data intensive than usual I had no monitoring and no way of using the Virgin web site to discover how much was used. If Virgin bill me extra for data use I ll complain and demand that they alter the bill.This is even worse for pre-paid customers who can t add credit to their account while this happens!Virgin state that they are upgrading the hardware, operating system and database our platform uses to ensure that we can service our Members even better for years to come [2]. I wonder how people get themselves into such a mess. I guess they didn t have a decent test environment to allow testing the upgrade process before doing it on the live data, I can understand a routine small upgrade going wrong and corrupting data in a way that takes some time to resolve. But when everything is upgraded then everything should be tested, and tested before going live! One thing that Virgin could do to regain some credibility is to publish what went wrong and what they learned from it. I would be much more happy to trust my personal data and my business critical phone to a company that learns from it s mistakes and publishes plans on how they do better than one that just does PR.According to Whirlpool they did the same thing on the 25th of January, so they have had two outages of their billing/CRM system in two months! [3]. The Whirlpool thread has discussion about last month s down-time and this month s down-time.On the up-side, Whirlpool user Kevin JD is a Virgin representative who advised customers to send email to telesales@virginmobile.com.au with a contact phone number if they have any issues. It s good to see a company engaging with it s customers.Update: An hour after emailing the URL to this post to Virgin I got a call from a service representative. It wasn t a very productive call as I already knew that their servers are down and they can t do anything. But it s good to know that they are very enthusiastic about making things better. One useful thing that I learned is that my wife s service is probably on a pro-rated bandwidth quota. As she got the phone in the middle of the month we can only do 100M of data transfer not 200M before the end of the month, by my rough calculations I ve downloaded well over 50M of data (maybe as much as 100M) from the Android marketplace. So it might be necessary to negotiate about the bill as soon as their CRM system works.

• [1] http://etbe.coker.com.au/2011/02/18/xperia-x10i-first-experience/
• [2] http://www.virginmobile.com.au/upgrade
• [3] http://forums.whirlpool.net.au/forum-replies.cfm?t=1626553

There are many ways to run some commands simultaneously on multiple hosts like cssh or dsh. They come handy for example when you are installing software updates on a set of hosts. dsh is a rather simple comandline tool allowing to execute a command over ssh on multiple hosts. However it doesn't allow any interactive input -- so you can't look at the potentially upgrading packages and press y to accept and you can't go through debconf promts or similar. This is solved by cssh which opens a XTerm for every host and a input area that is broadcastet to all of them. this is working really well -- you can execute your update on all hosts and still do individual adjustments just as needed: switch focus from the broadcasted input to one of the terminal windows and anything you type just goes there. Now cssh has a big disadvantage: it requires a running X server (and doesn't do too well with a fullscreen windowmanager). Requiring X is quite a blocker if you need to run that ssh multiplexer on a remote host, for example if the firewalling doesn't allow direct connections. Fortunately you can make tmux behave as we want -- in a simple terminal: First you need a script spawning the ssh sessions in separate tmux panes and direct input to all of them -- here called ssh-everywhere.sh (you could also write a tmux config I guess):

#/bin/sh
# ssh-everywhere.sh
for i in $HOSTS
do
  tmux splitw "ssh $i"
  tmux select-layout tiled
done
tmux set-window-option synchronize-panes on

Now start the whole thing:

tmux new 'exec sh ssh-everywhere.sh'

And be done. Update If you want to type in just one pane (on one host) you can do that as well: C-b : set-window-option synchronize-panes off and moving to the right pane (C-b + Arrow keys)

Nokia is in the news for it s CEO announcing that they have a Burning Platform , Jeff Waugh gives an interesting analysis of the situation [1].What is a Platform for Phones?This whole scenario makes me wonder about what a platform should mean in terms of mobile devices. Wikipedia says A computing platform is some sort of hardware architecture and software framework (including application frameworks) that allows software to run. Typical platforms include a computer s architecture, operating system, programming languages and related user interface (run-time system libraries or graphical user interface). [2]I think that Nokia s problem was that it didn t leverage it s strength in making great phone hardware. Nokia used to have a commanding share of the market due to simply making better phones. They had phones that were upwardly compatible (the same UI and used the same chargers), their phones were always very solidly constructed, and they had some good design features such as having the battery comprise part of the case which allowed a replacement battery to be physically larger for longer battery life. Even now they have hardware features which are better than the rest such as the N8 which is widely regarded as having the best camera of any phone. Nokia also did some smart things like releasing their N800 series of phones with more open software than most companies used, they supported running arbitrary Linux applications.While they kept the platform as the API interface on a phone OS and maintained several different OSs (with multiple versions in production at any time) they made a less desirable platform for developers and a lot more development work for themselves.Nokia fragmented their own market. There is no technical reason why they couldn t manufacture a phone that is capable of running a choice of their own OS as well as Android (or alternatively their own OS or the Windows Phone OS). There may be technical issues that prevent designing a phone to run either Android or Windows phone, but they could have got two out of three options available.On top of a hardware phone platform we could have multiple software platforms, Android, Windows, Meego, etc. The users could then decide which one to use.A Hardware Platform for PhonesI think that it would be ideal if a phone company offered phones with a choice of OS and allowed the user to change the OS. The N8 is a great phone and the hardware is really appealing, I would have definitely bought one for my wife if it ran Android. Also the resale value of phones can be improved if they can be re-purposed. If someone knew that people on ebay were paying good rates for an old phone because it could be re-programmed then they would be more likely to buy it. As phones are essentially free for most people due to telco deals, a phone that has a resale value of $200 is a lot more desirable than one with a resale value of $100 if all other things are equal.I would like to see the mobile phone as a platform just as the IBM PC compatible was a platform. Imagine if IBM had sued Compaq into oblivion when they first cloned the PC, if that had happened I don t think that I would be using an IBM PC (Thinkpad) right now. The early IBM PCs really weren t great systems for home use, for pretty much anything you might want to do the early PCs didn t compete well with CP/M systems and systems from Commodore and Apple. It was only when cheap clones flooded the market that the PC platform took off. Modern PCs can boot from disks that were created 20 years ago. At every time during the PC s history there has been a choice of OSs available that work with the standard BIOS boot loader and expect certain essential hardware (such as a BIOS interface for video, keyboard, and storage). This choice increased the market allowing economies of scale in production. It also allowed competition between vendors which forced the margins to be low and resulted in PCs becoming cheap over the last 20 years every aspect of a PC apart from Intel CPUs and Microsoft Software has steadily dropped in price.I would like to see phones designed with a common boot loader, a common core set of CPU functions (I believe that with ARM CPUs a lot is optional), and common interfaces to the touch-screen and the GSM system. The boot loader would ideally have support for booting multiple OSs, in a typical end-user case the store would provide the phone configured to quietly only boot one of the available OSs but it would be trivial to offer the user a choice of OS at boot time for advanced users. The process of installing a custom ROM image in a phone is regarded as dangerous at the moment and most people who would like to do such things are hesitant due to the risk of bricking their phone. If booting the last good OS was an option then more people would try new OSs. With the current situation I am a little hesitant to re-program my new phones (and I m braver about such things than most people).One of the many problems with the current situation is that the phone vendor has to officially support every phone OS. One example of the problems with this is the Sony Ericsson Xperia X10 which is a fine piece of hardware that has been saddled with an older version of Android for it s entire life, Sony has stated that they won t produce any more updates and thus the demand for this phone is a lot lower than it might be. I believe that this is why the Xperia X10 is significantly cheaper than any other phone with similar features. Sony is losing money because they have a closed hardware platform and don t support the recent versions of Android!Ways of Booting the OSIt shouldn t be a technical challenge to have support for basic touch-screen IO and 3G Internet access in the boot loader of the phone. Then an OS such as Android could be loaded from the network before being booted without needing an older version of Android running. 80386 systems with a few megs of RAM used to be able to boot from a LAN, a modern phone has much more powerful hardware and a network connection that s at least as good as 10base2 networking so it should be able to do the same.Modern phones have micro-SD cards for local storage. There s no reason why the OS couldn t be loaded on micro-SD card. Fixing the OS on a phone that was bricked could be achieved by putting the micro-SD card in a PC and then copying the files across just as I often fix PC OS problems by installing the hard drive in a working system.Changing the OS of a phone should be easier than changing a PC between Ubuntu and Fedora (the Linux distributions that seem to have the most usage by less technical users).ConclusionIf Sony had made their platform the hardware then they could have spent their effort on what they do well (making great phone hardware) and delegated almost all of the work related to creating the OS to other companies (Google, MS, telcos, etc).If the entire industry moved to a hardware platform for phones then the result would be lower prices and some thin margins for manufacturers. This would be good for users and OK for manufacturers. The current situation however is quite bad for users who can never get phones to do exactly what they want and not so good for phone manufacturers such as Nokia (and everyone else who does business with Microsoft) who find themselves on the wrong side of market forces. The phone as a hardware platform shouldn t be too bad for Microsoft either, they did OK on the PC hardware platform.

• [1] http://bethesignal.org/blog/2011/02/11/elopocalypse-nokia-chooses-microsoft/
• [2] http://en.wikipedia.org/wiki/Computing_platform

In Ubuntu, the use of ptrace is restricted. The default allowed relationship between the debugger and the debuggee is that parents are allowed to ptrace their descendants. This means that running gdb /some/program and strace /some/program Just Works. Using gdb s attach and strace s -p options need CAP_SYS_PTRACE, care of sudo. The next most common use-case was that of crash handlers needing to do a live ptrace of a crashing program (in the rare case of Apport being insufficient). For example, KDE applications have a segfault handler that calls out to kdeinit and requests that the crash handling process be started on it, and then sits in a loop waiting to be attached to. While kdeinit is the parent of both the crashing program (debuggee) and the crash handling program (debugger), the debugger cannot attach to the debugee since they are siblings, not parent/descendant. To solve this, a prctl() call was added so that the debugee could declare who s descendants were going to attach to it. KDE patched their segfault handler to make the prctl() and everything Just Works again. Breakpad, the crash handler for Firefox and Chromium, was updated to do effectively the same thing, though they had to add code to pass the process id back to the debuggee since they didn t have it handy like KDE. Another use-case was Wine, where for emulation to work correctly, they needed to allow all Wine processes to ptrace each other to correctly emulate Windows. For this, they just declared that all descendants of the wine-server could debug a given Wine process, there-by confining their ptrace festival to just Wine programs. One of the remaining use-cases is that of a debugging IDE that doesn t directly use ptrace itself. For example, qtcreator will launch a program and then later attach to it by launching gdb and using the attach command. This looks a lot like the crash handler use-case, except that the debuggee doesn t have any idea that it is running under an IDE. A simple solution for this is to have the IDE run its programs with the LD_PRELOAD environment variable aimed at a short library that just calls prctl() with the parent process id, and suddenly the IDE and its descendants (i.e. gdb) can debug the program all day long. I ve got an example of this preloadable library written. If it turns out this is generally useful for IDEs, I could package it up like fakeroot and faketime.

In case you missed the news on webkit-dev, Collabora has been working on developing a clutter port of WebKit. It shares the build system with EFL, and most of the backend code comes from the GTK+ port. That means networking is handled by soup, drawing by cairo, multimedia by GStreamer, and so on. If you d like to give it a try, you can clone the repository from gitorious: $ git clone git://gitorious.org/webkit-clutter/webkit-clutter.git Then to build it you use cmake. From inside the source code directory do this: $ mkdir build
$ cd build
$ cmake .. -DPORT=Clutter -DSHARED_CORE=1 -DBUILD_MX_LIB=1
$ make
The BUILD_MX_LIB option is optional it will build what we call the Mx toolkit library in addition to the vanilla one. Then you can test that the library is built and working by running the programs inside the Programs directory. Enjoy!

I have just got a new Xperia X10i for my wife. It s with Virgin who use the Optus network. Optus apparently have better coverage than Three (who we have been using for 6.5 years) and Virgin offers the best deals they have good phones on low monthly rates and do nice things like selling most phones unlocked (the Xperia is one that comes unlocked).I chose the phone for use in system administration work. A large screen should be good for reading email and running a ssh client. Hopefully I will be able to use the phone s Wifi to provide mobile net access to a laptop for any problem that can t be solved on a tiny screen. My wife doesn t do much sysadmin work nowadays, but it ll save some effort if we can both use the same model of phone. Also according to Amobil.no the Xperia X10 is the second best phone for camera-phone functionality [1], and my wife would really like to take some good pictures with her phone (she loves the LG U990 Viewty that she has right now).ScreenOne major reason for choosing this phone is that it has the biggest and highest resolution of any phone that doesn t cost a huge amount of money. It does well in this regard, the screen is very good to look at. However it is apparently not particularly sensitive and we still have the protective cover on the screen which makes the swipe to unlock operation quite difficult. My wife generally likes to keep the protective plastic on such screens for a while to prevent damage, but she may take this off sooner than on other devices.Sony FoolishnessOne reason that this phone is cheaper than any other phone with similar specs is that it s made by Sony. Sony won t provide updates to Android 2.2 or better and they have their own stupid applications installed in an intrusive manner. The main screen of the phone has a large icon for the Sony Timescape program. If you choose a background image of a person s face then the Timescape logo is in the ideal location to obscure the person s eyes. Generally it s not desired to have the person in your phone background look like Zorro. Fortunately you can delete icons from the home screen, you just have to press and hold the icon until a trash-can appears at the bottom of the screen and then drag it to the trash.Phone BookThe core phone functionality is fairly important for a mobile phone. But it seems that every modern phone has some serious failings in this regard. The Xperia comes with a phone book that is reasonably good for managing contacts and partially coalesces multiple entries with the same name which is convenient as I somehow managed to import 3 copies of each entry in my wife s phone book. The down-side is that the multiple entries are still stored, so when it comes time to update someone s phone number my wife will have to separate the entries and then delete all but one of them, adding probably a minute of effort for each one. With 60 phone book entries that s probably an hour of wasted time during the course of her phone use.They really should do a time based analysis of phone use and optimise the UI for efficiently performing common tasks. I think that an ideal design would start off by asking the user to rate the importance of phone functionality, camera, facebook, twitter, etc and then give a UI that matches the important tasks. A device that has 384M of RAM, 1G of built-in Flash storage, and an 8G micro-SD card (which Virgin provided with the phone) should be able to offer a selection of UIs.CameraThe configuration for the camera allows some settings to be changed easily and others have to be changed through the advanced settings which requires one more button press and a few extra taps on the screen. One real stupidity is to have the phone light be in the advanced settings, so if the room happens to be a little darker than expected you suddenly need to go to a lot of extra effort to turn on the light.Finally it would be really good to be able to take a quick picture. I would like to be able to be able to press a combination of buttons and have it immediately take a picture regardless of what had been running before. Not press the main menu button, the camera icon (which I made a link from the home screen), wait a second for it to load, and then press the shutter button. Ideally I would like to press a combination of hardware buttons to have it immediately take a picture even when the screen is locked. I don t want to wait for the screen unlock process, sometimes the perfect shot is only available for a second.WifiIt doesn t support ad-hoc networks. This is really annoying for me as I don t own an access point and I don t plan to buy one just for a phone. My laptop (Thinkpad T61 with Intel Corporation PRO/Wireless 3945ABG [Golan]) doesn t seem to support running as an access point, and my EeePC 701 with an Atheros AR5001 has the same problem, here is the error message:# iwconfig wlan0 mode master essid x
Error for wireless request Set Mode (8B06) :
SET failed on device wlan0 ; Invalid argument.I have a cheap Realtec USB Wifi device that claims to support being an access point but just fails when I try it. So presumably it doesn t really support it but the driver is buggy.It seems that I may need to put a custom Android installation on it to get ad-hoc working.Also it s apparently necessary to root the Xperia to get Wifi tethering to work for providing net access to a laptop. So I guess I ll hacking it soon.Internet AccessThe Virgin web site couldn t help me set up the Internet access because I ported a phone number from another Telco. I had to use the phone number supplied by Virgin to login to their web site and the site then wanted to sent (probably SMS) the settings to my phone, but of course it couldn t SMS to that number as I ve got the Three number active. Presumably they were expecting that customers would setup Internet access before porting their old number.I phoned the Virgin tech support and used the settings they suggested, but it didn t work. Then I took the phone to a Virgin store and it started working after they rebooted it with no changes to settings. Just like Windows used to be. :(Email Access Default AppThe email setup defaults to port 143 IMAP access, so even when a mail server provides access on port 993 for IMAPS and on port 995 for POP3S probably most users will not use any encryption because their phone doesn t default to it. Also one annoying feature is that when I gave it the email address test@coker.com.au it decided to use test as the user-name for IMAP by default. It seems that a very large portion of the active email accounts nowadays are on systems that handle multiple domains and thus require the domain name in authentication, and a large portion of the remainder accept domain names even when they aren t required. Besides it s a lot easier to delete a domain name when it s not needed than to type it in when it is.The default hostname for sending and receiving mail is the domain name, if the MUA configuration program had checked the MX record it would have known that smtp.sws.net.au is a better default choice for the mail server. I ve now started the practice of creating a CNAME entry named m in every domain I run, so when someone is setting up a mobile phone they just have to add m. to the start of the default hostname to use for sending and receiving mail. By doing this I can probably save an average user at least 30 seconds of typing, which is particularly important if I happen to be talking them through the process!The MUA uses IMAP commands to load the text of the message without attachments and then loads attachments on demand, this will be handy for when people attach unreasonably large files. With IMAP caching and partial loads the core functionality seems OK, and some quick tests of standard but not commonly used IMAP functions such as noticing that mail has been deleted by another client seemed to work well. The performance of loading a new mailbox with 369 message was quite poor, I m not sure how much of this is due to the network connection and how much is due to the client implementation. This is a real concern for me as I typically have about 7,000 messages in my IMAP account and 3,000 unread messages most of which are in two folders.Another problem is the fact that there seems to be no way of searching in the list of messages, sorting/grouping by subject or sender, or showing unread messages. This makes the Xperia unsuitable for the task of reading mailing lists when on public transport. I would really like to be able to sort through a folder of mailing-list mail, delete messages that I don t need to keep and flag ones that need attention when I have the resources of a desktop system when I m on a tram or in other situations where a laptop can be inconvenient.One undeniable bug I discovered (as opposed to the missing features I listed) is that when a message has a line with a URL followed by many spaces it won t load correctly in the web browser, the spaces will be treated as part of the URL. I ve changed the default signature in my email to have a description followed by a URL to avoid triggering this bug. Presumably other Android phones are bug for bug compatible. :(Moxier MailThere is a complete set of email and calendaring applications preloaded from Moxier.com. The Moxier mail client uses company.com as an example domain name (they should use example.com does no-one read the RFCs?). It s nice that they verify the SSL certificate and display the details when it s not signed.Unfortunately Moxier doesn t support anything other than Exchange, and I can t delete it either as it s part of the base set of applications from Sony.SMSThe SMS application is supposedly conversation based. This doesn t seem like a benefit to me, it s a pity that there is no option to make it act more like every other SMS implementation I ve used over the last 12 years.BatteryThe battery is described as having a capacity of 1500mAh, for comparison a AA rechargeable batteries have capacities ranging from 1800mAh to 2800mAh. My first mobile phone had a rechargeable battery that took the same space as 4*AA batteries and the phone was designed to run on AA batteries if the main battery wasn t charged. I would like to have an Android phone that had a battery pack of similar size to at least 2*AA batteries, and 4*AA would be OK too.The Xperia battery was reported as being half discharged after playing with the phone for less than an hour. Basically when doing anything with the phone that is remotely intensive I need to have it plugged in. Unless of course I m using one of the star-chart applications LightsThere are two white leds in the gaps between the keys for Settings, Main-Menu, and Back on the front. They are quite bright and annoying to look into at night.SSHSSH was one of the reasons for getting an Android phone, but it was one of the last things I worked on because everything else was so much fun. I m currently using ConnectBot, it supports storing ssh host keys, generating a key-pair from random screen touches, locking and unlocking the key-pair (with a pass-phrase) and pasting the public key to an open window. All the basic functions just work although it s rather inconvenient to enter the CTRL keys.The next thing to investigate is a small and light Bluetooth keyboard. Using an external hardware keyboard will save me from the pain of entering CTRL keys (which is even worse than the pain of entering passwords that contain digits and mixed case). It will also allow me to type a lot faster and use the entire screen for the ssh session.Current Bug SummaryWifi doesn t do ad-hoc mode or tethering.MUA doesn t correctly handle URLs in messages if many spaces follow the URL and is also unsuitable for mailing list mail.Sometimes requires a reboot to make Internet access settings take affect.Camera light can t be easily turned on and off.This isn t too bad.ConclusionMost things do what I want. Once I get a MUA working with all my email, Wifi doing what I want, and a Bluetooth keyboard it ll be a fairly ideal mobile sysadmin terminal. I ll probably order one for myself this weekend.

• [1] http://www.amobil.no/artikler/norges_beste_kameramobil/79081

Shock news: a Tasks release! Announcing 0.19:

• Lots of translations
• Fix i18n in the about dialog

Yeah, it's all go on the Tasks front... Tarballs on the Pimlico site, or gnome.org. NP: Yanqui U.X.O. - Godspeed You! Black Emperor

As you know, I very often advocate using the bycicle as the main means of transportation in Mexico City. The city is very apt for biking through it, and contrary to the fears of mostly everybody, the city is neither aggressive nor as dangerous as people say. However, I have seen cyclists which seem to be looking for the best ways to be hit by a car, or to hit a pedestrian. Yesterday, I read this note in La Jornada: The local Environment Secretary calls cyclists to respect vial regulations. Many cyclists assume they are a special case of pedestrians, and zig-zag as they please between the road and the walkway, or just stay in the walkway. That is dangerous, both to pedestrians and to themselves. You might find children, elderly or motion-challenged people on your way. Also, not only will pedestrians only expect other people in the walkway, moving at their pace or slightly more, but cars will also not expect somebody moving at 10-20Km/h. A car driver pulling out of his garage, or crossing a street, will not have enough warning when he sees you, and you are very likely to end up in an accident. Since ~5 years ago, Mexico City is growing a grid of Metrob s and confined troleb s cero emisiones (trolleybus) lanes. Many cyclists use those lanes That is VERY dangerous. Public transport vehicles are large, have a lot of inertia, and will take longer to react to finding you ahead of them. Besides, they can go way faster than a regular bike (~60Km/h for metrob s, ~40Km/h for troleb s), and have to stop every couple of blocks. So, you will be uncomfortable if trailing them, and you will be a liability to ~100 people if you go ahead of them. Besides, it is illegal to drive in the confined lanes if you are not a public, semimassive transport vehicle! Surprisingly many people have argued they prefer riding their bicycles against the traffic I think they prefer staring at Death into its blue, glowing eyes (or into its long, thin whiskers)... By far, most cars that hit a bicycle do so from the side, when crossing a road. And if you arrive at a crossing from the wrong way, the way a driver does not expect you, don't expect the driver to be aware of you. Also, in the much less likely event of a car running into you, would it be better to be hit at 80Km/h (60Km/h of the car plus 20Km/h of your own speed on a full frontal crash), or at 40Km/h (substracting instead of adding)? Yes, some people say that looking at the car will allow you to maneuver How far in advance would you know a car coming from the front will hit you? One second? That's 22 meters at 80Km/h (again, if you realize the 60Km/h car is heading straight to you, at 20Km/h). Too short notice for you to do anything Any maneuver will most likely end in an accident. And the driver would not be so much to blame, as he would not be anticipating you riding against the traffic. Make sure you get seen. By night, always use proper lights (red on the back, white on the front, and reflective material to the sides). Day or night, wear bright, reflective clothes (or over-clothes material). Act in a predictable fashion. Remember you are riding a vehicle and are subject to the same rules any driver is A cyclist is not exempt from driving correctly! Do not jump red lights. Never ride on the walkway. Do your best to enjoy the ride. And ride. Yes, ride, take the streets, enjoy the streets. But don't attempt to drive the traffic out of it The streets belong to us all, and we can all share them. PS I also saw this note in the same paper: Sunday Rides in "Campo Militar N mero Uno". The main military field was open as a park this weekend! I have to make sure it is regularly open I am definitively going there!

For a few months now I've been using SD for tracking bugs in many of my personal projects. SD is a distributed bug tracker, and I believe such systems to be the future, just like distributed version control has become the obviously correct choice. Distributed bug tracking is still in its infancy, and the tools are still developing. However, SD, at least, is already usable, at least for the small kinds of projects as mine are. I have no idea, at least yet, how it scales up to large projects with many developers. Life with SD is similar to life with a DVCS. You have your bug repository, and you make changes there as find bugs, fix them, and so on. When it is suitable and convenient, you synchronise your repository with others. I mainly sync to my server, so that others can sync from that if they wish. (I don't know if anyone does that.) There are several distributed bug trackers. I chose SD because it seemed the most mature one, but also because it does not tie the bug repository with the source code version control tree. This is quite important, I think. If there's a bug in the project's website, then that does not need to be committed to the project's source code. If I work on a bug in a separate code branch, I would like to make any necessary notes about it in a central bug repository for all the branches on my laptop. This is helpful, for example, if I have to back out of my code changes, because I did something stupid, but want to keep the notes I took in the bug repository. SD is not without its weirdness. The command line syntax is a bit un-Unixy, and it requires setting an environment variable to specify the location of the bug repository. (I wrote a little wrapper script to set it: all my branches for project foo are in ~/foo, and the bug repository is in ~/foo/bugs.) It could perhaps also be faster, but it's not exactly slow, either: it is a joy to be able to just run sd ticket close 123 rather than open a web browser, navigate to the right bug, and click close there. Also, once you have a bunch of projects, each with its own bug repository, it becomes an effort to synchronise each of them. I'm hoping mr will someday support SD, but until then I'm using a little wrapper that syncs my SD repositories to my server. I also have a third wrapper that allows me to run an SD command on each repository, so that I can, for example, easily get a list of all open bugs in all projects, which is helpful for my GTD weekly review.

Reading job listings for any sort of IT job is depressing. It s been quite some time since I ve had to do that, but how many times have you seen something like this:

• 5 years of Java experience required.
• 3 years of Java experience with modules X, Y, Z required.
• 6 years of experience administering Linux machines running RHEL 4 on a Windows 2000 domain with 1500 clients in an educational setting preferred.

I could go on and on. As a job seeker, that sort of thing is fundamentally devaluing to someone who has strengths in being adaptable and quickly learning new tools, languages, or even entire environments. As an employer, it sends a message that you re not interested in more than a surface look at someone s strengths, and probably don t care to hire the best and the brightest. After all, would you turn away a rockstar programmer simply because he or she had been writing filesystem code in C the last 3 years instead of the latest whizbang Java web widget that will probably be obsolete in a year and unsupported in two? I am quite certain that there are plenty of managers that do. Even if you are a company large enough to have an entire team of people that do nothing but work on that whizbang app, don t you still want the best you can find, realizing that some of the best people to work on that app may not have even heard of it yet? (And that when the app goes obsolete in 5 years, you d rather not have to lay off a large team of single-skill people) Some of you may know that I work in IT at a manufacturing company. We have a small IT team here, about seven people, and are a heavy Debian shop. And we have a vacancy open up in our development/Linux admin group. I m the manager of that group, which is why I m thinking about this right now. We re too small for single-subject specialists to make sense, yet we re big enough to appreciate skill, experience, flexibility, and rigor. Consequently, when the occasion arises for me to look for new employees, I don t prepare a laundry list of things we use in-house and would like experience with. The list of almost-required things generally begins with Linux and ends with experienced , and has nothing else in between. In other words, I d like it if I don t have to explain to you what a symlink or a hardlink is, but I d be willing to do so if I think you d internalize it quickly. On the experienced side, it would be nice if you already have a well-developed sense of fear at running rm when you re root, or have designed a storage infrastructure for a network before, or are paranoid about security. But again, if people can pick up those traits on the job, we are usually still interested. If learning how to package up software for Debian, fix bugs in software you ve never seen in a language you ve never heard of, raise good questions about things you may not have lots of experience with, and write documentation for it all on a wiki sounds like fun, then that s probably the kind of person I want, even if you ve never used our particular tools before. If I were to judge based on the stuff I normally see in job postings, I guess you might conclude I m nuts. I don t think I am, but then again I m also the only person I know that formats his own resume in hand-crafted LaTeX. What do you all think? The next question is: how should one evaluate candidates given this sort of philosophy? I m not a fan of canned tests, or even whiteboard tests that tend to be some sort of canned topic that may test the applicant s specific knowledge base more than overall skill and flexibility. Similarly, as an applicant in years past, I ve struggled with how to present the I ve never used $LANGUAGE, but I know I could pick it up quickly and do it very well vibe. To certain people, that might sound like BS. To the more geeky managers, perhaps it sounds like what they want. We ve built a fairly diverse team on the back of this approach, and it s worked out well for us so far. I m interested to hear your thoughts. Oh, and if you d like to work for us, you should probably be sending me an email. No, I m not going to list the address here on this blog post. If you can t figure it out, I don t want to hear from you <grin>

Those of you that watch my twitter know that we got dumped on with snow recently. The final estimate reported by local papers is 17 to 19 inches. That s my car. You might understand why I went home at 1PM Tuesday and worked from home the rest of the afternoon. The county didn t get the roads opened until 5:30PM Wednesday, so we were stuck. Terah s dad kindly opened the driveway for us though. Without that, we wouldn t have even known where it was. Jacob and Oliver loved playing in the snow. It was a HUGE hit for them. Both of them hated to go back inside when it was time. Oliver had trouble walking, or even standing up, but that didn t stop him from begging for more every chance he got. On Saturday, I took Jacob outside to play. We had in mind building a snow tunnel. We went to one of the big piles of snow left by the tractor after our driveway was cleared, and I dug out a space underneath it that Jacob could crawl through. He got a little scoop (he insisted on calling it my shovel ) and helped dig too. He had a great time with it. He was very anxious to go inside, and it probably didn t help his eagerness that the cats could freely walk through it long before he could. They were around and taking a great interest in the proceedings. One of then stretched out on my back while I was stretched out pulling snow out from the pile. Finally Jacob got to go through. He had a little trouble because it wasn t tall enough for him to crawl like he usually does (though still plenty big for him to get through with lots of room to spare), but he came out beaming. I then went and got the camera and got some photos of his second trip through. You d think that this might be the highlight of a 4-year-old s snow excursion. And given his excitement, you might be very convinced of that. But you d be wrong. What Jacob really wanted was a snow train. So I drew some train track-sized lines in the snow with my shovel. Then I was supposed to draw a train engine. He had been planning this for hours before we even went outside, and I knew very well what was supposed to go on the engine: a snowball for a headlight. We then added passenger cars and a coal car, because apparently that s what you do with snow trains. And the result: When we went inside, Terah asked Jacob about his snow experience. Jacob was very happy, until she asked him if his snow train was a real train or a drawing of a train. Then he became very sad when he said it was a drawing of a train, and decided it was not what he wanted AT ALL. So today we went out again. This time I built a snow train that stands up, complete with snowballs for wheels. I asked him if it was right, and he kept saying, No, I want a REAL train. Finally I asked him what the difference between a real snow train and this one was. Apparently a real train would be on tracks, move around, make noise, and take us places. Fortunately he didn t dwell too long when I explained I couldn t build one of those. And then today s highlight was writing JACOB in the snow. He had started by drawing larger and larger J letters with his arm, and finally asked me to make one. I made footprints in the shape of a very large J, which he loved. He had me make another one, and then the rest of the letters in his name.

A few years ago, I mentioned my dad's fight with cancer. He survived that, and has carried on for several more years since. He had a replacement hip fitted last year to improve his mobility, and was looking set to enjoy a good long retirement. Until... Just before Christmas symptoms suggested that the cancer had returned, and with a vengeance. Tests confirmed it. We were hoping that Dad would fight on and win through again this time, and I was looking forwards to him being at my wedding this September. But there have been a lot of unexpected complications this time and it's just not going to happen. I spoke to him last on Tuesday this week, but he was delirious at the time. He hasn't really been conscious in any meaningful way since. I'm composing this message in the hospital in Southport where I'm now waiting for my dad to die. We've been told by his doctors that it's not going to be long now, but they said that on Wednesday too. My dad has always been stubborn, always a fighter; we know that this is a fight he can't win, but that isn't stopping him from trying his hardest. At least he's being kept comfortable for his last few days, with plenty of strong painkillers on hand when needed. Please let it end soon.