Due to some real life impediments (including a beautiful one with just over 2 months of life called Daniel), I will not be able to go to this year's DebConf, even though it seems that this DebConf will exceed many expectations with the support that the governement of Nicaragua just offered. Things that I would like to have worked on Unfortunately, this means that I will not be working with some fellows that may be there, like people from the pkg-multimedia team. In particular, as I am working on some packages that are not that well suited to distributions (especially regarding HandBrake), it would be highly productive to have a hacking session or two with other people interested in having this popular and easy-to-use video ripper/transcoder on Debian. I would also like to be able to talk in person with other people (including Joey Hess) about controversial packaging of some programs, like GNU Parallel, which I maintain in Debian. There are many things else that I would love to work on, besides helping a bit with some of the still pending transitions of the the Debian Fonts Task Force umbrella, meeting friends and other social stuff. Oh, well. I guess that working at home will be the solution for some time.

• Latvian is now completed.
• Lithuanian is now completed.
• Galician and Welsh completed sublevels 1 and 2

At this very moment, it means that I would deactivate 14 languages:

• Amharic (5f17u in sublevel 2)
• Croatian (16f19u, 6f9u)
• Georgian (17f47u, 10f21u)
• Kurdish (12f25u, 6f22u)
• Malayalam (5u in sublevel 2 only!)
• Nepali (16f54u, 10f21u)
• Norwegian Nynorsk (13f34u, 13f23u)
• Albanian (19f50u, 13f23u)
• Tamil (2f1u, 1f)
• Tagalog (18f54u, 24f39u)

A few days ago I reported how to get the support status out of Dell using an unofficial and undocumented SOAP API, which I since have found out was discovered by Daniel De Marco in february. Combined with my web scraping code for HP, Dell and IBM from 2009, I got inspired and wrote a web service based on Scraperwiki to make it easy to look up the support status and get a machine readable result back. This is what it look like at the moment when asking for the JSON output:

% GET https://views.scraperwiki.com/run/computer-hardware-support-status/?format=json&vendor=Dell&servicetag=2v1xwn1
supportstatus( "servicetag": "2v1xwn1", "warrantyend": "2013-11-24", "shipped": "2010-11-24", "scrapestamputc": "2012-06-06T20:26:56.965847", "scrapedurl": "http://143.166.84.118/services/assetservice.asmx?WSDL", "vendor": "Dell", "productid": "" )
%

It currently support Dell and HP, and I am hoping for help to add support for other vendors. The python source is available on Scraperwiki and I welcome help with adding more features.

Exams period has almost come to an end, only one more to go, tomorrow morning. Thinks were quite busy during this period and my time was fairly limited. Multitasking did not go as well as I had expected, will consider this in the future.


I've spent the last days reading documentation, code and tutorials, which were very helpful, but I did not manage to do much. The Go Back button (#636211) is now done but there are still some fixes that need to be done on the link timeout (#610752). That one took me soooo long, and it's not like there was much to do, I don't know exactly what the problem was :).


One sad way to describe the way thinks worked during this period is by using a quote: "progress coming slowly, if it comes at all". That's from Lord of the Rings, but it's a bit modified (s/progress/change). Ok, that's overreacting. I do feel that I have made some progress, it's not necessarily visible (as in patches, features) but I feel much more confident about what I am doing.


Short post, now back to study. Programming Paradigms (last exam), here I come :))

Today, as an experiment, I attempted to fix a Debian bug while on the train to work. I use a 3G card from Three.co.uk in my Lenovo Thinkpad x121e, and my commute is from Southampton Central to Fleet (changing at Winchester) - just under an hour. 3G coverage is not 100%, but tends to be better around the major stops.

• First, I found a bug. I used udd.debian.org to browse for a relatively simple RC bug, and found bug #674992 in actionaz. The fix was outlined in the report already, so there was very little thinking required.
• Next, I confirmed the FTBFS using cowbuilder. Unfortunately, this required downloading roughly 120MB of dependencies - I have 1GB of data per month, but I couldn't afford to do this every day. I was lucky in that I was near Basingstoke at the time, so had a good HSDPA signal to get the bulk of this. The build had failed before I reached Fleet.
• In the background, I updated debian/control and debian/changelog with the fix. I was able to set off the build, but had to suspend the laptop until lunchtime before it could finish. Cowbuilder needed to download only a few extra build-deps, as the vast majority were cached from the initial run.
• On the train home, I checked over the result, signed it and uploaded. In this instance, the built package was small enough to upload, but I could see this being a problem with others.
• Finally, I sent the nmudiff, although that was delayed briefly by a drop in connectivity before Southampton Airport.

Thoughts: firstly, part of me is amazed that this is possible. Secondly, there could be a case for a local Debian mirror on my laptop. Otherwise, an interesting experimental extension to UDD would be "Required bandwidth" - the sum of the recursive build-dependencies plus the upload size of the diff/binaries.

This is time again for my annual report about the number of developers per country. This is now the fourth edition of this report, after the 2011, 2010 and 2009 editions. So, here we are with the June 2012 version, sorted by the ratio of *active* developers per million population for each country.

Act: number of active developers
Dev: total number of developers
A/M: number of active devels per million pop.
D/M: number of devels per million pop.
2009: rank in 2009
2010: rank in 2010
2011: rank in 2011 (June)
2012: rank now

Code	Name	Population	Act Dev	Dev	Act/Million	Dev/Million	2009	2010	June 2011	June 2012
fi	Finland	5259250	21	33	3,99	6,27	1	1	1	1
ie	Ireland	4670976	12	15	2,57	3,21	13	9	6	2	*
ch	Switzerland	7870134	20	26	2,54	3,3	2	2	2	3
mq	Martinique	396404	1	1	2,52	2,52			3	4
se	Sweden	9088728	22	37	2,42	4,07	3	6	7	5	*
no	Norway	4973029	12	15	2,41	3,02	5	4	4	6
nz	New Zealand	4331600	10	15	2,31	3,46	4	3	5	7
lu	Luxemburg	503302	1	1	1,99	1,99	8	5	8	8
de	Germany	81471834	155	223	1,9	2,74	7	7	9	9
at	Austria	8217280	14	24	1,7	2,92	6	8	10	10
fr	France	65350000	94	124	1,44	1,9	12	12	11	11
au	Australia	22607571	29	57	1,28	2,52	9	10	12	12
be	Belgium	11071483	13	16	1,17	1,45	10	11	13	13
uk	United-Kingdom	62698362	71	111	1,13	1,77	14	14	14	14
nl	Netherlands	16728091	18	38	1,08	2,27	11	13	15	15
ca	Canada	33476688	31	60	0,93	1,79	15	15	17	16	*
dk	Denmark	5529888	5	10	0,9	1,81	17	17	16	17
es	Spain	46754784	32	53	0,68	1,13	16	16	19	18	*
it	Italy	59464644	37	52	0,62	0,87	23	22	22	19	*
hu	Hungary	10076062	6	11	0,6	1,09	18	25	26	20	*
cz	Czech Rep	10190213	6	6	0,59	0,59	21	20	21	21
lt	Lithuania	3535547	2	2	0,57	0,57	28	19	20	22
bg	Bulgaria	7364570	4	4	0,54	0,54	25	23	23	23
us	USA	313232044	165	368	0,53	1,17	19	21	25	24	*
il	Israel	7740900	4	6	0,52	0,78	24	24	24	25
hr	Croatia	4290612	2	2	0,47	0,47	20	18	18	26	*
lv	Latvia	2204708	1	1	0,45	0,45	26	26	27	27
uy	Uruguay	3477778	1	2	0,29	0,58	22	27	28	28
jp	Japan	127078679	35	50	0,28	0,39	30	28	29	29
pl	Poland	38441588	9	13	0,23	0,34	29	29	30	30
cr	Costa Rica	4301712	1	1	0,23	0,23	31	30	31	31
ar	Argentina	40677348	8	10	0,2	0,25	34	33	35	32	*
sg	Singapore	5183700	1	1	0,19	0,19				33	*
pt	Portugal	10561614	2	4	0,19	0,38	27	32	32	34
gr	Greece	10787690	2	3	0,19	0,28	33	38	34	35
sk	Slovakia	5477038	1	1	0,18	0,18	32	31	33	36
tw	Taiwan	23040040	3	3	0,13	0,13	37	34	37	37
br	Brazil	192376496	21	22	0,11	0,11	36	35	38	38
by	Belarus	9577552	1	1	0,1	0,1	35	36	39	39
mg	Madagascar	21281844	2	2	0,09	0,09	44	37	40	40
cu	Cuba	11241161	1	1	0,09	0,09		38	41	41
kr	South Korea	48754657	4	6	0,08	0,12	39	39	42	42
ec	Ecuador	15007343	1	1	0,07	0,07		40	43	43
cl	Chile	16746491	1	2	0,06	0,12	42	41	44	44
ru	Russia	143030106	7	8	0,05	0,06	43	42	47	45	*
ro	Romania	21904551	1	2	0,05	0,09	45	43	45	46
co	Colombia	45566856	2	3	0,04	0,07	41	44	46	47
za	South Africa	50590000	2	9	0,04	0,18	38	48	48	48
ve	Venezuela	28047938	1	1	0,04	0,04	40	45	50	49
my	Malaysia	28250000	1	1	0,04	0,04			49	50
pe	Peru	29907003	1	1	0,03	0,03	46	46	51	51
tr	Turkey	74724269	2	2	0,03	0,03	47	47	52	52
mx	Mexico	112336538	2	2	0,02	0,02	49	49	53	53
th	Thailand	66720153	1	2	0,01	0,03	50	50	54	54
eg	Egypt	80081093	1	3	0,01	0,04	51	51	55	55
cn	China	1344413526	9	13	0,01	0,01	53	53	57	56
in	India	1210193422	8	8	0,01	0,01	52	52	56	57
sv	El Salvador	7066403	0	1	0	0,14			36	58	*
ua	Ukraine	45134707	0	0	0	0	48	53	58	59
			920	1490	61,74%

A few interesting facts:

• Ireland is the big surprise. It pops up to rank 2, thanks to 2 more developers in a relatively small country.
• Sweden climbs to rank 5. Three more active developers.
• Indeed, between Ireland (2) and New Zealand (7), the difference is really small and one more or less developer changes things significantly
• Finland is still way ahead others
• Canada takes over Denmark, thanks to four more active developers
• Spain, Italy and Hungary also increase their rank significantly.
• USA stops going down. The number of US developers is now increasing again
• No increase in number of german developers. Who said that Debian is a German cabal? Still, the devel/million ratio increases because....the population of Germany is decreasing..:-)
• One new country : Singapore, appearing immediately at rank 33
• We partly lose El Salvador where the only developer doesn't seem active anymore. Rene, wake up? :-)
• Once again, DebConf will be hosted in a country with NO Debian developer (Nicaragua). Just like last year, let's hope it will change.
• We have 24 more active Debian developers and 40 more developers overall.
• I kept Martinique separated from France. I already explained why last year..:)

PS: no, comments are still impossible on this blog. Yes, I know: not good. But I really don't want to have the hassle of dealing with spam and I'm old school. So, comments should be sent by mail to bubulle@d.o.

Do you remember how Intel used to be the white knight of graphic card manufacturers, the one from which you could buy hardware without looking, being sure that it would be supported by GNU/Linux directly? Do you remember how they broke that by releasing the infamous Poulsbo card? Well, they did it again. The newest Intel Atom D2500, D2550, D2700, N2600 and N2800 processors, codenamed CedarView and Cedar Trail , include a graphic core that is only supported by a proprietary driver only available on Meego (read: unusable on regular GNU/Linux.So, if you plan to buy a netbook, beware! Intel is not a manufacturer you can trust to provide freely usable hardware. Just like other manufacturers, check that the hardware you are planning to buy is supported, by asking on #intel-gfx on Freenode. The most relevant course of action probably consists in buying a previous model or waiting for the next models.

One interesting side-effect of setting aside a regular slot of time for a task is that you are guaranteed to spend some time, perhaps not much, thinking about that task, every week. As a consequence, even one evening a week is enough for me to think about my Debian work at other times. This means I am in a better position to use any unplanned free time more effectively. This Tuesday was a bank holiday which I spent on DIY, lounging in a chocolate shop and catching up on The Apprentice, so there wasn't a 'Debian Day' per se. However, I did upload a few versions of lhasa. Following on from last week's thoughts on sound-juicer, I'm pleased to report that the two folks working on a more recent musicbrainz update were in fact working together, and after some discussion Michael Biebl sponsored an upload. Thanks Michael! Once that had gone in I was free to sponsor flactag, which was packaged by the same folks. There's some follow-on work to do for musicbrainz5. In Debian, gnome-mplayer, gnome-sushi, goobox, kdemultimedia, rhythmbox and sound-juicer need updating; I'm fairly sure Michael sorted out a patch for gnome-sushi at least but upstream want a patch that implements falling back to version 4 if version 5 doesn't exist. Finally, I sponsored Gustavo's vavoom package.

I did some upgrades of what seemed like a million packages today on my debian sid computer. I was doing this remotely and when I tried to ssh back in I got kicked out after entering my password, hmmm. OK, so I waited until I could get in front of it and tried to login to the console and was greeted with the message Cannot make/remove an entry and kicked out, even as root; double hmmm. So it was boot into single user time which, thankfully worked. bash loaded ok so it wasn t a shell problem. However su failed which means we are into pam problem territory. I found a bug report that sounded like my problem, but with no clear solution except upgrade in Ubuntu. The solution was pretty simple for me. It s something funky with /etc/pam.d/common-sesssion and a simple /usr/sbin/pam-auth-update fixed it for me.

There are quite some tools to find duplicate files in Debian and depending on the task I use either hardlink (see this blog posting), fdupes (if I need output with all identical files on one line; see example below), or duff (if it has to be performant). But for code deduplication in historically grown code you sometimes need a tool which does not only find identical files, but also those which just differ in a few blanks or blank lines. I found two tools in Debian which can give you some kind of percentage of similarity: simhash (which is btw. orphaned; upstream homepage) and similarity-tester (upstream homepage). simhash has the shorter name and hecne sounds more usable on the command-line. But it seems only be able to compare two files at once and also only after first computing and writing down its similarity hash to a file. Not really usable for those one-liner cases on the command-line. similarity-tester has the longer name (and one which made me suspect that it may be a GUI tool), but provides what I was looking for:

$ find . -type f   sim_text -ipTt 75

This lists all files in the current directory which have at 75% ( -t 75 ) in common with another file in the list of files. The option -i causes sim_text to read the files to compare from standard input; -p causes sim_text to just output the similarity percentage; and -T suppresses the per-file list of found tokens. I used similarity-tester s sim_text tool to compare natural langauge as most of the files, I had to test, are shell scripts. But similarity-tester also provides tools to test the similarity of code in specific programming languages, namely C, Java, Pascal, Modula-2, Lisp and Miranda. Example output from the xen-tools project (after I already did a lot of code deduplication):

./intrepid/30-disable-gettys consists for 100 % of ./edgy/30-disable-gettys material
./edgy/30-disable-gettys consists for 100 % of ./intrepid/30-disable-gettys material
./common/90-make-fstab-rpm consists for 98 % of ./centos-5/90-make-fstab material
./centos-5/90-make-fstab consists for 98 % of ./common/90-make-fstab-rpm material
./gentoo/55-create-dev consists for 91 % of ./dapper/55-create-dev material
./dapper/55-create-dev consists for 90 % of ./gentoo/55-create-dev material
./gentoo/55-create-dev consists for 88 % of ./common/55-create-dev material
./common/90-make-fstab-deb consists for 87 % of ./common/90-make-fstab-rpm material
./common/90-make-fstab-rpm consists for 85 % of ./common/90-make-fstab-deb material
./common/30-disable-gettys consists for 81 % of ./karmic/30-disable-gettys material
./intrepid/80-install-kernel consists for 78 % of ./edgy/80-install-kernel material
./edgy/30-disable-gettys consists for 76 % of ./karmic/30-disable-gettys material
./karmic/30-disable-gettys consists for 76 % of ./edgy/30-disable-gettys material
./common/50-setup-hostname-rpm consists for 76 % of ./gentoo/50-setup-hostname material

Depending on the length of possible filenames and amount of files this can be made more readable using the column utility from the bsdmainutils package and reversed by using tac from the coreutils package:

$ find . -type f   sim_text -ipTt 75   tac   column -t
./common/50-setup-hostname-rpm  consists  for  76   %  of  ./gentoo/50-setup-hostname    material
./karmic/30-disable-gettys      consists  for  76   %  of  ./edgy/30-disable-gettys      material
./edgy/30-disable-gettys        consists  for  76   %  of  ./karmic/30-disable-gettys    material
./intrepid/80-install-kernel    consists  for  78   %  of  ./edgy/80-install-kernel      material
./common/30-disable-gettys      consists  for  81   %  of  ./karmic/30-disable-gettys    material
./common/90-make-fstab-rpm      consists  for  85   %  of  ./common/90-make-fstab-deb    material
./common/90-make-fstab-deb      consists  for  87   %  of  ./common/90-make-fstab-rpm    material
./gentoo/55-create-dev          consists  for  88   %  of  ./common/55-create-dev        material
./dapper/55-create-dev          consists  for  90   %  of  ./gentoo/55-create-dev        material
./gentoo/55-create-dev          consists  for  91   %  of  ./dapper/55-create-dev        material
./centos-5/90-make-fstab        consists  for  98   %  of  ./common/90-make-fstab-rpm    material
./common/90-make-fstab-rpm      consists  for  98   %  of  ./centos-5/90-make-fstab      material
./edgy/30-disable-gettys        consists  for  100  %  of  ./intrepid/30-disable-gettys  material
./intrepid/30-disable-gettys    consists  for  100  %  of  ./edgy/30-disable-gettys      material

Compared to that, fdupes only finds the two 100% identical files:

$ fdupes -r1 . 
./intrepid/30-disable-gettys ./edgy/30-disable-gettys 

But fdupes helped me already a lot to find the first bunch of identical files in xen-tools. :-)

Last week (28th May), I had five Internet services across two countries. This week, I have seven (7), with two new ones in Australia. It all begin, when Internode decided that in their own best interest to migrate me from my existing DSLAM to one of their own infrastructure. They had sent an email indicating when it was likely to occur. And they were sending invoices to the same address. Naturally, they would send the date of the migration to the same address they had used for the past 9 months to communicate to me with, right? Wrong. Internode still do not understand why they never sent the notification email, as it turns out. Compare the response time, update times, and restore times they specify to what actually occurred.

2012-05-28 12:05 BST

I'm informed of an outage. In the past, the problem has been at my end, so I run through the standard diagnostic procedure. Everything seems to be normal, so I call Internode. Sure enough, our service dropped off the Internet. According to Timothy, that happened at 11:15 AEST because of a migration to their own DSLAM. After some diagnosis from Internode, they realise that whilst the migration has successfully disconnected the service, they have not reconnected anything. There is no sync. Ticket #454448 raised - which is when I'm informed that I'm on the business SLA. I've asked for updated to both an Australian and UK mobile. "Not a problem". Estimated time of resolution: 48 hours. Yes, 48 hours. There goes the first SLA milestone.

2012-05-30 07:28 BST

I ring and speak with Carl. He indicates that as best he knows, the service is to be 7pm. for those keeping score, no updates have occurred during this time. To either number. I query him about the SLA period. He says the response time is 2hrs and restore time is 10hrs. He also indicates that this can all be checked online and that we can apply for a refund.

2012-05-30 10:27 BST

I ring, again, and this time speak with Justin. I ask why I've not have an updated, since 7pm AEST has well and truly gone. He authenticates me and places me on hold to investigate. Advises that the issue is fixed, says he sees sync on the line. I indicate that I do not. He asks me to turn off the ADSL modem, we do, and he still sees sync. on the line. Believes there is another problem at the exchange, another issue raised with wholesaler. No ticket number issues to me. ETA to fix 7pm 2nd June. Note:This was a 1 hour International call

2012-06-01 04:00 BST

I happened to check, and notice I have connectivity to the external interfaces (and thus, obviously, line sync.) again. No notification though.

2012-06-01 14:31 BST

Amidst all this, I also have to to source two external connections - Telstra & Optus, get them setup, reconfigure routing, email delivery and other network services so that the businesses which are relying on the Internet, still work. Not fun. Especially when I have no access to the routers. I had to get multiple people on site.

2012-06-04 11:XX CEST

I'm told that the second site has gone done. Inexplicably. After some round-trip and diagnosis, we determine that the routing change has occurred. However it has been done incorrectly. And, of course, no notice / or update occurred either. Multiple tickets are raised by multiple people, amongst them: #4562559 from my mobile phone whilst I am in Zurich.

2012-06-04 16:56 CEST

I am informed, and it is confirmed, that the attempted routing change has been reverted and we are back to how things were on 2012-06-01.

2012-06-05 05:44 CEST

I wake up, and find out that of at least three hours ago, they appear to have called (finally!) and actually fixed the routing to both sites. That is a week of down time, multiple people (4 people) around in various countries around the World and hundreds of dollars in calls between us all to get things working. Now, to look into compensation.

2012-06-05 09:43 CEST

I receive a missed call, which I later learn is Internode actually calling me (for once!). At that moment, I'm actually giving a lightening talk on Django-PaaS

And that, dear reader, is why Internode Business SLA is a waste of money. The services were broken by a change of no net benefit to me, was over a week ago. Also, during this process. My UK home Internet went down. My UK Three post-paid service "ran out of money" and Vodafone has a service outage for half a day. A perfect storm of Internet dis-connectivity. So, if you have been wondering, that is why I've been offline for various parts of the last week.

From time to time I need to convert scanned documents to a pdf format.


Usage scenario 1: I scan part of a book (i.e. some article) on a school's scanner that sends me 10 big separate color pdf files (one pdf per page). I want to get one nice, small (black and white) pdf file with all the pages.


Usage scenario 2: I download a web form, print it, fill it in, sign it, scan it on my own scanner using Gimp and now I want to convert the image into a nice pdf file (either color or black & white) to send back over email.

Solution: I save the original files (be it pdf or png) into a folder and use git to track it. Then create a simple reduce script to convert it to the final format (view it as a pipeline). Often I need to tweak one or two parameters in the pipeline.

Here is a script for scenario 1:
<script src="https://gist.github.com/2871625.js?file=reduce2.sh"></script>
And here for scenario 2:
<script src="https://gist.github.com/2871625.js?file=reduce1.sh"></script>
There can be several unexpected surprises along the way. From my experience:

• If I convert png directly to tiff, sometimes the resolution can be wrong. The solution is to always convert to ppm (color) or pbm (black and white) first, which is just a simple file format containing the raw pixels. This is the "starting" format (so first I need to convert the initial pdf or png into ppm/pbm) and then do anything else. That proved to be very robust.
• The tiff2pdf utility proved to be the most robust way to convert an image to a pdf. All other ways that I have tried failed in one way or another (resolution, positioning, paper format and other things were wrong....). It can create multiple pages pdf files, set paper format (US Letter, A4, ...) and so on.
• The linux convert utility is a robust tool for cropping images, converting color to black and white (using a threshold for example) and other things. As long as the image is first converted to ppm/pbm. In principle it can also produce pdf files, but that didn't work well for me.
• I sometimes use the unpaper program in the pipeline for some automatic polishing of the images.

Jo and I flew in to HK a few days early for the Linaro conference, so we got some time to ourselves for doing touristy stuff. That was a good plan - we got to see quite a lot of Kowloon for a couple of days, then we wandered over to Hong Kong island itself for a day. Jo also had more time during the week while I was in the sessions at the conference. Hong Kong is a busy place, very densely packed at all times of day and night that we could see. The first impression we got on the taxi from the airport on day one was that it was a city full of tower blocks - huge numbers of apartments stacked very closely together for living space. That makes sense, I guess, with the large population in such a small space. There are houses around with more space for people to live, but they are understandably very expensive. Later on in Kowloon, we got to meet lots of the local street vendors, mostly offering us "Copy Rolex", "Fake handbag" and the like. Even "Sharp trousers" at one point. We decided not to try them out... :-) We saw lots and lots of very expensive shops downtown, crammed into huge shopping centres in the tourist area. As an antidote for all of that, we headed up to the "Golden Computer Arcade" to go browsing around the vast array of tiny technology stalls and shops there. Picked up some nice cheap laptop memory and some SD cards there. Yay! Across on Hong Kong island itself on Sunday, we found my grandfather's grave then rode around on the trams for a while to get a good view of the city, including a democracy protest that happened to be on that day! We had a quick look around Tin Hau Temple, where we found a very specific set of rules for visitors:

I can only guess that they must have been terrified by gangs of kids racing their RC cars at one point! We finished the day by heading up to Victoria Peak for the sunset view and some dinner, then back to the hotel. Killing time in a shopping centre waiting for the bus to turn up, we found a truly astonishing little item on sale, a Solar Queen!

Apparently this gift has a solar panel on the handbag to provide power to a little motor in the Queen's hand to make her wave to her subjects. We decided that we'd somehow have to live without such a delight in our lives... :-) HK was a cool place to visit. It still feels very British in some ways, as you'd expect from an ex-colony (power sockets are still the same as ours, driving on the left, everybody spoke English, etc.) but utterly alien in others (the climate, working public transport *grin*). I look forwards to heading out there again!

Sometimes conferences can be dull and boring, but sometimes they can be just awesome in terms of finding the right people to collaborate with. Linaro Connect in Hong Kong last week was definitely one of the great ones! I chaired my my usual sessions (armhf status and cross-distro ARM) and we had some lively discussion in both. We're probably just about done with the armhf sessions, as most distros have accepted a hard-float ARMv7 port now and there's not so much specific work left there now that future sessions will be necessary. The cross-distro work for ARM ports is likely to continue into the future, but we're going to be concentrating on bootstrapping work for ARMv8 soon. Talking v8, there were a lot of meetings discussing the various work topics for this new 64-bit ARM architecture: kernel, toolchains, bootstrapping etc. More to come on that soon! On top of all this useful discussion and planning, we also found time during the week for some hacking. This was the highlight of the week for me, as I found some expert help to solve my long-standing Ruby on ARM bug (Debian bug #652674, ruby1.9.1: FTBFS on armhf: test suite segfaults). Ulrich Weigand (an IBM toolchain wizard seconded into the Linaro team) sat with me for a couple of hours while we worked through reproducing the problem, only to find that he could not reproduce it! The crash had looked very much like a pthreads locking bug, which was scaring me. After some digging, we worked out where the problem was, and how it was now fixed. For a long time on ARM, the Linux getcontext/setcontext system calls have never been implemented; apparently nobody really missed them, so they have never been a priority. In Ruby 1.9.x, the implementation of the new "fiber" primitive wanted to use getcontext/setcontext to control stack state etc. in different threads of control. In cases where they are not available or known not to work well, Ruby has fallback code to implement similar functionality. It seems that fallback code is buggy. Maybe it was correct at some point and has bit-rotted due to not being exercised, or maybe it was buggy as written, but it clearly does not work correctly for us now. In the last few months, getcontext/setcontext have finally been implemented for ARM in glibc trunk (by Michael Hope, also in the Linaro toolchain team!) and backported into current Debian and Ubuntu eglibc versions. Re-running configure and rebuilding Ruby against the most recent code in both Sid and Precise fixed the test suite crash we were seeing earlier. Yay! We could also provoke the bug again at will by quickly hacking around the Ruby source to force it to switch back to the fallback code, thereby verifying the fix. Why does this matter? As we're expecting ARM servers to enter the market soon, web apps written in Ruby on Rails are going to be an important part of the software stack that customers will want to run. Broken fibers and threading would not help here! It's great to meet up and work with talented folks like this; Linaro Connect is an excellent event for getting stuff done!

I never met my father's father, as he died many years before I was born. He was a soldier in the British Army and was killed in an accident in Burma in the 1950s. I'm told that back then the Army did not repatriate casualties as a matter of course like they do today, so none of the family got to see his funeral or even visit the grave. Like many of the military casualties in the Far East, he was buried in the Colonial Cemetery in Hong Kong. A few photographs were sent to my grandmother of the ceremony and the headstone, but that's all we ever saw.

Fast forward to 2012. I travelled to Hong Kong for a conference (Linaro Connect, more about that later...). I looked up the details of where grandad was laid to rest, and found a major coincidence. He was killed on 27th May 1952, almost exactly 60 years ago! I resolved to go to find him on 27th May 2012 to make the most of this lucky anniversary. Wikipedia told me that in the intervening years the Colonial Cemetery had been renamed to Happy Valley Cemetery, then simply Hong Kong Cemetery, and repurposed from military to civilian use. Jo and I bought some flowers and headed over to Happy Valley on Hong Kong Island on Sunday afternoon. We found a very large cemetery with a helpful map posted to show the different sections, but nothing to tell us where we should be looking. One of the attendants at the office on site looked at the details we had and shrugged to say "sorry, can't help". Ah well, nobody said this was going to be very easy... What we found after a few minutes was that the graves weren't laid out in any obviously logical fashion. In any given small area, you'd find people buried from roughly the same period, but people for any given period could (and would) be scattered across multiple sections at all corners of the site. Great, time for an exhaustive search then. I started combing the site, checking all the markers I could find that looked anything like the tiny grainy black and white photo we had of grandad's headstone.

Almost two hours later, I eventually found him. Two hours of heavy work: the cemetery is built on the side of quite a steep hill, and the prevailing weather was very hot with 100% humidity. But, I forgot all that as we eventually stumbled across the correct grave in (no exaggeration!) the very last part of the last section of the site, 20a.

I was relieved at this point: as things had taken so long, I had started to worry that maybe the grave had been moved or the headstone damaged and lost. But no, I found the man who had gone off to war leaving my dad and aunt as small children. We took some photos and took note of where we had found the final resting place of Sergeant William Alexander McIntyre of the Royal Signals, a man I never met in life but clearly a very important member of our family.

My curiosity was definitely nookd by this story. While this doesn't seem to have been an act of censorship or anything nefarious by Barnes and Noble itself, this does point to just how terrifying ebooks can be if we aren't careful. In the future, when we are dealing with the entirety of human knowledge, we better be careful with the find and replace function. And we better make sure we control our computer systems so that other people can't nook us without our permission. See also:

• War and Nookd
• Nookd

As someone who read War and Peace in paper form, much of it in bed, I definitely empathized with this:

Some weeks ago I decided that I wanted to read Tolstoy's War and Peace. Lou Ann loaned me her copy. At more than 1100 pages, reading it in bed required as much strength as balancing a box of bricks in my hands. In my senior years I have developed arthritis in my thumbs, which made the effort not only difficult, but painful.

New PostgreSQL microreleases with two security fixes and several bug fixes was just announced publically. I spent the morning with the packaging orgy for Debian unstable and experimental (now uploaded), Debian Wheezy (update sent to security team), Ubuntu hardy, lucid, natty, oneiric, precise (LP #1008317) and my backports PPA. I tested these fairly thoroughly, but please let me know if you encounter any problem with these.

I have been avoiding #629247 for quite a while. Not because I think we couldn t use a better shell parser, but because I dreaded having to write the parser. Of course, #629247 blocks about 16 bugs and that number will only increase, so someone has to solve it eventually Unfortunately, that someone is likely to be me . So I managed to scrabble down the following Perl snippet. It does a decent job at getting lines split into words (which may or may not contain spaces, newlines, quotes etc.). It currently tokenizes the <<EOF -constructs (heredocs?). Also it does not allow one to distinguish between EOF and EOF (the former ends the heredoc, the latter doesn t.). Other defects includes that it does not tokenize all operators (like >& ). Probably all I need is a list of them and all the special cases (Example: >& can optionally take numbers on both sides, like >&2 or 2>&1 ). It does not always appear to terminate (I think EOF + unclosed quote triggers this). If you try it out and notice something funny, please let me know. You can also find an older version of it in the bug #629247 and the output it produced at that time (that version used instead of as token marker).

#!/usr/bin/perl
use strict;
use warnings;
use Text::ParseWords qw(quotewords);
my $opregex;
 
  my $tmp = join( " ", map   quotemeta $_   qw (&&     ; ));
  # Match & but not >& or <&
  # - Actually, it should eventually match those, but not right now.
  $tmp .= ' (?<![\>\<])\&';
  $opregex = qr/$tmp/ox;
 
my @tokens = ();
my $lno;
while (my $line = <>)  
  chomp $line;
  next if $line =~ m/^\s*(?:\# $)/;
  $lno = $. unless defined $lno;
  while ($line =~ s,\\$,,)  
  $line .= "\n" . <>;
  chomp $line;
   
  $line =~ s/^\s++//;
  $line =~ s/\s++$//;
  # Ignore empty lines (again, via "$empty \ $empty"-constructs)
  next if $line =~ m/^\s*(?:\# $)/;
  my @it = quotewords ($opregex, 'delimiters', $line);
  if (!@it)  
  # This happens if the line has unbalanced quotes, so pop another
  # line and redo the loop.
  $line .= "\n" . <>;
  redo;
   
  foreach my $orig (@it)  
  my @l;
  $orig =~ s,",\\\\",g;
  @l = quotewords (qr/\s++/, 1, $orig);
  pop @l unless defined $l[-1] && $l[-1] ne '';
  shift @l if $l[0] eq '';
  push @tokens, map   s,\\\\",",g; $_   @l;
   
  print "Line $lno: -" . join ("- -", map   s/\n/\\n/g; $_   @tokens ) . "-\n";
  @tokens = ();
  $lno = undef;
 

Here is a little example script and the tokenization of that script (no, the example script is not supposed to be useful).

$ cat test
#!/bin/sh
for p in *; do
    if [ -d "$p" ];then continue;elif
    [ -f "$p" ]
    then echo "$p is a file";fi
done
$ ./test.pl test
Line 3: -for- -p- -in- -*- -;- -do-
Line 4: -if- -[- --d- -"$p"- -]- -;- -then- -continue- -;- -elif-
Line 5: -[- --f- -"$p"- -]-
Line 6: -then- -echo- -"$p is a file"- -;- -fi-
Line 7: -done-

Probably by the time you read this, I ll be on the road to kohacon12. It s been a fun week or so with the final arrangements on top of our other planned work and some unexpected work too, but I think we ve done enough and it will all come together. (There are still a few places left if you d like to register, but not lots.) The most nerve-wracking bit has been the sponsorship. Kohacons are free to attend and funded entirely by sponsors, which is great in so many ways. It s a bit scary for the host organisation(s), but I think the community has helped so much that it ll break even. We won t know for sure until the final reckoning in a month or so. One unneeded worry was Paypal freezing our account for a couple of weeks and sending us contradictory and absurd demands for information. Why isn t there an easier way to get money out of the USA (and Australia and NZ, actually) that isn t either slow (cheques) or expensive (wire)? Once again, I m left feeling that banks are being a big problem for business. The best bit has been the spirit of the volunteers. Our co-op couldn t have done this without them. Some of them have gone a really long way to help metaphorically, bailing us out while Paypal was chewing up our time, and sometimes physically, flying around the world I ll be raising a glass to all of them, whether they re in Edinburgh or not. So, next update should be from Edinburgh!

A lot of people interested in free software, and user autonomy and network services are very worried about Facebook. Folks are worried for the same reason that so many investors are interested: the networks effects brought by hundreds of millions of folks signed up to use the service. Network effects -- the concept that a good or service increases in value as more people use it -- are not a new problem for free software. Software developers target Microsoft Windows because that is where the large majority of users are. Users with no love for Microsoft and who are otherwise sympathetic to free software use Windows because programs they need will only run there. Folks worried about Facebook are afraid for similar reasons. Sure, you can close down your Facebook account and move to Diaspora. But who will you talk to there? You can already hear people complaining about Facebook the same way they've been complaining about Windows or Office for years. People feel that their hands are tied and that their software, and their social network, will be determined by what everybody is doing. I'm worried about Facebook. But I'm not too intimidated by Facebook's network effects for two reasons. First, using Facebook doesn't preclude using anything else. Twitter has enormous overlapping functionality with Facebook. Sure, people use the systems very differently. But they both ask you to create lists of friends and followers and are designed around sending and receiving short status messages. Millions of people do both and both systems are thriving. For the millions of people who use both Facebook and Twitter, the two services have had to negotiate their marginal utility in a world they share with the other one. People decide that Twitter is for certain types of short messages and Facebook is for others. But these arrangements shift over time. And the relationships between services aren't always peaceful coexistence. Remember Friendster? Remember Orkut? Remember Tribe? Remember MySpace? MySpace, and all the others, are great examples of how social networks die. They very slowly fade away. MySpace users signed up for Facebook accounts and used both. They almost never just switched. Over time, as one platform became more attractive than the other, for many complicated reasons, attention and activity shifted. People logged in on MySpace less and Facebook more and, eventually, realized they were effectively no longer MySpace users. Anyone that has been on the Internet long enough to watch a few of these shifts from one platform to another knows that they're not abrupt -- even if they can be set in motion by a particular event or action. Users of social networking sites simply don't have to choose in the way that a person choosing to boot Windows and GNU/Linux does. I'm sure the vast majority of people with Diaspora accounts use Facebook actively. This is not a problem for Diaspora. It is how Diaspora -- or whatever else eventually achieves what many of us hoped Diaspora would -- could win. Second, Facebook is for the ephemeral. Facebook is primarily used for information that was produced very recently. This week if not today. If not this hour. Facebook has an enormous amount of data that users have fed it that may be hard to get out and move somewhere else. But most people don't care very much about having any regular access to the large majority of this information. What people care deeply about is having access to the data that they and their friends created today. And that data can just as easily be created somewhere else tomorrow. Or, with the right tools, created just as easily in both places. Compare this to something like Windows where moving away would require learning, converting, and perhaps even writing, new software. Perhaps even in new programming languages that most developers don't know yet. Compared to Windows, a migration away from Facebook will be easy. Facebook's photo galleries are an example of an important place where this holds less well. Social network information -- i.e., the list of who is friends with who -- is another example of something that is persistently valuable. That said, people really enjoy the act of finding and friending. Indeed, this process was part of the initial draw of Facebook and other social networks. None of this means that Facebook is over. It doesn't even mean that its ascendancy will be slowed. What it does mean is that Facebook is vulnerable to the next thing more than many technology firms that have benefited from network effects in the past. If users are given compelling reasons to switch to something else, they can with less trouble and they will. That compelling reason might be a new social network with better features or an awesome distributed architecture that allows freedom for users and the ability of those users to benefit from new and fantastic things that Facebook's overseers would never let them have and without the things Facebook's users suffer through today. Or it might be a sexier proprietary box to store users' private information. It doesn't mean that I'm not worried about Facebook. I remain deeply worried. It's just not very hard for me to imagine the end.