Recently I was walking through a park and heard a women call out Take off that stupid helmet . Usually I ignore what other people are saying but that seemed noteworthy. It turned out that a young boy (maybe 4yo) was being taught to ride a bike and his parents seemed to think that wearing a helmet was a bad idea. There is ongoing debate about the benefit to an adult in wearing a helmet while riding a bike. But it seems clear that for a young child riding on a concrete path a helmet is a really good thing. When it became apparent that everyone in the park was watching the parents decided to have him ride on the grass instead. On a related note I was recently talking to an employee of a roadside assistance company about what happens when a child is locked in a car. Apparently if a child is locked in a car with the keys the emergency services people won t smash a window as long as the child is kicking and screaming. While the child is obviously in distress they apparently aren t going to immediately die and that s OK, but when they go quiet it s time to damage the car to save them! I can imagine situations when it s OK for the emergency services people to wait for a car expert to open the car without damage, if the weather is cool and the child seems happy then a delay probably doesn t matter much. But if the child is in distress then the attitude that anything which doesn t kill the kid is OK seems wrong. Related posts:

1. Old Mobile Phones as Toys In the past I have had parents ask for advice...
2. Too Stupid to be a Bishop A Stupid Bisop breaks the Godwin Rule The Sydney Morning...
3. Which People are Stupid on the Internet? I don t think that the answer is everyone or even...

Due to some real life impediments (including a beautiful one with just over 2 months of life called Daniel), I will not be able to go to this year's DebConf, even though it seems that this DebConf will exceed many expectations with the support that the governement of Nicaragua just offered. Things that I would like to have worked on Unfortunately, this means that I will not be working with some fellows that may be there, like people from the pkg-multimedia team. In particular, as I am working on some packages that are not that well suited to distributions (especially regarding HandBrake), it would be highly productive to have a hacking session or two with other people interested in having this popular and easy-to-use video ripper/transcoder on Debian. I would also like to be able to talk in person with other people (including Joey Hess) about controversial packaging of some programs, like GNU Parallel, which I maintain in Debian. There are many things else that I would love to work on, besides helping a bit with some of the still pending transitions of the the Debian Fonts Task Force umbrella, meeting friends and other social stuff. Oh, well. I guess that working at home will be the solution for some time.

• Latvian is now completed.
• Lithuanian is now completed.
• Galician and Welsh completed sublevels 1 and 2

At this very moment, it means that I would deactivate 14 languages:

• Amharic (5f17u in sublevel 2)
• Croatian (16f19u, 6f9u)
• Georgian (17f47u, 10f21u)
• Kurdish (12f25u, 6f22u)
• Malayalam (5u in sublevel 2 only!)
• Nepali (16f54u, 10f21u)
• Norwegian Nynorsk (13f34u, 13f23u)
• Albanian (19f50u, 13f23u)
• Tamil (2f1u, 1f)
• Tagalog (18f54u, 24f39u)

A few days ago I reported how to get the support status out of Dell using an unofficial and undocumented SOAP API, which I since have found out was discovered by Daniel De Marco in february. Combined with my web scraping code for HP, Dell and IBM from 2009, I got inspired and wrote a web service based on Scraperwiki to make it easy to look up the support status and get a machine readable result back. This is what it look like at the moment when asking for the JSON output:

% GET https://views.scraperwiki.com/run/computer-hardware-support-status/?format=json&vendor=Dell&servicetag=2v1xwn1
supportstatus( "servicetag": "2v1xwn1", "warrantyend": "2013-11-24", "shipped": "2010-11-24", "scrapestamputc": "2012-06-06T20:26:56.965847", "scrapedurl": "http://143.166.84.118/services/assetservice.asmx?WSDL", "vendor": "Dell", "productid": "" )
%

It currently support Dell and HP, and I am hoping for help to add support for other vendors. The python source is available on Scraperwiki and I welcome help with adding more features.

Exams period has almost come to an end, only one more to go, tomorrow morning. Thinks were quite busy during this period and my time was fairly limited. Multitasking did not go as well as I had expected, will consider this in the future.


I've spent the last days reading documentation, code and tutorials, which were very helpful, but I did not manage to do much. The Go Back button (#636211) is now done but there are still some fixes that need to be done on the link timeout (#610752). That one took me soooo long, and it's not like there was much to do, I don't know exactly what the problem was :).


One sad way to describe the way thinks worked during this period is by using a quote: "progress coming slowly, if it comes at all". That's from Lord of the Rings, but it's a bit modified (s/progress/change). Ok, that's overreacting. I do feel that I have made some progress, it's not necessarily visible (as in patches, features) but I feel much more confident about what I am doing.


Short post, now back to study. Programming Paradigms (last exam), here I come :))

While on a recent trip to Italy, I gave a talk two times about Getting Things Done for Hackers. I made a few slides, which I've made public in case anyone's interested, but, really, the website is a better replacement for slides.

So I was trying to run grml-deboostrap in a Debian squeeze VM inside VirtualBox. I got this error:

/proc/misc: No entry for device-mapper found
Is device-mapper driver missing from kernel?

I was using grml-deboostrap with the vmfile parameter. To fix it, I simply did this:

modprobe dm_mod

This will load the device-mapper module.

Today, as an experiment, I attempted to fix a Debian bug while on the train to work. I use a 3G card from Three.co.uk in my Lenovo Thinkpad x121e, and my commute is from Southampton Central to Fleet (changing at Winchester) - just under an hour. 3G coverage is not 100%, but tends to be better around the major stops.

• First, I found a bug. I used udd.debian.org to browse for a relatively simple RC bug, and found bug #674992 in actionaz. The fix was outlined in the report already, so there was very little thinking required.
• Next, I confirmed the FTBFS using cowbuilder. Unfortunately, this required downloading roughly 120MB of dependencies - I have 1GB of data per month, but I couldn't afford to do this every day. I was lucky in that I was near Basingstoke at the time, so had a good HSDPA signal to get the bulk of this. The build had failed before I reached Fleet.
• In the background, I updated debian/control and debian/changelog with the fix. I was able to set off the build, but had to suspend the laptop until lunchtime before it could finish. Cowbuilder needed to download only a few extra build-deps, as the vast majority were cached from the initial run.
• On the train home, I checked over the result, signed it and uploaded. In this instance, the built package was small enough to upload, but I could see this being a problem with others.
• Finally, I sent the nmudiff, although that was delayed briefly by a drop in connectivity before Southampton Airport.

Thoughts: firstly, part of me is amazed that this is possible. Secondly, there could be a case for a local Debian mirror on my laptop. Otherwise, an interesting experimental extension to UDD would be "Required bandwidth" - the sum of the recursive build-dependencies plus the upload size of the diff/binaries.

This is time again for my annual report about the number of developers per country. This is now the fourth edition of this report, after the 2011, 2010 and 2009 editions. So, here we are with the June 2012 version, sorted by the ratio of *active* developers per million population for each country.

Act: number of active developers
Dev: total number of developers
A/M: number of active devels per million pop.
D/M: number of devels per million pop.
2009: rank in 2009
2010: rank in 2010
2011: rank in 2011 (June)
2012: rank now

Code	Name	Population	Act Dev	Dev	Act/Million	Dev/Million	2009	2010	June 2011	June 2012
fi	Finland	5259250	21	33	3,99	6,27	1	1	1	1
ie	Ireland	4670976	12	15	2,57	3,21	13	9	6	2	*
ch	Switzerland	7870134	20	26	2,54	3,3	2	2	2	3
mq	Martinique	396404	1	1	2,52	2,52			3	4
se	Sweden	9088728	22	37	2,42	4,07	3	6	7	5	*
no	Norway	4973029	12	15	2,41	3,02	5	4	4	6
nz	New Zealand	4331600	10	15	2,31	3,46	4	3	5	7
lu	Luxemburg	503302	1	1	1,99	1,99	8	5	8	8
de	Germany	81471834	155	223	1,9	2,74	7	7	9	9
at	Austria	8217280	14	24	1,7	2,92	6	8	10	10
fr	France	65350000	94	124	1,44	1,9	12	12	11	11
au	Australia	22607571	29	57	1,28	2,52	9	10	12	12
be	Belgium	11071483	13	16	1,17	1,45	10	11	13	13
uk	United-Kingdom	62698362	71	111	1,13	1,77	14	14	14	14
nl	Netherlands	16728091	18	38	1,08	2,27	11	13	15	15
ca	Canada	33476688	31	60	0,93	1,79	15	15	17	16	*
dk	Denmark	5529888	5	10	0,9	1,81	17	17	16	17
es	Spain	46754784	32	53	0,68	1,13	16	16	19	18	*
it	Italy	59464644	37	52	0,62	0,87	23	22	22	19	*
hu	Hungary	10076062	6	11	0,6	1,09	18	25	26	20	*
cz	Czech Rep	10190213	6	6	0,59	0,59	21	20	21	21
lt	Lithuania	3535547	2	2	0,57	0,57	28	19	20	22
bg	Bulgaria	7364570	4	4	0,54	0,54	25	23	23	23
us	USA	313232044	165	368	0,53	1,17	19	21	25	24	*
il	Israel	7740900	4	6	0,52	0,78	24	24	24	25
hr	Croatia	4290612	2	2	0,47	0,47	20	18	18	26	*
lv	Latvia	2204708	1	1	0,45	0,45	26	26	27	27
uy	Uruguay	3477778	1	2	0,29	0,58	22	27	28	28
jp	Japan	127078679	35	50	0,28	0,39	30	28	29	29
pl	Poland	38441588	9	13	0,23	0,34	29	29	30	30
cr	Costa Rica	4301712	1	1	0,23	0,23	31	30	31	31
ar	Argentina	40677348	8	10	0,2	0,25	34	33	35	32	*
sg	Singapore	5183700	1	1	0,19	0,19				33	*
pt	Portugal	10561614	2	4	0,19	0,38	27	32	32	34
gr	Greece	10787690	2	3	0,19	0,28	33	38	34	35
sk	Slovakia	5477038	1	1	0,18	0,18	32	31	33	36
tw	Taiwan	23040040	3	3	0,13	0,13	37	34	37	37
br	Brazil	192376496	21	22	0,11	0,11	36	35	38	38
by	Belarus	9577552	1	1	0,1	0,1	35	36	39	39
mg	Madagascar	21281844	2	2	0,09	0,09	44	37	40	40
cu	Cuba	11241161	1	1	0,09	0,09		38	41	41
kr	South Korea	48754657	4	6	0,08	0,12	39	39	42	42
ec	Ecuador	15007343	1	1	0,07	0,07		40	43	43
cl	Chile	16746491	1	2	0,06	0,12	42	41	44	44
ru	Russia	143030106	7	8	0,05	0,06	43	42	47	45	*
ro	Romania	21904551	1	2	0,05	0,09	45	43	45	46
co	Colombia	45566856	2	3	0,04	0,07	41	44	46	47
za	South Africa	50590000	2	9	0,04	0,18	38	48	48	48
ve	Venezuela	28047938	1	1	0,04	0,04	40	45	50	49
my	Malaysia	28250000	1	1	0,04	0,04			49	50
pe	Peru	29907003	1	1	0,03	0,03	46	46	51	51
tr	Turkey	74724269	2	2	0,03	0,03	47	47	52	52
mx	Mexico	112336538	2	2	0,02	0,02	49	49	53	53
th	Thailand	66720153	1	2	0,01	0,03	50	50	54	54
eg	Egypt	80081093	1	3	0,01	0,04	51	51	55	55
cn	China	1344413526	9	13	0,01	0,01	53	53	57	56
in	India	1210193422	8	8	0,01	0,01	52	52	56	57
sv	El Salvador	7066403	0	1	0	0,14			36	58	*
ua	Ukraine	45134707	0	0	0	0	48	53	58	59
			920	1490	61,74%

A few interesting facts:

• Ireland is the big surprise. It pops up to rank 2, thanks to 2 more developers in a relatively small country.
• Sweden climbs to rank 5. Three more active developers.
• Indeed, between Ireland (2) and New Zealand (7), the difference is really small and one more or less developer changes things significantly
• Finland is still way ahead others
• Canada takes over Denmark, thanks to four more active developers
• Spain, Italy and Hungary also increase their rank significantly.
• USA stops going down. The number of US developers is now increasing again
• No increase in number of german developers. Who said that Debian is a German cabal? Still, the devel/million ratio increases because....the population of Germany is decreasing..:-)
• One new country : Singapore, appearing immediately at rank 33
• We partly lose El Salvador where the only developer doesn't seem active anymore. Rene, wake up? :-)
• Once again, DebConf will be hosted in a country with NO Debian developer (Nicaragua). Just like last year, let's hope it will change.
• We have 24 more active Debian developers and 40 more developers overall.
• I kept Martinique separated from France. I already explained why last year..:)

PS: no, comments are still impossible on this blog. Yes, I know: not good. But I really don't want to have the hassle of dealing with spam and I'm old school. So, comments should be sent by mail to bubulle@d.o.

Do you remember how Intel used to be the white knight of graphic card manufacturers, the one from which you could buy hardware without looking, being sure that it would be supported by GNU/Linux directly? Do you remember how they broke that by releasing the infamous Poulsbo card? Well, they did it again. The newest Intel Atom D2500, D2550, D2700, N2600 and N2800 processors, codenamed CedarView and Cedar Trail , include a graphic core that is only supported by a proprietary driver only available on Meego (read: unusable on regular GNU/Linux.So, if you plan to buy a netbook, beware! Intel is not a manufacturer you can trust to provide freely usable hardware. Just like other manufacturers, check that the hardware you are planning to buy is supported, by asking on #intel-gfx on Freenode. The most relevant course of action probably consists in buying a previous model or waiting for the next models.

One interesting side-effect of setting aside a regular slot of time for a task is that you are guaranteed to spend some time, perhaps not much, thinking about that task, every week. As a consequence, even one evening a week is enough for me to think about my Debian work at other times. This means I am in a better position to use any unplanned free time more effectively. This Tuesday was a bank holiday which I spent on DIY, lounging in a chocolate shop and catching up on The Apprentice, so there wasn't a 'Debian Day' per se. However, I did upload a few versions of lhasa. Following on from last week's thoughts on sound-juicer, I'm pleased to report that the two folks working on a more recent musicbrainz update were in fact working together, and after some discussion Michael Biebl sponsored an upload. Thanks Michael! Once that had gone in I was free to sponsor flactag, which was packaged by the same folks. There's some follow-on work to do for musicbrainz5. In Debian, gnome-mplayer, gnome-sushi, goobox, kdemultimedia, rhythmbox and sound-juicer need updating; I'm fairly sure Michael sorted out a patch for gnome-sushi at least but upstream want a patch that implements falling back to version 4 if version 5 doesn't exist. Finally, I sponsored Gustavo's vavoom package.

Why not just avoid the entire Secure Boot problem by using Coreboot? Because the reason we have the Secure Boot problem is because Microsoft's Windows 8 certification requirements mean vendors have to ship a UEFI implementation with Secure Boot. You could satisfy that by using Coreboot with a Tiano payload, but it'll still have Secure Boot enabled so you still have the same set of problems. But maybe you could just reflash your system with Coreboot? No, because another part of the requirements states that all firmware updates have to be cryptographically signed now. The only way to reflash will be to attach a flash programmer directly to your motherboard.

So why not just use Coreboot? Because it doesn't help solve this problem in any way.

comments

I did some upgrades of what seemed like a million packages today on my debian sid computer. I was doing this remotely and when I tried to ssh back in I got kicked out after entering my password, hmmm. OK, so I waited until I could get in front of it and tried to login to the console and was greeted with the message Cannot make/remove an entry and kicked out, even as root; double hmmm. So it was boot into single user time which, thankfully worked. bash loaded ok so it wasn t a shell problem. However su failed which means we are into pam problem territory. I found a bug report that sounded like my problem, but with no clear solution except upgrade in Ubuntu. The solution was pretty simple for me. It s something funky with /etc/pam.d/common-sesssion and a simple /usr/sbin/pam-auth-update fixed it for me.

There are quite some tools to find duplicate files in Debian and depending on the task I use either hardlink (see this blog posting), fdupes (if I need output with all identical files on one line; see example below), or duff (if it has to be performant). But for code deduplication in historically grown code you sometimes need a tool which does not only find identical files, but also those which just differ in a few blanks or blank lines. I found two tools in Debian which can give you some kind of percentage of similarity: simhash (which is btw. orphaned; upstream homepage) and similarity-tester (upstream homepage). simhash has the shorter name and hecne sounds more usable on the command-line. But it seems only be able to compare two files at once and also only after first computing and writing down its similarity hash to a file. Not really usable for those one-liner cases on the command-line. similarity-tester has the longer name (and one which made me suspect that it may be a GUI tool), but provides what I was looking for:

$ find . -type f   sim_text -ipTt 75

This lists all files in the current directory which have at 75% ( -t 75 ) in common with another file in the list of files. The option -i causes sim_text to read the files to compare from standard input; -p causes sim_text to just output the similarity percentage; and -T suppresses the per-file list of found tokens. I used similarity-tester s sim_text tool to compare natural langauge as most of the files, I had to test, are shell scripts. But similarity-tester also provides tools to test the similarity of code in specific programming languages, namely C, Java, Pascal, Modula-2, Lisp and Miranda. Example output from the xen-tools project (after I already did a lot of code deduplication):

./intrepid/30-disable-gettys consists for 100 % of ./edgy/30-disable-gettys material
./edgy/30-disable-gettys consists for 100 % of ./intrepid/30-disable-gettys material
./common/90-make-fstab-rpm consists for 98 % of ./centos-5/90-make-fstab material
./centos-5/90-make-fstab consists for 98 % of ./common/90-make-fstab-rpm material
./gentoo/55-create-dev consists for 91 % of ./dapper/55-create-dev material
./dapper/55-create-dev consists for 90 % of ./gentoo/55-create-dev material
./gentoo/55-create-dev consists for 88 % of ./common/55-create-dev material
./common/90-make-fstab-deb consists for 87 % of ./common/90-make-fstab-rpm material
./common/90-make-fstab-rpm consists for 85 % of ./common/90-make-fstab-deb material
./common/30-disable-gettys consists for 81 % of ./karmic/30-disable-gettys material
./intrepid/80-install-kernel consists for 78 % of ./edgy/80-install-kernel material
./edgy/30-disable-gettys consists for 76 % of ./karmic/30-disable-gettys material
./karmic/30-disable-gettys consists for 76 % of ./edgy/30-disable-gettys material
./common/50-setup-hostname-rpm consists for 76 % of ./gentoo/50-setup-hostname material

Depending on the length of possible filenames and amount of files this can be made more readable using the column utility from the bsdmainutils package and reversed by using tac from the coreutils package:

$ find . -type f   sim_text -ipTt 75   tac   column -t
./common/50-setup-hostname-rpm  consists  for  76   %  of  ./gentoo/50-setup-hostname    material
./karmic/30-disable-gettys      consists  for  76   %  of  ./edgy/30-disable-gettys      material
./edgy/30-disable-gettys        consists  for  76   %  of  ./karmic/30-disable-gettys    material
./intrepid/80-install-kernel    consists  for  78   %  of  ./edgy/80-install-kernel      material
./common/30-disable-gettys      consists  for  81   %  of  ./karmic/30-disable-gettys    material
./common/90-make-fstab-rpm      consists  for  85   %  of  ./common/90-make-fstab-deb    material
./common/90-make-fstab-deb      consists  for  87   %  of  ./common/90-make-fstab-rpm    material
./gentoo/55-create-dev          consists  for  88   %  of  ./common/55-create-dev        material
./dapper/55-create-dev          consists  for  90   %  of  ./gentoo/55-create-dev        material
./gentoo/55-create-dev          consists  for  91   %  of  ./dapper/55-create-dev        material
./centos-5/90-make-fstab        consists  for  98   %  of  ./common/90-make-fstab-rpm    material
./common/90-make-fstab-rpm      consists  for  98   %  of  ./centos-5/90-make-fstab      material
./edgy/30-disable-gettys        consists  for  100  %  of  ./intrepid/30-disable-gettys  material
./intrepid/30-disable-gettys    consists  for  100  %  of  ./edgy/30-disable-gettys      material

Compared to that, fdupes only finds the two 100% identical files:

$ fdupes -r1 . 
./intrepid/30-disable-gettys ./edgy/30-disable-gettys 

But fdupes helped me already a lot to find the first bunch of identical files in xen-tools. :-)

I've had a little command line tool called errno for many years now. It's a short Python script, and lets me look up Unix errno names and values given values or names.

$ errno 2
ENOENT 2 No such file or directory
$ errno EEXIST
EEXIST 17 File exists
$

This is occasionally very useful when reading logs from various programs. I thought it might be useful for others as well, so I decided to rewrite it in C so I can offer it to become part of moreutils. It is now in the master branch of moreutils. Today I added a couple of options: --list lists all the errno values, and --search does a full text (case insensitive) search of all descriptions. I also added a script that lets you do the search in all installed locales. (These changes are waiting for Joey to merge them. Meanwhile, they're at gitorious.)

$ errno-all onnistu   sort -u
EDEADLOCK 0 Onnistui
ENOMEM 12 Muistin varaaminen ei onnistu
ENOTSUP 0 Onnistui
EWOULDBLOCK 0 Onnistui
$ 

(That's Finnish.) Hopefully others will find this useful as well.

Last week (28th May), I had five Internet services across two countries. This week, I have seven (7), with two new ones in Australia. It all begin, when Internode decided that in their own best interest to migrate me from my existing DSLAM to one of their own infrastructure. They had sent an email indicating when it was likely to occur. And they were sending invoices to the same address. Naturally, they would send the date of the migration to the same address they had used for the past 9 months to communicate to me with, right? Wrong. Internode still do not understand why they never sent the notification email, as it turns out. Compare the response time, update times, and restore times they specify to what actually occurred.

2012-05-28 12:05 BST

I'm informed of an outage. In the past, the problem has been at my end, so I run through the standard diagnostic procedure. Everything seems to be normal, so I call Internode. Sure enough, our service dropped off the Internet. According to Timothy, that happened at 11:15 AEST because of a migration to their own DSLAM. After some diagnosis from Internode, they realise that whilst the migration has successfully disconnected the service, they have not reconnected anything. There is no sync. Ticket #454448 raised - which is when I'm informed that I'm on the business SLA. I've asked for updated to both an Australian and UK mobile. "Not a problem". Estimated time of resolution: 48 hours. Yes, 48 hours. There goes the first SLA milestone.

2012-05-30 07:28 BST

I ring and speak with Carl. He indicates that as best he knows, the service is to be 7pm. for those keeping score, no updates have occurred during this time. To either number. I query him about the SLA period. He says the response time is 2hrs and restore time is 10hrs. He also indicates that this can all be checked online and that we can apply for a refund.

2012-05-30 10:27 BST

I ring, again, and this time speak with Justin. I ask why I've not have an updated, since 7pm AEST has well and truly gone. He authenticates me and places me on hold to investigate. Advises that the issue is fixed, says he sees sync on the line. I indicate that I do not. He asks me to turn off the ADSL modem, we do, and he still sees sync. on the line. Believes there is another problem at the exchange, another issue raised with wholesaler. No ticket number issues to me. ETA to fix 7pm 2nd June. Note:This was a 1 hour International call

2012-06-01 04:00 BST

I happened to check, and notice I have connectivity to the external interfaces (and thus, obviously, line sync.) again. No notification though.

2012-06-01 14:31 BST

Amidst all this, I also have to to source two external connections - Telstra & Optus, get them setup, reconfigure routing, email delivery and other network services so that the businesses which are relying on the Internet, still work. Not fun. Especially when I have no access to the routers. I had to get multiple people on site.

2012-06-04 11:XX CEST

I'm told that the second site has gone done. Inexplicably. After some round-trip and diagnosis, we determine that the routing change has occurred. However it has been done incorrectly. And, of course, no notice / or update occurred either. Multiple tickets are raised by multiple people, amongst them: #4562559 from my mobile phone whilst I am in Zurich.

2012-06-04 16:56 CEST

I am informed, and it is confirmed, that the attempted routing change has been reverted and we are back to how things were on 2012-06-01.

2012-06-05 05:44 CEST

I wake up, and find out that of at least three hours ago, they appear to have called (finally!) and actually fixed the routing to both sites. That is a week of down time, multiple people (4 people) around in various countries around the World and hundreds of dollars in calls between us all to get things working. Now, to look into compensation.

2012-06-05 09:43 CEST

I receive a missed call, which I later learn is Internode actually calling me (for once!). At that moment, I'm actually giving a lightening talk on Django-PaaS

And that, dear reader, is why Internode Business SLA is a waste of money. The services were broken by a change of no net benefit to me, was over a week ago. Also, during this process. My UK home Internet went down. My UK Three post-paid service "ran out of money" and Vodafone has a service outage for half a day. A perfect storm of Internet dis-connectivity. So, if you have been wondering, that is why I've been offline for various parts of the last week.

From time to time I need to convert scanned documents to a pdf format.


Usage scenario 1: I scan part of a book (i.e. some article) on a school's scanner that sends me 10 big separate color pdf files (one pdf per page). I want to get one nice, small (black and white) pdf file with all the pages.


Usage scenario 2: I download a web form, print it, fill it in, sign it, scan it on my own scanner using Gimp and now I want to convert the image into a nice pdf file (either color or black & white) to send back over email.

Solution: I save the original files (be it pdf or png) into a folder and use git to track it. Then create a simple reduce script to convert it to the final format (view it as a pipeline). Often I need to tweak one or two parameters in the pipeline.

Here is a script for scenario 1:
<script src="https://gist.github.com/2871625.js?file=reduce2.sh"></script>
And here for scenario 2:
<script src="https://gist.github.com/2871625.js?file=reduce1.sh"></script>
There can be several unexpected surprises along the way. From my experience:

• If I convert png directly to tiff, sometimes the resolution can be wrong. The solution is to always convert to ppm (color) or pbm (black and white) first, which is just a simple file format containing the raw pixels. This is the "starting" format (so first I need to convert the initial pdf or png into ppm/pbm) and then do anything else. That proved to be very robust.
• The tiff2pdf utility proved to be the most robust way to convert an image to a pdf. All other ways that I have tried failed in one way or another (resolution, positioning, paper format and other things were wrong....). It can create multiple pages pdf files, set paper format (US Letter, A4, ...) and so on.
• The linux convert utility is a robust tool for cropping images, converting color to black and white (using a threshold for example) and other things. As long as the image is first converted to ppm/pbm. In principle it can also produce pdf files, but that didn't work well for me.
• I sometimes use the unpaper program in the pipeline for some automatic polishing of the images.

...should be banned. On the flight back from Hong Kong to London, a collection of about half a dozen small children conspired to cry, shout, wail and shriek nigh-on constantly. For twelve hours. :-( I think it's incredibly selfish to take kids on planes when they're too young to understand what's going on, or to behave reliably. Children make noise, that's natural and expected. Forcing hundreds of other people around you to deal with that noise in an enclosed space for extended periods with no way to escape is just wrong, in my opinion. </rant>

Jo and I flew in to HK a few days early for the Linaro conference, so we got some time to ourselves for doing touristy stuff. That was a good plan - we got to see quite a lot of Kowloon for a couple of days, then we wandered over to Hong Kong island itself for a day. Jo also had more time during the week while I was in the sessions at the conference. Hong Kong is a busy place, very densely packed at all times of day and night that we could see. The first impression we got on the taxi from the airport on day one was that it was a city full of tower blocks - huge numbers of apartments stacked very closely together for living space. That makes sense, I guess, with the large population in such a small space. There are houses around with more space for people to live, but they are understandably very expensive. Later on in Kowloon, we got to meet lots of the local street vendors, mostly offering us "Copy Rolex", "Fake handbag" and the like. Even "Sharp trousers" at one point. We decided not to try them out... :-) We saw lots and lots of very expensive shops downtown, crammed into huge shopping centres in the tourist area. As an antidote for all of that, we headed up to the "Golden Computer Arcade" to go browsing around the vast array of tiny technology stalls and shops there. Picked up some nice cheap laptop memory and some SD cards there. Yay! Across on Hong Kong island itself on Sunday, we found my grandfather's grave then rode around on the trams for a while to get a good view of the city, including a democracy protest that happened to be on that day! We had a quick look around Tin Hau Temple, where we found a very specific set of rules for visitors:

I can only guess that they must have been terrified by gangs of kids racing their RC cars at one point! We finished the day by heading up to Victoria Peak for the sunset view and some dinner, then back to the hotel. Killing time in a shopping centre waiting for the bus to turn up, we found a truly astonishing little item on sale, a Solar Queen!

Apparently this gift has a solar panel on the handbag to provide power to a little motor in the Queen's hand to make her wave to her subjects. We decided that we'd somehow have to live without such a delight in our lives... :-) HK was a cool place to visit. It still feels very British in some ways, as you'd expect from an ex-colony (power sockets are still the same as ours, driving on the left, everybody spoke English, etc.) but utterly alien in others (the climate, working public transport *grin*). I look forwards to heading out there again!

Sometimes conferences can be dull and boring, but sometimes they can be just awesome in terms of finding the right people to collaborate with. Linaro Connect in Hong Kong last week was definitely one of the great ones! I chaired my my usual sessions (armhf status and cross-distro ARM) and we had some lively discussion in both. We're probably just about done with the armhf sessions, as most distros have accepted a hard-float ARMv7 port now and there's not so much specific work left there now that future sessions will be necessary. The cross-distro work for ARM ports is likely to continue into the future, but we're going to be concentrating on bootstrapping work for ARMv8 soon. Talking v8, there were a lot of meetings discussing the various work topics for this new 64-bit ARM architecture: kernel, toolchains, bootstrapping etc. More to come on that soon! On top of all this useful discussion and planning, we also found time during the week for some hacking. This was the highlight of the week for me, as I found some expert help to solve my long-standing Ruby on ARM bug (Debian bug #652674, ruby1.9.1: FTBFS on armhf: test suite segfaults). Ulrich Weigand (an IBM toolchain wizard seconded into the Linaro team) sat with me for a couple of hours while we worked through reproducing the problem, only to find that he could not reproduce it! The crash had looked very much like a pthreads locking bug, which was scaring me. After some digging, we worked out where the problem was, and how it was now fixed. For a long time on ARM, the Linux getcontext/setcontext system calls have never been implemented; apparently nobody really missed them, so they have never been a priority. In Ruby 1.9.x, the implementation of the new "fiber" primitive wanted to use getcontext/setcontext to control stack state etc. in different threads of control. In cases where they are not available or known not to work well, Ruby has fallback code to implement similar functionality. It seems that fallback code is buggy. Maybe it was correct at some point and has bit-rotted due to not being exercised, or maybe it was buggy as written, but it clearly does not work correctly for us now. In the last few months, getcontext/setcontext have finally been implemented for ARM in glibc trunk (by Michael Hope, also in the Linaro toolchain team!) and backported into current Debian and Ubuntu eglibc versions. Re-running configure and rebuilding Ruby against the most recent code in both Sid and Precise fixed the test suite crash we were seeing earlier. Yay! We could also provoke the bug again at will by quickly hacking around the Ruby source to force it to switch back to the fallback code, thereby verifying the fix. Why does this matter? As we're expecting ARM servers to enter the market soon, web apps written in Ruby on Rails are going to be an important part of the software stack that customers will want to run. Broken fibers and threading would not help here! It's great to meet up and work with talented folks like this; Linaro Connect is an excellent event for getting stuff done!