In no particular order, here are my 4 picks for the DPL thunderdome:

• algernon
• gregoa
• rra
• KiBi

I hope they all run. Really.

Garmin Montana notes Sometimes, I get waaay too excited about gadgets, and I have to share it. So, sorry, here it goes For some reason that is not very important, I was looking for a better solution for GPSes with maps. Side-note: if you ask why Android is not good enough , here's the story: last year, knowing that Google Maps supports caching (finally), I upgraded Maps, preloaded London, and went on a trip. Only to exit the Victoria underground station and to stare ~20 minutes in light rain at my phone, wondering why it doesn't get a fix. I gave up in the end, and used the cached maps for getting around. Later I learned the reason: Maps was not broken, but without data, Android can't get a lock easily; and more-over, the recent Maps update without GPS active even once afterwards, to do whatever it needed, meant I couldn't get a lock until back in Switzerland. Or, alternatively: while on a trip to Japan, it turns out that Google Maps caching is not supported there, probably because of rights over mapping data. So I had location, but only temporary cached maps (from the wifi in the hotels). Due to this kind of problems makes I consider Android maps a toy, so I wanted a GPS device with proper maps (on the device, not in the cloud), and with good GPS capabilities (no fancy A-GPS or anything, just plain good standalone GPS). Having had good experiences with the sport line of Garmin (but which doesn't have maps, so it's out of the question), I looked at Garmin. Many people complained about some things (same all over the place):

• newer devices less tested, software with lots of bugs
• dumbing down of devices, ultra-simplification of the UI

The latter is something that jumped at me from many of the blogs/fora that I read; I like customisable power tools , not one-button mice. The other thing that I didn't really like was that most devices are very focused on a niche: the n vi line on automotive, the Zumo line on motorbikes, and the hand-held lines with some blurring of hiking/geocaching/etc. So no one-size fits all device Features! Customisation! That is, until I found out the Montana series. My my, I was blown. Both by the price (I couldn't believe how expensive it was, without maps!) and by the features. As it turns out, the Montana has different profiles , and it can act:

• as a hand-held device for hiking/geocaching (yay!)
• as a bicycle-mounted device for biking through the woods (yay!)
• as a motorcycle-mounted device (hmmm )
• as a car-mounted device, for automotive use, including guidance, both TTS and non-TTS voices, lane assist, speed limits, etc.

That is for flexibility. As for performance/features:

• it has a Transflective liquid crystal display screen, so it works best (as opposed to worst) in bright sun, and of course also works in the dark, via its back light
• the touchscreen is resistive, so glove-friendly: tested it with my regular gloves on and with my biking gloves (which are very clumsy), and it works like a charm
• it is water-proof (IPX7); that means I can keep it in the open in the rain, and not worry about dropping it in the snow
• it has both a (replaceable) LiIon battery (many hours of runtime, > 10) and support for powering via 3 AA batteries (> 20 hours), which coupled with some Eneloops is awesome
• GPS sync is awesome; from hard poweroff, it syncs in ~3 seconds I would say (exited building, powered on, was already synced by the time I entered the satellite menu); even in-house, it gets around 2 satellites, and sometimes manages to get a lock

Now for the downsides. Being a multi-functional device, it's not as good as a dedicated one:

• for handheld use, it's a bit heavy; I personally don't mind, but it's something that has to be said: ~300 g
• for motorcycle use, it's worse than the Zumo: no bluetooth compatibility, so you can't have it talk to you in the helmet; but I don't have a motorcycle, so
• for automotive use, it lacks many things compared to a top-of-the-live n vi: no voice recognition, no junction view, no birds-eye view, etc.; but I don't have a car, so

But the small (for me) draw-backs aside, it's definitely not a dumbed-down device. It's the first device that I have which I feel it a bit too customisable . From completely changeable menus, to profiles that remember their maps selection, to many dashboards, to almost-programmable shortcuts (e.g. a shortcut to change profile, set destination to a given way point, reset the trip log, and turn track logging on) a bit overwhelming at first. Add the 3-axis compass, barometric altimeter, ANT+ support and you get a very powerful device. Software As to the software, yes it has some bugs - mainly that the device crashes sometimes (abruptly) if you play too much with the routes (didn't happen to me yet on the road , only when testing it at home). On the PC-side, Garmin's BaseCamp is nice, but doesn't hold a candle to Google Maps, for example, in terms of UI; I don't mean shininess, but simply how well the UI works. The most annoying thing is how it doesn't scale detail appropriately with the zoom level. But it works, and I can plan hikes offline, and upload both tracks and routes to it (this is important: it seems other recent Garmins can't replay tracks, only routes). And I can load many maps, and toggle them at will, etc. The many maps feature is not to be overlooked. I could have, for example, standard Garmin maps loaded, but just in case they're insuficiently detailed or out of date (hey, Navteq is slow ), I can also have Open Street Maps from two days ago. The routing is then done on the currently active maps, so you are not restricted to a single mapping provider. And of course you can also have custom POIs (Point of interest) on it. Bycicle use The bicycle mount is solid enough, so that on a rough roads it's held firmly. From my ~20 minute test this morning, a few things to note:

• it wanted me to take bigger paths, rather than the trails I like to take; probably I haven't customised it well enough yet
• recalculating a route (when it realises you insist on taking another path than it planned) is very fast
• whenever you approach a change-of-direction, it beeps (see below) and turns on the back light (temporarily), so visibility is improved
• you can actually use the touchscreen with winter gloves (of course, when stopped or at very very low speed)

Voice output By default, the unit can only beep, but if coupled via a 3.5" jack to headphones or via the automotive mount, it can properly speak; quality varies with voice, and whether it is TTS or not. I managed to find one voice (non-TTS) that sounds really well, but of course with less information, and one TTS voice that sounds OK-ish but gives full info. By full info , I mean the following: the behaviour of the various TTS voices is not consistent. For example, on a simulated route, various voices were saying the following:

• non-TTS: keep ahead (voice A), or keep-ahead, then keep right (voice B)
• voice C, TTS: in 94 kilometers, exit right
• voice D, TTS: in 94 kilometers, take exit 14 on the right
• voice E, TTS: in 94 kilometers, take exit 14 on the right in the direction Landquart, then keep right

I don't know why these differences, or if they are consistent across routes. But I'll keep with voice E for now Summary So, sorry again for sounding like an ad (I don't have any relation with Garmin the company), but I find this device awesome - powerful, rugged, and working (so far) very well. I'm glad that there still exist companies who don't believe they know better than their customers, and instead offer you with a tool that you can make it work just like you want. Or almost Garmin's tag line: Big and Tough Goes anywhere, everywhere . I tend to agree

Technology Technology is... weird. I am in a cabbin in the middle of Lapland an hour away from the next city to speak of. GPS

N 66 15.958
E 24 59.981

look it up. We haven't had power for the last 60 minutes, the electrical heaters are out and we are trying to keep the main room warm with a wood stove while the wood-burning sauna above is ready to receive us the minute I hI push this post. The cabbin has become noticiably colder already, supply of logs small enough to actually burn is limited, we don't have an axe, but we have a car so we can leave if need be. The next house is about six kilometers away, no idea if they have power or not; the woods are dark all round other than for snowshine. Yet, I am sitting here, LED lamp lighting the room, GPS telling me what location to write into this post, laptop running on battery, cell phone with EGPRS supplying me with bugstats plus picture (even though I had to do most of it by hand as LWP::Simple barfs on the timeout) and allowing me to push to ikiwiki. As I said, technology is... weird. Bugology The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 786
  • Affecting wheezy: 194 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 128 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 34 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 12 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 90 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting wheezy only: 66 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 40 bugs are in packages that are unblocked by the release team.
      • 26 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Diff
43	284 (213+71)	468 (332+136)	+184 (+119/+65)
44	261 (201+60)	408 (265+143)	+147 (+64/+83)
45	261 (205+56)	425 (291+134)	+164 (+86/+78)
46	271 (200+71)	401 (258+143)	+130 (+58/+72)
47	283 (209+74)	366 (221+145)	+83 (+12/+71)
48	256 (177+79)	378 (230+148)	+122 (+53/+69)
49	256 (180+76)	360 (216+155)	+104 (+36/+79)
50	204 (148+56)	339 (195+144)	+135 (+47/+90)
51	178 (124+54)	323 (190+133)	+145 (+66/+79)
52	115 (78+37)	289 (190+99)	+174 (+112/+62)
1	93 (60+33)	287 (171+116)	+194 (+111/+83)
2	82 (46+36)	271 (162+109)	+189 (+116/+73)
3	25 (15+10)	249 (165+84)	+224 (+150/+74)
4	14 (8+6)	244 (176+68)	+230 (+168/+62)
5	2 (0+2)	224 (132+92)	+222 (+132/+90)
6	release!	212 (129+83)	+212 (+129/+83)
7	release+1	194 (128+66)	+194 (+128/+66)

Graphical overview of bug stats thanks to azhag:

Silly question: when you use diff(1) to produce a file to be applied with patch(1), do you call the result a diff or a patch? Do you name it foo.diff or foo.patch?

In January, I did a TV-shop ad -style post on a little script called lintian-overrider and it prompted Simon to ask:

That s a great tool, but don t you fear it makes unjustified overrides too easy ?

In my experience, people sometimes have issues writing overrides (justified or not). In fact it sometimes leaves them really frustrated with Lintian. If that frustration eventually leads to them reject Lintian, then we are doing the project a disservice. I will quote Russ Allbery as I believe he covered when he wrote:

I care most about all of the regular Debian developers [...] continuing to use Lintian, so that Lintian can stay as effective as it is now at getting people to make archive-wide changes. [...] This only works if we can get nearly everyone uploading packages to run Lintian all the time.

If a handful of people adds overrides for tags they should not have, then we can fix that (e.g. by filing a bug against their packages). But it is unlikely that we will ever make them run Lintian again once they have boycott it.

In his latest bits from the DPL, Stefano wrote:

I'd like to respond (also) here to inquiries I'm receiving these days: I will not run again as DPL. So you have about 20 days to mob\^Wconvince other DDs to run, or decide to run yourself. Do not to wait for the vary last minute, as that makes for lousy campaigns.

Ladies and gentlemen, I am pleased to present you... THE DPL GAME GOALS:
The goal of the game is to let people know you think they'd be nice DPLs.
The point is not to pressure them, but to let them know they're awesome and make them at least consider the idea to run for DPL. The winners are those who have at least one of their Fantastic Four running for DPL. Bonus points if one of them ends being the next DPL. RULES:
Name three persons (plus a reserve, just in case) you'd like to see as candidates for DPL. Publicly list them (on your blog or on identi.ca using the hashtag #DPLgame) or at least let them know that you'd like to have them as candidate for DPL (via private mail).
You may want to add a couple of lines explaining the rationale for your choices. AGE:
0-99 NUMBER OF PLAYERS:
The more the merrier Some suggestions on how to play:
First think of the qualities a DPL needs to do, in your opinion, a good job. Then look around you: the people you work with, the people you see interact on mailing list, etc. There must be someone with those qualities.

---

Here are my Fantastic Four (in rigorous alphabetic order):

• Russ Allbery
• Gregor Herrmann
• Christian Perrier
• Martin Zobel-Helas

In my opinion, they all more or less have: enthusiasm, a general understanding of dynamics inside the project and of various technical sides of the project itself, ability to delegate and coordinate with different people (inside and outside the project), good communication skills and some diplomacy and ability in de-escalating conflicts. These are people I worked with or I observed working and discussing on mailing lists, and I think they'd do a good job. But -hey!- we are almost a thousand of developers and you cannot possibly know everyone or observe all the people who work in the various teams. This is why you should pick your four names!

As you have probably already noticed, Google Summer of Code 2013 has been officially announced. As usual, we want to participate with phpMyAdmin. We really want good students to participate, so our selection process will be again slightly stricter than in last years. So if you want to participate as a student on phpMyAdmin, please check our applicant guide. The most important thing is that we expect you to show knowledge of the code before being accepted and fixing some bug for upcoming 4.0 release is great opportunity to start with that! Also student themselves usually don't bring new ideas, so it's pretty much on the organization to come with projects it would like to see. We are currently collecting these on our wiki. We will most likely add some new ones to the list as well, so feel free to get in touch if you have some brilliant idea. Also voting on feature requests might help us to choose what users see as most important improvements.

Filed under: English Phpmyadmin 0 comments Flattr this!

If you, like me, want an updated a map for your Garmin GPS, there is now a great source of free maps available from Frikart. To download a map, just click on the country you are interested in, and download the map type you want. There are 8 different maps available, using different colours and data selection. Pick one of Roadmap, Topo Summer, Topo Winter, Roadmap II, Topo Summer II, Topo Winter II, "Trails - overlay map" and "Cross country - overlay map" (see the web page for descriptions). The maps are updated weekly, so if you find something wrong in the map you can just edit the OpenStreetmap map source (anyone can contribute) and fetch a fixed map a week later. :)

Of course, as happens sadly often, the scope creeps.. Additional pain points Zope s test runner runs things that are not tests, but which users want to know about layers . At the moment these are reported as individual tests, but this is problematic in a couple of ways. Firstly, the same test runs on multiple backend runners, so timing and stats get more complex. Secondly, if a layer fails to setup or teardown, tools like testrepository that have watched the stream will think a test failed, and on the next run try to explicitly run that test but that test doesn t really exist, so it won t run [unless an actual test that needs the layer is being run]. Openstack uses python coverage to gather coverage statistics during test runs. Each worker running tests needs to gather and return such statistics. The current subunit protocol has no space to hand this around, without it pretending to be a test [see a pattern here?]. And that has the same negative side effect test runners like testrepository will try to run that test . While testrepository doesn t want to know about coverage itself, it would be nice to be able to pass everything around and have a local hook handle the aggregation of that data. The way TAP is reflected into subunit today is to mangle each tap test into a subunit test , but for full benefits subunit tests have a higher bar they are individually addressable and runnable. So a TAP test script is much more equivalent to a subunit test. A similar concept is landing in Python s unittest soon subtests which will give very lightweight additional assertions within a larger test concept. Many C test runners that emit individual tests as simple assertions have this property as well there may be 5 or 10 executables each with dozens of assertions, but only the executables are individually addressable there is no way to run just one assertion from an executable as a test . It would be nice to avoid the friction that currently exists when dealing with that situation. Minimum requirements to support these Layers can be supported via timestamped stdout output, or fake tests. Neither is compelling, as the former requires special casing in subunit processors to data mine it, and the latter confuses test runners. A way to record something that is structured like a test (has an id the layer, an outcome in progress / ok / failed, and attachment data for showing failure details) but isn t a test would allow the data to flow around without causing confusion in the system. TAP support could change to just show the entire output as progress on one test and then fail or not at the end. This would result in a cognitive mismatch for folk from the TAP world, as TAP runners report each assertion as a test , and this would be hidden from subunit. Having a way to record something that is associated with an actual test, and has a name, status, attachment content for the TAP comments field that would let subunit processors report both the addressable tests (each TAP script) and the individual items, but know that only the overall scripts are runnable. Python subtests could use a unique test for each subtest, but that has the same issue has layers. Python will ensure a top level test errors if a subtest errors, so strictly speaking we probably don t need an associated-with concept, but we do need to be able to say that a test-like thing happened that isn t actually addressable. Coverage information could be about a single test, or even a subtest, or it could be about the entire work undertaken by the test process. I don t think we need a single standardised format for Coverage data (though that might be an excellent project for someone to undertake). It is also possible to overthink things . We have the idea of arbitrary attachments for tests. Perhaps arbitrary attachments outside of test scope would be better than specifying stdout/stderr as specific things. On the other hand stdout and stderr are well known things. Proposal version 2 A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, file, test), test-id, runnable, test-status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The std/stdout/stderr content tags are gone, replaced with file. The names stdin,stdout,stderr can be placed in the attachment name field to signal those well known files, and any other files that the test process wants to hand over can be simply embedded. Processors that don t expect them can just pass them on. Runnable is a boolean, indicating whether this packet is describing a test that can be executed deliberately (vs an individual TAP assertion, Python sub-test etc). This permits describing things like zope layers which are top level test-like things (they start, stop and can error) though they cannot be run.. and it doesn t explicitly model the setup/teardown aspect that they have. Should we do that? Testid is for identifying tests. With the runnable flag to indicate whether a test really is a test, subtests can just be namespaced by the generator reporters can choose whether to be naive and report every test , or whether to use simple string prefix-with-non-character-seperator to infer child elements. Impact on Python API If we change the API to:

class TestInfo(object):
    id = unicode
    status = ('exists', 'inprogress', 'xfail', 'xsuccess', 'success', 'fail', 'error', 'skip')
    runnable = boolean
class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        passs
    def estimate(self, count, route_code=None, timestamp=None):
        pass
    def file(self, name, bytes, eof=False, mime=None, test_info=None, route_code=None, timestamp=None):
        """Inform the result about the contents of an attachment."""
    def status(self, test_info, route_code=None, timestamp=None):
        """Inform the result about a test status with no attached data."""

This would permit the full semantics of a subunit stream to be represented I think, while being a narrow interface that should be easy to implement. Please provide feedback! I ll probably start implementing this soon.

To convert video files between formats I use Makefiles, this means I can run make -j2 on my dual-core server to get both cores going at once. avconv uses 8 threads for it s computation and I ve seen it take up to 190% CPU time for brief periods of time, but overall it seems to average a lot less, if nothing else then running two copies at once allows one to calculate while the other is waiting for disk IO. Here is a basic Makefile to generate a subdirectory full of mp4 files from a directory full of flv files. I used to use this to convert my Youtube music archive for my Android devices until I installed MX Player which can play every type of video file you can imagine [1]. I ll probably encounter some situation where this script becomes necessary again so I keep it around. It s also a very simple example of how to run a batch conversion of video files. MP4S:=$(shell for n in *.flv ; do echo $$n sed -e s/^/mp4\\// -e s/flv$$/mp4/ ; done)

all: $(MP4S)

mp4/%.mp4: %.flv
avconv -i $< -strict experimental -b $$(~/bin/video-encoding-rate $<) $@ > /dev/null Here is a more complex Makefile. I use it on my directory of big videos (more than 1280*720 resolution) and scales them down for my favorite Android devices (Samsung Galaxy S3, Samsung Galaxy S, and Sony Ericsson Xperia X10). My Galaxy S3 can t play a FullHD version of Gangnam Style without going slow so I need to do this even for the fastest phones. This makefile generates three subdirectories of mp4 files for the three devices. S3MP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/s3\\// -e s/.mp4$$/-s3.mp4/ -e s/.flv$$/-s3.mp4/; done)
XPERIAMP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/xperiax10\\// -e s/.mp4$$/-xperiax10.mp4/ -e s/.flv$$/-xperiax10.mp4/; done)
SMP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/galaxys\\// -e s/.mp4$$/-galaxys.mp4/ -e s/.flv$$/-galaxys.mp4/; done)

all: $(S3MP4S) $(XPERIAMP4S) $(SMP4S)

s3/%-s3.mp4: %.mp4
avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 1280 720 $<) $@ > /dev/null

galaxys/%-galaxys.mp4: %.mp4
echo avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 800 480 $<) $@ > /dev/null

xperiax10/%-xperiax10.mp4: %.mp4
echo avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 854 480 $<) $@ > /dev/null The following script is used by the above Makefile to determine the resolution to use. Some Youtube videos have unusual combinations of width and height (Linkin Park seems to like doing this) so I scale them down so it fits the phone in one dimension and the other dimension is scaled appropriately. This requires a script from the Mplayer package and expects it to be in the location that it s used in the Debian package, for distributions other than Debian a minor change will be required. #!/bin/bash
set -e
OUT_VIDEO_WIDTH=$1
OUT_VIDEO_HEIGHT=$2

eval $(/usr/share/mplayer/midentify.sh $3)
XMULT=$(echo $ID_VIDEO_WIDTH*100/$OUT_VIDEO_WIDTH bc)
YMULT=$(echo $ID_VIDEO_HEIGHT*100/$OUT_VIDEO_HEIGHT bc)
if [ $XMULT -gt $YMULT ]; then
NEWX=$OUT_VIDEO_WIDTH
NEWY=$(echo $OUT_VIDEO_WIDTH*$ID_VIDEO_HEIGHT/$ID_VIDEO_WIDTH/2*2 bc)
else
NEWX=$(echo $OUT_VIDEO_HEIGHT*$ID_VIDEO_WIDTH/$ID_VIDEO_HEIGHT/2*2 bc)
NEWY=$OUT_VIDEO_HEIGHT
fi
echo $ NEWX x$ NEWY Note that I can t preserve TAB characters in a blog post. So those Makefiles won t work until you replace strings of 8 spaces with a TAB character.

• [1] https://play.google.com/store/apps/details?id=com.mxtech.videoplayer.ad

I wrote that I would double-check how secure the module selection and downloading is in Puppet. Well, puppet module resolves, fetches and downloads unsigned tarballs from a HTTP source and unpacks them without any verification whatsoever. Related: I've been looking at rpm/yum GPG behaviour. rpm supports checking the signature of RPMs as a separate operation from installing them. You can't ask it to not install a package if the signature is absent or not correct. yum is better when dealing with repositories. It can be told to check the GPG signature on all RPMs both globally (the [main] section of yum.conf) and on a per-repository basis. GPG signature checking can be disabled on the command line with --nogpgcheck. It cannot be selectively enabled on the command line. However, yum install can install local RPMs and RPMs on web servers as well as from repositories. In both of these cases, it will not check the GPG signature at all, no matter what you've put in your yum.conf. Finally, even if all the above worked properly, the GPG keys published by Fedora have almost no public signatures (none at all for EPEL), so neither you nor I could establish a trust path to them. Luckily I can establish a trust path to the RH security key.

So as previously mentioned I gave a talk at FOSDEM this year on the future of paravirtualisation under Xen. I thought the talk was reasonably well attended despite being at six o'clock in the evening. In the end I underran my half hour slot but fortunately people had plenty of interesting questions! I've posted my slides on my xenbits space. I spent a fair chunk of the rest of the time on the Xen.org booth, which is hard work but it is always nice to speak to users and other developers. I also managed to catch up with various friends from other projects and managed to meet a few folks who I've only ever met in email before. I didn't get to see as many talks as I wanted (for example I missed pretty much all of the ARM track :-() due to clashes with my stints on the booth, my own talk, and talks clashing with each other. Still it was all in all a worthwhile, hopefully I'll be able to make it again next year! Next up: LinaroConnect in Hong Kong!

Subunit is seven and a half years old now Conrad Parker and I first sketched it up at a CodeCon camping and coding, a brilliant combination in mid 2005.

revno: 1
committer: Robert Collins <robertc@robertcollins.net>
timestamp: Sat 2005-08-27 15:01:20 +1000
message:  design up a protocol with kfish

It has proved remarkably resilient as a protocol the basic nature hasn t changed at all, even though we ve added tags, timestamps, support for attachments of arbitrary size. However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format. Existing design The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of make or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined. This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity but also requires that state be buffered until the end of a test, as two tests cannot be executing at once. Dealing with debuggers The initial protocol was intended to support dropping into a debugger just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like. All in-principle fixable, right ? When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else. We ve added some idioms in testrepository that control test execution by a similar but different format one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying you need two parsers, and two output formats. Good points The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility). Pain points The debugging support was a total failure, and the way the parser depraminates it s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout). Dealing with concurrency The original design didn t cater for concurrency. There are three concurrency issues the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues? Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we ve lost information. Or at best muddled it all up. Secondly, buffering imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1 s test is taking more than 5 seconds or something but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer. Thirdly, scheduling if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time. Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn t help when it is W1 and W2 reading from stdin. So additional requirement over baseline subunit:

1. make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well the protocol doesn t need to care about that, just make it possible.
2. make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
3. make it possible (but optional) to pass tests to run to a running process that accepts subunit.
4. make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)

Dealing with corruption Consider the following subunit fragment:

test: foo
starting serversuccess:foo

This is a classic example of corruption: the test foo started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like make runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn t finish it. That looks like:

test: foo
test: bar
success: bar

One way that this sort of corruption can be mitigated is to put subunit on it s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn t solve the failing test script case. I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work so the make -j2 case can be ignored at the wire level though we should create a simple way to safely mux the output from such tests when the execute. The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path. Dependencies Various suggestions have been made JSON, Protobufs, etc A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal. High level proposal A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The content tags:

• estimate the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate TestSuite objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
• stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
• test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.

Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed. Attachment names must be unique per routing code + testid. So how does this line up? Interleaving and recovery We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data. listing Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once. Selecting tests to run Same as for listing while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band. Debugging Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append 1 or 2 to select which of W1/W2 the content came from, and then forward the packet. S would append 1 or 2 to indicate I1/I2 the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question. Multiplexing Very straight forward unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for). Parsing / representation in Python or other languages The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams. Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:

• Concurrent tests (e.g. in multiprocessing) wouldn t need multiplexers and special adapters a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
• The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
• The listing facility and incrementally running tests would be useful too I think we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult

The API might be something like:

class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        pass
    def estimate(self, count, route_code=None):
        pass
    def stdin(self, bytes, route_code=None):
        pass
    def stdout(self, bytes, route_code=None):
        pass
    def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None):
        pass

This would support just-in-time debugging by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently Generation new streams Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. test foo completed + some attachments or test foo starting .

Over the last few days I've done quite a lot of work to try and get our puppet configuration up to modern best practises. The Puppet Labs folks strongly encourage you to make as much use of puppet modules as possible. A puppet module gathers together puppet manifests, facter facts and other bits and pieces into a reusable component that you could potentially share with others. Many modules (of very mixed quality) are available on the web, in particular at github and via Puppet Lab's own Forge. Since version 2.7.14, the puppet command-line tool has built-in support for managing modules:

# puppet module list
/etc/puppet/modules
  auth (???)
  interfaces (???)
 
# puppet module install puppetlabs-apt
 

However, they have not provided support for managing modules in puppet manifests themselves. This strikes me as a bit odd: the whole point of using puppet to manage your machines is to capture the configuration in one place. If your configuration depends on a collection of modules and module versions, you'd ideally record that in the puppet configuration itself. Otherwise building a new puppet master is a mixture of manually installing modules and setting it up as a client of your existing master. Other people to the rescue: Ryan Coleman (and Pieter van de Bruggen) have written puppet_module_provider which does exactly this. (Note that they both work for Puppet Labs). Now I can define which modules and what versions are necessary:

class puppetmaster  
   
  module   'rcoleman/puppet_module': ensure => '0.0.3',  
  module   'puppetlabs/firewall':    ensure => '0.0.4',  
  module   'puppetlabs/lvm':         ensure => '0.1.1',  
  module   'puppetlabs/ntp':         ensure => '0.2.0',  
   

I've decided to pin specific versions given the highly variable nature of module quality. There's no guarantee of backwards compatibility except by reputation of the publisher. The puppetlabs modules are likely to be of higher quality than average, so over time I'll probably gain confidence to change the specific versioning to 'latest' or similar. I also want to double-check how secure the module selection and downloading is (whether it's over HTTPS or cryptographic signatures are checked etc.)

Next round of phpMyAdmin 4.0 translation status report is coming. So let's look at which translations are at 100% right now (some changes this week due to new strings to translate):

• Czech
• Dutch
• English (United Kingdom)
• French
• Greek
• Russian
• Spanish
• Turkish

There are also several languages which need just few strings to be complete:

• Danish
• Estonian
• Galician
• German
• Japanese
• Polish
• Portuguese (Brazil)
• Slovenian
• Swedish

As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin. Also you are welcome to translate phpMyAdmin 4.0 using translation server. If your language is already fully translated and you want to help as well, you can translate our documentation as well.

Filed under: English Phpmyadmin 0 comments Flattr this!

From time to time it occurs that two people answer a mail in the same way where one would do closing an unblock request, for example. When this almost happened on debian-release the other day I amused myself by dreaming up an SMTP header that would prevent such embarrassment. I wasn t being serious in the slightest, but nevertheless X-RaceProtection was born (and it turns out at least one resident of a certain IRC channel thought I was).

X-RaceProtection can be a message identifier or the simple value yes and is intended to prevent duplicate replies to, for example, mailing lists. When set as a mail ID, list software should silently drop the message being delivered if the identified message has already received a reply that is, another message quoting that ID in In-Reply-To. If X-RaceProtection is simply yes , the mail ID of In-Reply-To for the message being delivered is used, providing a shortcut.

This means you can set X-RaceProtection when replying to a mail where there is a chance of collision. If someone beat you to it, there is no embarrassment at your mail arriving with a later timestamp. If someone fancies implementing this for smartlist/debbugs, please be my guest!

---

X-RaceProtection: yes is a post from: jwiltshire.org.uk Flattr

Brett points at a Guardian article on Terry Deary and his opinion that libraries are no longer relevant. In fact, Mr. Deary goes further: he thinks libraries are destroying the book business by preventing authors, publishers, and bookshops from getting paid. From the article:

As one of the most popular library authors his books were borrowed more than 500,000 times during 2011/12 Deary will have received the maximum amount possible for a writer from the Public Lending Right scheme, which gives authors 6.2p every time one of their books is borrowed, up to a cap of 6,600. "If I sold the book I'd get 30p per book. I get six grand, and I should be getting 180,000. But never mind my selfish author perception what about the bookshops? The libraries are doing nothing for the book industry. They give nothing back, whereas bookshops are selling the book, and the author and the publisher get paid, which is as it should be. What other entertainment do we expect to get for free?" he asked.

He's making several of the same false claims that people in the media business always make whenever they see people enjoying any kind of cultural products without paying for them.

1. Every loan from a library is not a lost sale. More than 500 thousand loans does not mean 500 thousand lost sales. When borrowing a book is free, people will do it even if they aren't sure if they're going to like a book. The cheapest book by Mr. Deary I Amazon UK is 4.27 when sold as new by amazon (I'll ignored used books for the moment). The threshold for buying a book at that price is much, much higher than borrowing on for free.
2. Letting people have stuff for free does not automatically mean they won't pay for other stuff. Indeed, they may well pay for the very stuff they got for free. See the several experiments to let customers set their own price for music, books, or computer games, or the success Cory Doctorow has letting people download e-books of his novels for free, and then letting people donate one or more hardcopies of the book to schools and libraries. This is not a model that will work well for everyone, but it proves the point: charging people money for each copy is not necessary to make good profit. In fact, studies of "Internet pirates", or people who download music and movies (and books) more or less illegally from the Internet, show that such people tend to spend more money on music and movies (and books!) than the average person. That's not something the publishers want to talk about, or hear. (I am talking here about real research, not the propganda material that lobbying organisations produce.)
3. Why would people buy a book in the first place? They need to know they'll like it, but more importantly, they need to like reading in the first place. See point 1: if books cost any amount of money, the threshold for getting one is high. If you don't have easy, convenient access to books when you're a child, you're unlikely to be a reader of books when you're an adult. Public libraries are the perfect way to give people access to a wide variety of literature. In fact, they're about the most perfect marketing ploy in the history of media: the libraries pay for the books, and they pay each time they loan out a book, and every time a new future customer is brainwashed that much more into getting more stuff in the future. Libraries are not just for children, of course. Adult readers use them as well, and they're just as important as children, as far as the marketing ploy is concerned.

It's a bit sad that bookshops are dying out. Public libraries have, however, nothing to do with it. Online booksellers are usually cheaper, and have vastly larger catalogues, and that's a much more likely culprit. Also, bookshops have no intrinsic right to exist: they're middle-men who make a living by making it more convenient for customers to get what they want, and if they get out-competed, that's business life. Now, I neglected above to consider the possibility of buying used books. The reason for that is that media companies, and their lobbying organisations, would like to make selling of used books, music recordings, movies, and computer games not happen anymore. For example, Sony, and files patents for, stopping people from playing used games on its game consoles, even without Internet access. Mr. Deary seems to be following the MPAA and RIAA checklist on how to talk about those evil people who enjoy something without paying for it every time. It can't be long before he starts asking for a ban on selling used books. Let's hope, however, that reality and facts and sense win this time, and that we can let Mr. Deary live alone in his little horrible hallucination.

At FOSDEM I was talking even two times about Debian Med. The talk in the morning was attached to the FOSS for scientists track and was only a short overview about the use scientists can make from Debian Med. As usual the slides are available from my web page.
The main talk was in the evening in the Cross Distro Devroom and it was even video recorded.
It was fun to be at FOSDEM for the first time and meet a lot of Debian people and discuss new ideas with them.