Subunit is seven and a half years old now Conrad Parker and I first sketched it up at a CodeCon camping and coding, a brilliant combination in mid 2005.
revno: 1
committer: Robert Collins <robertc@robertcollins.net>
timestamp: Sat 2005-08-27 15:01:20 +1000
message: design up a protocol with kfish
It has proved remarkably resilient as a protocol the basic nature hasn t changed at all, even though we ve added tags, timestamps, support for attachments of arbitrary size.
However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format.
Existing design
The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of make or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined.
This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity but also requires that state be buffered until the end of a test, as two tests cannot be executing at once.
Dealing with debuggers
The initial protocol was intended to support dropping into a debugger just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like. All in-principle fixable, right ?
When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else.
We ve added some idioms in
testrepository that control test execution by a similar but different format one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying you need two parsers, and two output formats.
Good points
The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility).
Pain points
The debugging support was a total failure, and the way the parser depraminates it s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout).
Dealing with concurrency
The original design didn t cater for concurrency. There are three concurrency issues the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues?
Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we ve lost information. Or at best muddled it all up.
Secondly, buffering imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1 s test is taking more than 5 seconds or something but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer.
Thirdly, scheduling if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time.
Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn t help when it is W1 and W2 reading from stdin.
So additional requirement over baseline subunit:
- make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well the protocol doesn t need to care about that, just make it possible.
- make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
- make it possible (but optional) to pass tests to run to a running process that accepts subunit.
- make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)
Dealing with corruption
Consider the following subunit fragment:
test: foo
starting serversuccess:foo
This is a classic example of corruption: the test foo started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like make runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn t finish it. That looks like:
test: foo
test: bar
success: bar
One way that this sort of corruption can be mitigated is to put subunit on it s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn t solve the failing test script case.
I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work so the make -j2 case can be ignored at the wire level though we should create a simple way to safely mux the output from such tests when the execute.
The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path.
Dependencies
Various suggestions have been made JSON, Protobufs, etc
A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal.
High level proposal
A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes.
The content tags:
- estimate the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate TestSuite objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
- stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
- test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.
Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed.
Attachment names must be unique per routing code + testid.
So how does this line up?
Interleaving and recovery
We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data.
listing
Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once.
Selecting tests to run
Same as for listing while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band.
Debugging
Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append 1 or 2 to select which of W1/W2 the content came from, and then forward the packet. S would append 1 or 2 to indicate I1/I2 the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question.
Multiplexing
Very straight forward unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for).
Parsing / representation in Python or other languages
The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams.
Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:
- Concurrent tests (e.g. in multiprocessing) wouldn t need multiplexers and special adapters a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
- The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
- The listing facility and incrementally running tests would be useful too I think we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult
The API might be something like:
class StreamingResult(object):
def startTestRun(self):
pass
def stopTestRun(self):
pass
def estimate(self, count, route_code=None):
pass
def stdin(self, bytes, route_code=None):
pass
def stdout(self, bytes, route_code=None):
pass
def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None):
pass
This would support just-in-time debugging by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently
Generation new streams
Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. test foo completed + some attachments or test foo starting .
I'll soon be disposing of the Nissan Leaf that we leased a few
months ago, so I thought it a useful exercise to write about my experiences
with it.
I am not a car man. I am a gadget man. For me, driving is a means to an end,
and I'm much more interested in what I call the "cabin" experience, than the
"driving" experience, so this is going to slanted much more that way.
That said, I found the Nissan Leaf a fun car to drive, from my limited
experiences of having fun driving cars. I liked how responsive it was when
you put your foot down. It has two modes of operation, "D" and "Eco". I've
actually taken to driving it in "Eco" mode most of the time, as it squeezes
more range out of the batteries, but occasionally I'll pop it back into "D"
to have a bit of fun.
The main difference between the two modes, from a driving perspective, is it
seems to limit the responsiveness of the accelerator. In "Eco" mode it feels
more like you're driving in molasses, whereas in "D" mode, when you put your
foot down, it responses instantly. "D" is great for dragging people off at
the lights. It's a very zippy little car in "D" mode. It feels lighter.
I've noticed that it has a bit of a penchant for over steering. Or maybe
that's just my driving. If I have floored it a bit to take a right turn into
oncoming traffic, I've noticed slight over steering.
That's about all the driving type "real car stuff" I'll talk about.
Now to the driver's "cabin experience".
It's absolutely fabulous. I love sitting in the driver's seat of this car.
Firstly, the seat itself is heated (in fact all of them are). As is the
steering wheel. Nissan has gone to great lengths to allow you to avoid
needing to run the car's heating system to heat the car, as doing so
immediately drops at least 10 miles off the range. Unfortunately I found the
windscreen had a tendency to fog up in the winter rainy periods, so I'd have
to intermittently fire up the air conditioning to defog the windscreen. Of
course, in the summer months, you're going to want to run the AC to cool
down, so the range hit in that situation is unavoidable. I've only had this
car from late Autumn until late Winter so far, so that hasn't been an issue
I've had to contend with.
The dashboard is all digital, and looks relatively high tech, which appeals
to my inner geek. It's a dashboard. It tells you stuff. The stuff you'd
expect it to tell you. Enough said.
The audio system is nice. It supports Bluetooth audio, so one can be
streaming Pandora from one's phone, through the sound system, for example.
Or listening to audio from the phone. There's also a USB port, and it will
play various audio files from USB storage. I found the way it chose to sort
the files on a USB stick to be slightly surprising though. I haven't
invested the time to figure out how I should be naming files so that they
play in the order I expect them to play. The ability to play from USB
storage compensates nicely for the fact that it only has a single CD player.
(We have a 6 disc stacker in our 2006 Prius).
The car also came with a 3 month free trial of Sirius XM satellite radio.
This was fun. The only dance music FM station in the Bay Area has a weak
signal in Mountain View, and I hate dance music with static, whereas there
was an excellent electronic music station that I could listen to in glorious
high definition. As long as I wasn't driving under a bridge. There's no way
I'd pay money for satellite radio, but it was a fun gimmick to try out.
The navigation system is really, really good. I haven't bothered using
Google Maps on my phone at all. It gives such good spoken directions, that
you don't even need to have the map on the screen. It names all the streets.
I couldn't figure out a way to get distances in metric.
The telematics service, Carwings, is probably my favourite feature. This is
what really makes it feel like a car of the future. Through the companion
Android application, I can view the charging status (if it's plugged in) or
just check the available range (if it's not plugged in). From a web browser,
I can plan a route, and push the route to the vehicle's navigation system.
If the car is plugged in, I can also remotely turn on the vehicle's climate
control system, pre-warming or cooling the car.
It's a little thing, but the door unlocking annoyed me a little bit. I'm
used to the Prius, where if you unlock the boot (that's trunk, Americans),
or the front passenger door, all the doors unlocked. This was a convenient
way of unlocking the car for multiple people as you approached it. With the
Leaf, unlocking the boot only unlocks the boot. Unlocking the front
passenger door only unlocks that door. It requires a second unlock action to
unlock all the doors. I've found this to be slightly cumbersome when trying
to unlock the car for everyone all at once, quickly (like when it's
raining).
Another minor annoyance is the headlights. I've gotten into the habit of
driving with the headlights on all the time, because I believe it's safer.
In the Prius, one could just leave the headlights switched to the "on"
position, and they'd turn off whenever the driver's door was opened after
the car was switched off. If you try that in the Leaf, the car beeps at you
to warn you you've left the headlights on. It has an "auto" mode, where the
car will choose when to turn the headlights on, based on ambient light
conditions. In that case, when you turn the car off, it'll leave the
headlights on for a configurable period of time and then turn them off. This
is actually slightly unsettling, because it makes you think you've left your
headlights on. The default timeout was quite long as well, something like 60
seconds.
The way multiple Bluetooth phones are handled is just as annoying in the
Leaf as it is in the Prius, which disappoints me, given 6 years have passed.
The way I'd like to see multiple phones handled is the car chooses to pair
with whichever phone is in range, or if multiple phones are in range, it
asks or uses the last one it paired with. In reality, it tries to pair with
whatever it paired with last time, and one has to press far too many buttons
to switch it to one of the other phones it knows about.
Range anxiety is definitely something of a concern. It can be managed by
using the GPS navigation for long or otherwise anxiety-inducing trips, and
then one can compare the "miles remaining" on the GPS with the "miles
remaining" on the battery range, and reassure oneself that they will indeed
make it to where they're trying to go. The worst case I had was getting to
within 5 miles of empty. The car started complaining at me.
The charging infrastructure in the Bay Area is pretty good. There are plenty
of charging stations in San Francisco and San Jose. I'm spoiled in that I
have free charging available at work (including a building at the end of my
street, so I never bothered with getting a home charger installed). I've
almost never had to pay for charging, so it's been great while gas prices
have been on the rise.
The car's navigation system knows about some charging stations, so you can
plan a route with the charging stations it knows about in mind. The only
problem is it doesn't know if the charging stations are in use. If you use
the ChargePoint Android app, you can see if the charging stations are in
use, but then you have to do this cumbersome dance to find an available
charging station and plug the address into the vehicle's navigation system.
Of course, what can then happen is in the time you're driving to the
charging station, someone else starts using it. I actually got bitten by
this yesterday.
Would I buy a Leaf again? Not as my sole car. It makes a perfect
second/commuter car, but as a primary vehicle, it's too limited by its
range. They're also ridiculously expensive in Australia, and Brisbane has
absolutely no charging infrastructure.
Taking advantage of a new hard drive, I decided to reinstall my Ubuntu (Precise 12.04.2) laptop from scratch so that I could easily enable full-disk encryption (a requirement for Mozilla work laptops).
Reinstalling and reconfiguring everything takes a bit of time though, so here's the procedure I followed to keep the configuration to a minimum.
For once, I think we are going to have a funny DPL campaign thanks to Francesca who brought to us a 
A few weeks ago, a new version of 
If you would like more information about lldpd, please have a look
at 
DRI3000 Even Better Direct Rendering
This all started with the presentation that Eric Anholt and I did at
the 2012 X developers conference, and subsequently wrote about in my
A new minor release 
I will be speaking at the 
For some reason that is not very important, I was looking for a better
solution for GPSes with maps.
Side-note: if you ask why Android is not good enough , here's the
story: last year, knowing that Google Maps supports caching (finally),
I upgraded Maps, preloaded London, and went on a trip. Only to exit
the Victoria underground station and to stare ~20 minutes in light
rain at my phone, wondering why it doesn't get a fix. I gave up in the
end, and used the cached maps for getting around. Later I learned the
reason: Maps was not broken, but without data, Android can't get a
lock easily; and more-over, the recent Maps update without GPS active
even once afterwards, to do whatever it needed, meant I couldn't get a
lock until back in Switzerland.
Or, alternatively: while on a trip to Japan, it turns out that Google
Maps caching is not supported there, probably because of rights over
mapping data. So I had location, but only temporary cached maps (from
the wifi in the hotels).
Due to this kind of problems makes I consider Android maps a toy, so I
wanted a GPS device with proper maps (on the device, not in the
cloud), and with good GPS capabilities (no fancy A-GPS or anything,
just plain good standalone GPS).
Having had good experiences with the sport line of Garmin (but which
doesn't have maps, so it's out of the question), I looked at
Garmin. Many people complained about some things (same all over the
place):
Garmin's tag line: Big and Tough Goes anywhere, everywhere . I tend
to agree 
I wrote that I would