I write a fair number of Unix command line programs, and I want to
have test suites for them. A year and a half ago I wrote a tool to
make that easier:
cmdtest. However, it turned out to not be a
particularly good one, primarily because it doesn't make it easy to
write good test code. Tests and test code should be obviously
correct, so you don't need a test suite for your test suite.
Earlier this year,
Daniel and I started a slow, but long discussion
about this. We'd talk it over during many a lunch, figuring out what
we actually want. We eventually converged on an approach based on
for Unix command line programs, and based on
Markdown documents.
We then wrote that up as a
README, and then took another month
refining the README before writing any code. The naming discussion
only took a week.
- We borrowed the given/when/then structure from BDD, and the idea of
using that structure so that tests are specified using lightly
structured text in a way that allowsnon-developers to participate.
- This borrowed structure gives a way to express sequences of things
that happen, including setup and checks. We call these sequences
test scenarios, and thus this approach to testing is called scenario
testing.
- We embed the test scenarios, and the code that implements them, as
code blocks in Markdown documents. This makes it easy to explain
what is going on in the tests, when that's needed, right next to the
tests. The Markdown files can easily be formatted as web pages or
PDFs (among other formats), making it easy to share and discuss them
with people who might not be comfortable accessing a git repository.
As a concrete example, here is a real test scenario, from Obnam, for
testing its new FUSE plugin:
SCENARIO Browsing backups with FUSE plugin
ASSUMING user is in group fuse
GIVEN a live data directory
AND a 0 byte file called empty
AND a 1 byte file called one
AND a 4096 byte file called 4k
AND a 10485760 byte file called 10meg
AND a hardlink to 4k called 4k-hardlink
AND a symlink to 4k called 4k-symlink
AND a directory called some-dir
AND a symlink to ../4k called some-dir/other-symlink
WHEN live data is backed up
AND repository is fuse mounted
THEN latest generation can be copied correctly from fuse mount
FINALLY unmount repository
Some explanations:
SCENARIO is the name of the scenario.ASSUMING is a condition for the entire scenario: the FUSE plugin
can only be tested when the user can use FUSE.GIVEN is a setup step, WHEN is the step that does the thing that
is being tested, and THEN checks that the right thing happened
during WHEN.AND is an alias for the keyword on the previous line; it makes the
test scenario seem more like normal English textFINALLY is for cleaning up afterwards, whether the scenario passed
or not
Test scenarios like this are obviously not directly executable. We
need to provide the code that actually executs each step. For this,
yarn provides an IMPLEMENTS section, of which I will show one
example: the creation of the files.
IMPLEMENTS GIVEN a (\d+) byte file called (\S+)
dd if=/dev/zero of="$DATADIR/live/$MATCH_2" bs=1 count="$MATCH_1"
(The other IMPLEMENTS sections for that Obnam test scenario are also
only about a line of shell. The longest one is four lines, because it
has an
if statement that I like to format in a fairly verbose
way. I'm skipping the others to avoid needless repetition.)
The text after the keyword (GIVEN, WHEN, THEN, etc) in the test
scenario is a single line of free-form text, which is matched by a
regular expression in the IMPLEMENTS section. There must be exactly
one matching IMPLEMENTS for each line in a test scenario.
The body of IMPLEMENTS is a shell script snippet. Yarn provides a
temporary directory for each scenario, and the path to that is given
in the
DATADIR environment variable. Yarn also cleans up the
directory afterwards, unless the user running yarn requests otherwise,
so tests that only change things in the temporary directory don't need
any FINALLY steps. The above example uses FINALLY to unmount the FUSE
filesystem at the end.
The regular expressions can capture parts of the text in the scenario
step, and these are provided to the shell script using environment
variables:
MATCH_1 etc. This is a simple, but powerful mechanism for
implementing very flexible steps, which also means the number of steps
can be kept down.
The IMPLEMENTS sections are shared between all test scenarios. This
makes it very easy to write new scenarios using already-existing
steps, by choosing which ones to use, in which order, and making use
of any variability in the steps that their implementations provide.
Yarn is a new tool, and I expect it to mature in the coming
months. It's included in the cmdtest Debian package, and is already in
Debian jessie and unstable, for those who want to try it. (My personal
apt repository has a .deb for wheezy as well; backports are welcome).
The yarn scenario testing language itself is not set in stone, but
Daniel and I both want to be careful about making changes to it. It's
all meant to be very simple: an obviously correct testing tool is
better than an intricate one.
I've used yarn in several of my own little programs now, as has
Daniel, and we've used it at
work as well. Undoubtedly we'll find
bugs in the implementation, and we're still feeling our way to
figuring out how to write really excellent scenarios, and how to
structure steps with the best possible taste. We'll also start
designing a shell library that yarn provides to the IMPLEMENTS
sections, for doing things that we find ourselves doing in many
projects.
It is already clear to me, however, that yarn is oh so much better
than cmdtest could ever be. I'm even starting to express ideas for new
tools as yarn test documents, so I'm clear what they should do. If and
when I ever get to actually implementing the tools, I'll have a test
suite for them already. See, for example, first attempts at specifying
how a fast file transfer tool,
fling, which is to use Linux-only
system calls to reach wire speeds over wired LAN, and a tool to
recompress compressed files with another compression tool.
We hope yarn will prove to be useful for others as well. If you try
it, and have suggestions or comments, please e-mail us.
As usual with our phpMyAdmin team meetings, quite a lot things get decided there and it was no difference this year, when we've met at DebConf 13.
We've discussed quite a lot of topics and some had quite interesting outcome.
One of already changes is that we're going to consolidate our support efforts on Stack Overflow. We've provided various support channels so far (IRC, mailing list, web forums and support tracker), which have overlapped quite a lot and we simply lack manpower to do all of that properly.
So users on IRC mostly get bored and left before somebody got to them. The mailing list received just few mails in a month, so they were almost not used. The web forums worked quite well for answering questions, but nobody was able to find answers there (the forums on SourceForge are quite hard to get properly indexed by Google), so the questions got repeated quite a lot. In the end we've decide to use Stack Overflow as it was anyway already used by lot of our users and concentrating our effort on single channel will even improve this.
A couple of days ago, I gave a talk to the
As I slowly read my good friends wishing each other a good trip, telling they got home safely, and the IRC channels form thick drops of a bitter-sweet etheral substance, I cannot help feeling DebConf13 is over For me as well, from the distance. Many friends gave me warm greetings, and without being there, gave me that beautiful feeling of real community that Debian has given me for ten years already, since I met in real-life many of its developers at DebConf3 in Oslo. And yes, I have stated this far too many times I have attended every DebConf since (and worked organizing most of them). This year, 
WebRTC: premium voice/video chat in modern browsers, no plugin required
Another member of Debian's GSoC program, 
As the grand finale, we took 
The grand-finale of the 
As of today, thanks to the tremendous work of the French localization
team in Debian, 
Somehow last year I missed the fact that I've been around 
This weekend I'm mostly alternating between reading, writing, and trying to avoid death by the plague.[*]
I've switched Lumail to using a UTF-8 aware string library, which means we can now handle the obvious case:
Due to the extremely r
The 
