Recently
Rob and I visited
Soile and
Lars. We had a lovely
time wandering around Helsinki with them, and I also spent a good chunk of time
with Lars working on some design and planning for the
Yarn test specification and tooling.
You see, I wrote a
Rust implementation of Yarn
called
rsyarn "for fun" and in doing
so I noted a bunch of missing bits in the understanding Lars and I shared about
how Yarn should work. Lars and I filled, and re-filled, a whiteboard with
discussion about what the 'Yarn specification' should be, about various
language extensions and changes, and also about what functionality a normative
implementation of Yarn should have.
This article is meant to be a write-up of all of that discussion, but before I
start on that, I should probably summarise what Yarn is.
Yarn is a mechanism for specifying tests in a form which is more like
documentation than code. Yarn follows the concept of
BDD story based
design/testing and has a very
Cucumberish scenario
language in which to write tests. Yarn takes, as input,
Markdown documents which contain code
blocks with Yarn tests in them; and it then runs those tests and reports on the
scenario failures/successes.
As an example of a poorly written but still fairly effective Yarn suite, you
could look at
Gitano's tests or perhaps
at
Obnam's tests
(
rendered as HTML). Yarn is not trying
to replace unit testing, nor other forms of testing, but rather seeks to be one
of a suite of test tools used to help validate software and to verify
integrations. Lars writes Yarns which
test his server setups for example.
As an example, lets look at what a simple test might be for the behaviour of
the
/bin/true tool:
SCENARIO true should exit with code zero
WHEN /bin/true is run with no arguments
THEN the exit code is 0
AND stdout is empty
AND stderr is empty
Anyone ought to be able to understand exactly what that test is doing, even
though there's no obvious code to run. Yarn statements are meant to be easily
grokked by both developers and managers. This should be so that managers can
understand the tests which verify that requirements are being met, without
needing to grok python, shell, C, or whatever else is needed to implement the
test where the Yarns meet the metal.
Obviously, there needs to be a way to join the dots, and Yarn calls those
things
IMPLEMENTS, for example:
IMPLEMENTS WHEN (\S+) is run with no arguments
set +e
"$ MATCH_1 " > "$ DATADIR /stdout" 2> "$ DATADIR /stderr"
echo $? > "$ DATADIR /exitcode"
As you can see from the example, Yarn
IMPLEMENTS can use regular expressions
to capture parts of their invocation, allowing the test implementer to handle
many different scenario statements with one implementation block. For the rest
of the implementation, whatever you assume about things will probably be okay
for now.
Given all of the above, we (Lars and I) decided that it would make a lot of
sense if there was a set of Yarn scenarios which could validate a Yarn
implementation. Such a document could also form the basis of a Yarn
specification and also a manual for writing reasonable Yarn scenarios. As
such, we wrote up a three-column approach to what we'd need in that test suite.
Firstly we considered what the core features of the Yarn language are:
- Scenario statements themselves (
SCENARIO, GIVEN, WHEN, THEN,
ASSUMING, FINALLY, AND, IMPLEMENTS, EXAMPLE, ...)
- Whitespace normalisation of statements
- Regexp language and behaviour
IMPLEMENTS current directory, data directory, home directory, and also
environment.- Error handling for the statements, or for missing
IMPLEMENTS
- File (and filename) encoding
- Labelled code blocks (since commonmark includes the backtick code block
kind)
- Exactly one
IMPLEMENTS per statement
We considered unusual (or corner) cases and which of them needed defining in
the short to medium term:
- Statements before any SCENARIO or IMPLEMENTS
- Meaning of split code blocks (concatenation?)
- Meaning of code blocks not at the top level of a file (ignore?)
- Meaning of HTML style comments in markdown files
- Odd scenario ordering (e.g.
ASSUMING at the end, or FINALLY at the
start)
- Meaning of empty lines in code blocks or between them.
All of this comes down to how to interpret input to a Yarn implementation. In
addition there were a number of things we felt any "normative" Yarn
implementation would have to handle or provide in order to be considered
useful. It's worth noting that we don't specify anything about an
implementation being a command line tool though...
- Interpreter for
IMPLEMENTS (and arguments for them)
- "Library" for those implementations
- Ability to require that failed
ASSUMING statements lead to an error
- A way to 'stop on first failure'
- A way to select a specific scenario to run, from a large suite.
- Generation of timing reports (per scenario and also per statement)
- A way to 'skip' missing
IMPLEMENTS
- A clear way to identify the failing step in a scenario.
- Able to treat multiple input files as a single suite.
There's bound to be more, but right now with the above, we believe we have two
roughly conformant Yarn implementations. Lars' Python based implementation
which lives in
cmdtest (and which I shall refer to as
pyyarn for now) and
my Rust based one (
rsyarn).
One thing which
rsyarn supports, but
pyyarn does not, is running multiple
scenarios in parallel. However when I wrote that support into
rsyarn I
noticed that there were plenty of issues with running stuff in parallel. (A
problem I'm sure any of you who know about threads will appreciate).
One particular issue was that scenarios often need to share resources which
are not easily sandboxed into the
$ DATADIR provided by Yarn. For example
databases or access to limited online services. Lars and I had a good chat
about that, and decided that a reasonable language extension could be:
USING database foo
with its counterpart
RESOURCE database (\S+)
LABEL database-$1
GIVEN a database called $1
FINALLY database $1 is torn down
The
USING statement should be reasonably clear in its pairing to a
RESOURCE
statement. The
LABEL statement I'll get to in a moment (though it's only
relevant in a
RESOURCE block, and the rest of the statements are essentially
substituted into the calling scenario at the point of the
USING.
This is nowhere near ready to consider adding to the specification though.
Both Lars and I are uncomfortable with the
$1 syntax though we can't think of
anything nicer right now; and the
USING/
RESOURCE/
LABEL vocabulary isn't
set in stone either.
The idea of the
LABEL is that we'd also require that a normative Yarn
implementation be capable of specifying resource limits by name. E.g. if a
RESOURCE used a
LABEL foo then the caller of a Yarn scenario suite could
specify that there were 5
foos available. The Yarn implementation would then
schedule a maximum of 5 scenarios which are using that label to happen
simultaneously. At bare minimum it'd gate new users, but at best it would
intelligently schedule them.
In addition, since this introduces the concept of parallelism into Yarn proper,
we also wanted to add a maximum parallelism setting to the Yarn implementation
requirements; and to specify that any resource label which was not explicitly
set had a usage limit of 1.
Once we'd discussed the parallelism, we decided that once we had a nice syntax
for expanding these sets of statements anyway, we may as well have a syntax for
specifying scenario language expansions which could be used to provide
something akin to macros for Yarn scenarios. What we came up with as a
starter-for-ten was:
CALLING write foo
paired with
EXPANDING write (\S+)
GIVEN bar
WHEN $1 is written to
THEN success was had by all
Again, the
CALLING/
EXPANDING keywords are not fixed yet, nor is the
$1
type syntax, though whatever is used here should match the other places where
we might want it.
Finally we discussed multi-line inputs in Yarn. We currently have a syntax
akin to:
GIVEN foo
... bar
... baz
which is directly equivalent to:
GIVEN foo bar baz
and this is achieved by collapsing the multiple lines and using the whitespace
normalisation functionality of Yarn to replace all whitespace sequences with
single space characters. However this means that, for example, injecting
chunks of
YAML into a Yarn scenario is a pain, as would be including any
amount of another whitespace-sensitive input language.
After a lot of to-ing and fro-ing, we decided that the right thing to do would
be to redefine the
... Yarn statement to be whitespace preserving and to then
pass that whitespace through to be matched by the
IMPLEMENTS or whatever. In
order for that to work, the regexp matching would have to be defined to treat
the input as a single line, allowing
. to match
\n etc.
Of course, this would mean that the old functionality wouldn't be possible, so
we considered allowing a
\ at the end of a line to provide the current kind
of behaviour, rewriting the above example as:
GIVEN foo \
bar \
baz
It's not as nice, but since we couldn't find any real uses of
... in any of
our Yarn suites where having the whitespace preserved would be an issue, we
decided it was worth the pain.
None of the above is, as of yet, set in stone. This blog posting is about me
recording the information so that it can be referred to; and also to hopefully
spark a little bit of discussion about Yarn. We'd welcome emails to our usual
addresses, being poked on Twitter, or on IRC in the common spots we can be
found. If you're honestly unsure of how to get hold of us, just comment on
this blog post and I'll find your message eventually.
Hopefully soon we can start writing that Yarn suite which can be used to
validate the behaviour of
pyyarn and
rsyarn and from there we can implement
our new proposals for extending Yarn to be even more useful.
The past few weeks have been randomly busy, nothing huge has happened, but several minor diversions.
Coding
Visiting the FSFE team in Berlin
Having recently joined the FSFE's General Assembly as the 
Steak and SDR night
After a hard day of SIP hacking and a long afternoon at Kamailio World's open bar, a developer needs a decent meal and something previously unseen to hack on. A group of us settled at Escados, Alexanderplatz where my SDR kit emerged from my bag and other Debian users 
Next stop OSCAL'17, Tirana
Having left Berlin, I'm now in Tirana, Albania where I'll give an SDR workshop and Free-RTC talk at 
Two days ago one of the most influential logician of the 20th century has passed away, 
I don t want to recall his achievements in mathematical logic, in particular proof theory, because I am not worth to write about such a genius. I want to recall a few personal stories from my own experience.
I came into contact with Prof. Takeuti via is famous book 
Debian has many years of history, about 25 years already.
With such a long travel over the continous field of developing our Universal
Operating System, some myths, false accusations and bad reputation has arisen.
Today I had the opportunity to discuss this topic, I was invited to give a
Debian talk in the 
myth #1: Debian is old software
Please, use testing or stable-backports. If you use Debian stable your system
will in fact be stable and that means: updates contain no new software but
only fixes.
myth #2: Debian is slow
We compile and build most of our packages with industry-standard compilers and
options. I don t see a significant difference on how fast linux kernel or
mysql run in a CentOS or in Debian.
myth #3: Debian is difficult
I already discussed about this issue back in
If you see Thunderbird hanging during startup for a minute and then continuing to load fine, you are probably running into an issue similar to what I saw when Debian migrated Icedove back to the "official" Mozilla Thunderbird branding and changed 
Kamailio World features a range of talks about developing and using SIP and telephony applications and offers many opportunities for SIP developers, WebRTC developers, network operators and users to interact. Wednesday, at midday, there is a Dangerous Demos session where cutting edge innovations will make their first (and potentially last) appearance.
OSCAL'17, Tirana
OSCAL'17 is an event that has grown dramatically in recent years and is expecting hundreds of free software users and developers, including many international guests, to converge on Tirana, Albania this weekend.
On Saturday I'll be giving a workshop about 
Recently, 
Fuzzing is method of producing random malformed inputs for a software and
observe the software behavior. If a software crashes then there is a bug and it
can have security implications. Fuzzing has gained a lot of interest now a days,
especially with automated tools like American Fuzzy Lop (AFL) which can easily
help you to fuzz the program and record inputs which causes crash in the
software.
American Fuzzy Lop is a file based fuzzer which feeds input to program via
standard input. Using it with network program like server's or clients is not
possible in the original state. There is a 
Debian Stretch stable includes the nftables framework, ready to use.
Created by the Netfilter project itself, nftables is the firewalling tool
that replaces the old iptables, giving the users a powerful tool.
Back in October 2016, I wrote a 
Did you know that the nano editor includes nft syntax highlighting?
Starting with Debian stable Stretch and nftables, packet filtering and network
policing will never be the same.
Debian LTS
April marked the 24th month I contributed to 
In the above example, we have hypervisors hosting a virtual
machines from different tenants. Each virtual machine is given
access to a tenant-specific virtual Ethernet segment. Users are
expecting classic Ethernet segments: no 
Compared to 
VXLANs can be provisioned with the following script. Source-address
learning is disabled as we will rely solely on 
To illustrate our examples, we use the following setup:
The 