Russ Allbery: Automated Perl testing
I started several coding projects today, but none of them reached an
announcable conclusion, so this seems like a good day to write up my
current discoveries in automated Perl testing modules.
This post isn't about test frameworks like Test::More and friends, or even
about helper modules (although I do recommend Test::Warn if you want to
test warning code in your modules). It's not about writing functionality
tests. Rather, it's about various CPAN modules that can be used almost
identically in just about every Perl module distribution (and in some
other places, like Perl scripts inside a larger package written in another
language). You can drop a small driver script into your
t
directory and (mostly) run make test with no further customization.
If you want to see the driver scripts for all of the below that I'm
currently using, see the t directory of the
Term::ANSIColor distribution. They'll
slowly make their way into my other modules, and probably, despite the
name of the package, rra-c-util, since
that's my equivalent of gnulib and I have tools to synchronize shared code
from that package. The driver script generally tests whether the module
is installed, skips the test if not, and otherwise dispatches to the "test
everything" routine in the module.
Test::MinimumVersion: This tests that you're not accidentally
using a newer Perl feature than you were expecting. It's very handy if
you want to support backward compatibility to a particular version of
Perl. It knows about features as far back as 5.004_05. It won't be able
to catch everything, but it will catch some of the most obvious things.
Test::Perl::Critic: This is sort of the thousand-pound gorilla.
Test::Perl::Critic runs all of your code through perlcritic and reports
any errors. You will probably need to combine this with a perlcriticrc
file in your distribution that you configure it to use, since otherwise
the person running the tests may have customized perlcritic. If you let
it run perltidy to check for formatting problems, you'll also want a
perltidyrc. See the driver script mentioned above for how I set that up.
I usually configure this test to only run for me (by checking an
environment variable) since the results vary a lot depending on what
version of perlcritic is installed, and it's likely new versions will
cause the test to start failing. If you use this, expect to need to add
## no critic annotations to your code.
Test::Pod: Checks for syntax errors in your POD documentation
so that you won't have those embarassing POD ERRORS sections. Note that
the latest version of podlators turns
syntax errors into fatal errors, so soon you won't have to worry as much
about this. But if you'd been using this module all along, you wouldn't
have to worry about the fact that pod2man may start faliing due to syntax
errors you never fixed!
Test::Pod::Coverage: Only of interest to module distributions,
not ramdom scripts, this checks that every sub provided by your module has
POD documentation. It understands various subs that don't need to be
documented (such as the ones you provide to implement a tied class) and
lets you mark some subs as not requiring documentation.
Test::Pod::Spelling: Checks your POD documentation for spelling
mistakes with a spell checker. (Note that perlcritic, and hence
Test::Perl::Critic, can also do this, but I use this module separately
since I think the output is more readable.) You will almost certainly
want to make this test maintainer-only, since spell checkers and spelling
dictionaries vary a lot. But it prevents a lot of embarassment when you
read your documentation on search.cpan.org and find that obvious spelling
error you never saw before.
Test::Strict: Checks all Perl code to ensure that it compiles,
checks whether it has use strict, and optionally checks whether it
has use warnings. This is great; I use this even with packages
that just have a few Perl utility scripts in a package otherwise written
in C, since I previously kept releasing packages with helper scripts with
obvious syntax errors since I didn't run them much. Another place it's
useful is if you have internal Debian packages that wrap up Perl utility
scripts; just create a small test suite with this module and run it from
override_dh_auto_check and you won't ever release a package with scripts
that won't run. You can also do test suite coverage testing with this
module (and Devel::Cover), but that you'll want to mark maintainer-only
since it takes a while.
Test::Synopsis: Checks the code in the SYNOPSIS section of any
POD documentation to ensure that it compiles. No more syntax errors in
your example code in the documentation!
All of these modules are, of course, packaged for Debian (thanks largely
to the wonderful Perl packaging team!), and otherwise available on CPAN.
There are a variety of others, of course; these are just the main ones
that I've found personally useful.
One final whine, however: all this wonderful Perl testing infrastructure
desperately needs a common module and a common interface for providing a
list of files to check. Every one of these modules reinvents that wheel,
and most of them do it a slightly different way than all the others. Most
of them check in either blib or lib (or sometimes both).
Most of them also add t. Most of them don't include
Makefile.PL, but some of them do. Almost none of them look in
examples. Some have a simple way to add more files or directories;
some don't. Some can be run from the t directory and still find
things, but most of them can't. And of course the syntax is different in
every case! Most of the work in my driver scripts is finding the right
way to add a few additional directories to the scope of the testing, and
in one annoying case (Test::Synopsis), I actually had to use
Perl::Critic::Utils (for lack of a better module) to build a list of files
to check instead of using the build-in file list, since the build-in file
list required all module code be in lib.
A module to abstract out all of this and provide a common interface would
be easy to write. Most of the work would involve talking to all the
authors of all these different modules and asking them to use it. If
someone in the Perl community felt like doing that, it would make life so
much easier for everyone.
Let s start a convention: bare-metal machines have the linguistic male
gender (
Recently, I've started programming in 
The Microsoft Surface is a fairly attractive bit of tablet hardware, and as a result people have shown interest in running Linux on it. The immediate problem is that (like many ARM devices) it has a locked-down firmware that will only run signed binaries - unlike many other ARM devices, this is implemented using an existing standard (UEFI Secure Boot). Microsoft provide a signing service for UEFI binaries, so it's tempting to think that getting around this restriction would be as simple as taking an existing Linux bootloader, signing it and then booting. Unfortunately Microsoft's signing service signs binaries using a different key (the "Microsoft Windows UEFI Driver Publisher" key) to the one used to sign Windows, and the Surface doesn't carry that key. Booting Linux on these devices would involve finding a flaw in the firmware and using that to run arbitrary code.
A couple of days ago I made a new release of