There s a test code maintenance issue I ve been grappling with, and watching others grapple with for a while now. I ve blogged about some infrastructural things related to it before, but now I think its time to talk about the problem itself. The problem shows up as soon as you start writing setUp functions, or custom assertThing functions. And the problem is where do you put this code?
If you have a single TestCase, its easy. But as soon as you have two test classes it becomes more difficult. If you choose either class, the other class cannot use your setUp or assertion code. If you create a base class for your tests and put the code there you end up with a huge base class, and every test paying the total overhead of your test needs, rather than just the overhead needed to test the particular system you want to test. Or with a large and growing list of assertions most of which are irrelevant for most tests.
The reason the choices have to be made is because test code is just code; and all the normal issues there separation of concerns, composition often being better than inheritance, do-one-thing-well all apply to our test code. These issues are exacerbated by pyunit (that is the Python unittest module included with the standard library and extended by various projects)
Lets look some (some) of the concerns involved in a test environment: Test execution, fixture management, outcome decision making. I m using slightly abstract terms here because I don t want to bind the discussion down to an existing implementation. However the down side is that I need to define these terms a little.
Test execution by this I mean the basic machinery of running a single test: the test framework calling into user code and receiving back an outcome with details. E.g. in pyunit your test_method() code is called, success is determined by it returning successfully, and other outcomes by raising specific exceptions. Other languages without exceptions might do this returning an outcome object, or passing some object into the user code to be called by the test.
Fixture management the non trivial code that prepares a situation where you can make assertions. On the small side, creating a few object instances and glueing them together, on the large end, loading data into a database (and creating the database instance at the same time). Isolation issues such as masking out environment variables and creating temp directories are included in this category in my opinion.
Outcome decision making possibly the most obtuse label I ve ever given this, I m referring the process of deciding *what* outcome you wish to have happen. This takes different forms depending on your testing framework. For instance, in Python s doctest:
>>> x
45
provides a specification the test framework calls str(x) and then compares that to the string 45 . In pyunit assertions are typically used:
self.assertEqual(45, x)
Will call 45 == x and if the result is not True, raise an exception indicating a Failure has occured. Unexpected exceptions cause Errors, and in the most recent pyunit, and some extensions, other exceptions can signal that a test should not be run, or should have failed.
So, those are the three concerns that we have when testing; where should each be expressed (in pyunit)? Pragmatically the test execution code is the hardest to separate out: Its partly outside of user control , in that the contract is with the test framework. So lets start by saying that this core facility, which we should very rarely need to change, should be in TestCase.
That leaves fixture management and outcome decision making. Lets tackle decision making if you consider the earlier doctest and assertion examples, I think its fairly clear that there are multiple discrete components at play. Two in particular I d like to highlight are: matching and signalling. In the doctest example the matching is done by string matching the reference object(s) are stringified and compared to an example the test writer provides. In the pyunit example the matching is done by the __eq__ protocol. The signalling in the doctest example is done inside the test framework (so we don t see any evidence of it at all). In the pyunit example the signalling is done by the assertion method calling self.fail(), that being the defined contract for causing a failure. Now for a more complex example: testing a float. In doctest:
>>> %0.3f % x
0.123
In pyunit:
self.assertAlmostEqual(0.123, x, places=3)
This very simple check that a floating point number is effectively 0.123 exposes two problems immediately. The first, in doctest, is that literal string comparisons are extremely limited. A regex or other language would be much more powerful (and there are some extensions to doctest; the point remains though the operator is not enough). The second problem is in pyunit. It is that the contract of assertEqual and assertAlmostEqual are different: you cannot substitute one in where the other was expected without partial function application something that while powerful is not the most obvious thing to reach for, or to read in code. The JUnit folk came up with a nice way to address this: they decoupled /matching/ and /deciding/ with a new assertion called assertThat and a language for matching expressed as classes. The initial matcher library, hamcrest, is pretty ugly in Python; I don t use it because it tries too hard to be english like rather than being honest about being code. (Aside, what would is_() in a python library mean to you? Unless you ve read the hamcrest code, or are not a Python programmer, you ll probably get it wrong. However the concept is totally sound. So, outcome decision making should be done by using a matching language totally seperate from testing, and a small bit of glue for your test framework. In testtools that glue is assertThat , and the matching language is a narrow Matcher contract (in testtools.matchers) which I m going to describe here, in case you cannot or don t want to use the testtools one.
class Matcher:
def __str__(self):
"Describe this matcher."""
def match(self, something):
"""Determine if something is matched.
:param something: Something to match.
:return: None if something matched, or a Mismatch object otherwise.
"""
class Mismatch:
def describe(self):
"""Describe a mismatch that has occured."""
This permits composition and inheritance within your matching code in a pretty clean way. Using == only permits this if you can simultaneously define an __eq__ for your objects that matches with arbitrarily sensitivity (e.g. you might not want to be examining the process_id value for a process a test ran, but do want to check other fields).
Now for fixture management. This one is pretty simple really: stop using setUp (and other similar on-TestCase methods). If you use them, you will end up with a hierarchy like this:
BaseTestCase1
+TestCase1
+TestCase2
+BaseTestCase2
+TestCase3
+TestCase4
+BaseTestCase3
+TestCase5
...
That is, you ll have a tree of base classes, and hanging off them actual test cases. Instead, write on your base TestCase a single glue method e.g.
def useFixture(self, fixture):
fixture.setUp()
self.addCleanup(fixture.tearDown)
return fixture
And then rather than having a setUp function which performs complex operations, define a fixture an object with a setUp and a tearDown method. Use this in tests that need that code::
def test_foo(self):
server = self.useFixture(NewServerWithUsers())
self.assertThat(server, HasUser('fred'))
Note that there are some things around that offer this sort of convention already: thats all it is convention. Pick one, and run with it. But please don t use setUp; it was a conflated idea in the first place and is a concrete problem. Something like testresources or testscenarios may fit your needs if it does, great! However they are not the last word they aren t convenient enough to replace just calling a simple helper like I ve presented here.
To conclude, the short story is:
- use assertThat and have a seperate hierarchy of composable matchers
- use or create a fixture/resouce framework rather than setUp/tearDown
- any old TestCase that has the outcomes you want should do at this point (but I love testtools).
Some students of the Google Summer of Code are required to make a weekly report describing their activities. As far as I know, I m not, but I think it s a good idea in general, to comunicate to my mentor and other interested folks what I ve been doing. So I ll start to do them here.
My project is Implementing the Immix Garbage Collector Algorithm on GHC.
Last week I was most of the time involved in my school activities, so I didn t have much time to work on the project. I went to the bureaucratic services of my university to get a Proof of Enrollment in English, print/sign/scanned the Foreign Certification Form and sent them to Google on Friday afternoon. I asked them to confirm the receival, which they didn t. I m planning to ask again for a confirmal today in the afternoon, if I still get no reply.
I also tried to build the current version of GHC, from the darcs repository. I was getting an error with my old (and updated) repository, so I donwnloaded a new one and could build nicely. Then I noticed that with make distclean I could also build with my older one, so I just removed the new. I downloaded the nofib benchmark suite and made an initial benchmark.
While I was reading the documentation of the build process of GHC, I noticed a bug, and submitted a patch to the cvs-ghc mailing list. The patch was applied, with some changes.
This week I plan to focus more on the project, and spend the weekly 36 hours working on it. I m planning to start reading the Immix Technical Report to see if there s anything new there with regard to the article. After that I ll read the wiki documentation on the GHC current GC, together with the GHC source code. I ll also experiment with Criterion.
While I was writing this post I noticed that the GC benchmark in nofib is failing:
I ve been reading about 
Today I strapped a kayack on a car (badly) and gingerly drove it out to the
country, stopping for Greek food enroute. Now at my 
UPDATE: the bug is already fixed after a few hours, and only affected a
minority of users (of a now deprecated, experimental option in the 'unstable'
distribution, and only users that rebooted with the affected version).The sysvinit version that
I didn't write much about running stuff since 
Part of the problem that we face in providing Linux hardware support is that we're lucky if there's a spec, and even if there's a spec there almost certainly isn't a test suite. Linux still isn't high on the list of things that vendors test with, so as a result hardware and firmware tend to be written to work with Windows rather than some more ideal notion of what a spec actually says.
A scandal has been brewing in Latvia over the last half year and yesterday the activity spiked shocking the media and some IT people in the country. I ll go back and explain what happened first, what is happening now and why this could have a heavy impact on IT and journalists in Latvia.
At the end of last year, there were rumours that the IT system of Latvia s Internal Revenue System was hacked and millions of documents had been downloaded by multiple organizations. Shortly thereafter more details on the glaring security hole became public (after it was closed).
There is a full electronic interface to give all reports to the IRS electronically (at http://eds.vid.gov.lv) and as part of that system you could also view and export monthly report summaries about your organization into XML and PDF files. After the system checked that you are authorized to access the report, you were redirected to the URL to actually download the report by report ID (as a single param in a GET request). Unfortunately, report IDs were predicable and the script that gave the reports for download did not check if you were authorized to get that report. It did not even check if were logged into the system.
There were suspicions that the authorization was disabled on purpose to allow to leak data on purpose, but apparently it was an error of forgetting to disable debug code in production environment.
The error was discovered only because the firewall administrator noticed an unexplained stable increase of traffic, especially during night hours when typically the traffic fully stopped. Apparently a single hacker (who later identified himself as Neo to the press) discovered the flaw and wrote a script to just try all possible report ids and get as much data out as possible. This had been going on for months, before someone noticed.
After the flaw was discovered and a bit of time passed, Neo made his first move he published the list of top salaries in a governmental company, that clearly showed that the top leadership of this company failed to cut their salary by 40%, like everyone elses during harsh budget cuts of 2009. He stripped the names and ids of the specific employees, but named the company which made it pretty easy to figure out who was who.
The society was outraged that the top managers in a government owned company failed to comply with the strict pay cut that everyone else in government had to endure. But after a few weeks the outrage subsided and no action followed from the government or law enforcement.
Neo continued to release documents detailing salaries of top managers in different Latvian government companies. And each time after short outrage, nothing happened. Neo gave an interview where he said that he was disappointed in the passivity of the Latvian people in face of such blatant injustices.
After a few month Neo went silent, promising to return before parliamentary elections this fall.
However, this week a new development shocked everyone in the middle of the night two police SWAT teams went into action: one detained Ilm rs Poik ns, a researcher in artificial intelligence at the University of Latvia s Computer Science department and another raided the home of a Latvian TV journalist Ilze Nagle who interviewed Neo. Poik ns confessed of being Neo the next day and was released (with travel restrictions, pending trial) today.
Politicians reacted immediately opposition demanded the resignation of the Interior Minister over such blatant disregard of freedom of press and another politician (who is also a famous lawyer) Aleksejs Loskutovs volunteered to defend Neo pro-bono (on Twitter, no less). Almost all Latvian online media have the arrest of Neo and the raid on the home of a journalist as main stories of the day.
As a legal titbit, we also know that Neo is being charged with breaking statutes 145 and and 244p2 of the criminal law. Statute 145 is hard to find applicable in this situation as talks about actions done by people authorized (..) to access [private] information . Statute 244p2 will also be hard to pin down as it mentions influencing system resources of (an IT system) and if such action caused severe harm . It looks like the first part talks about at least a DoS attack (which did not happen in this case) and also there was no measurable harm from these leaks.
Also Neo was careful to strip all personally identifying information (such as names, social security numbers and addresses of the employees in question), so it will be hard to pin him on that. Also no actual breaking or other modification of an IT system occurred. And no specialized software was used beyond a trivial script such as :
Contratulations to Bob Finch and Nathan Dalrymple for becoming our
first customers to fly a rocket using a
I just uploaded 
I'm happy to conclude quoting the acknowledgments section of the
paper:
Acknowledgments
Italian
I woke up with a loud BZZZT It happens every couple of years. The electric transformer for the circuit where my house is located, at the Northern edge of Ciudad Universitaria, decided to die (or at least, to take a break literally).
About one hour later, I decided it was time to wake up and start being a useful person. I gave breakfast to my cats and had breakfast myself, and called the electrical company to report this mishap. They told me the report was registered, and I hope to have electricity soon (meanwhile, I'm sitting at a nearby restaurant, as there is some job to do Yes, besides writing blog entries). And they told me, as is often told in Mexico, a general anticorruption phrase With a twist: recuerde que usted no debe renumerar ning n servicio que realice el personal de la Comisi n Federal de Electricidad . (You should not renumber any service done by the Federal Electrical Commission personnel). Yes, remunerado (not renumerado).
This shows once again the power of asking people to read things they don't understand.
