Skip to main content

What role are your tests serving.

We had a discussion at work after watching the excellent Kelvin Henney talk about good unit tests. And it helped me clarify the many roles tests play and why people get a habit of complaining about their tests getting in the way. So here is the four categories a test is trying to balance:


  • Requirements
  • Regression
  • Documentation
  • Refactoring guide


Most test today are written as requirements. And this is a huge improvement over the past when we'd write tests later and try and get code coverage. Those 'later' tests are useless. The requirement tests happen for high level acceptance tests and in low level TDD too. Your test defines a behaviour you want. Once your test passes you don't go back to change it at all beyond some obvious cleanups like code duplication. These tests allow rapid development and reliable progress. But they can hinder refactoring, provide poor feedback on failures for regression and may be too implementation focused to be readable.

And easy correction is to make tests act as good documentation. Unit tests should explain component behaviour to other programmers. Think of it as the examples section in your book that any programmer is going to flip to and read first. Higher level tests can use BDD, tables or other techniques to create highly readable descriptions that can be shared with your product owner or customer. One thing you'll notice if you try and make unit tests act as good documentation is you'll want to avoid the london-style or mockist style test where every secondary object is mocked out. All those mocks heavily hinder the tests being a document as it shifts away from being an example of how the code runs on the product.

You may think all those requirements focused tests are all you need to cover the regression case. But for good regression tests you want highly informative error messages. I find those get introduced later when you start seeing such failures. That might be when a bug is introduced or a new feature makes old tests inaccurate. That's a good time to go back through those assert statements and make sure they give enough information to know how to correct it. As once corrected those asserts will benefit you ever after. There are also some frameworks that can help like the highly informative asserts from py.test.

Unit tests are your best guide for refactoring. So you want a complete suite of tests that run very fast. That way you can change a line of code and know within a few seconds if that change is okay. Higher level tests are helpful too in that you could replace an entire component (perhaps something custom with something off-the-shelf) and see everything still works. Can you replace your database with confidence? That's the kind of feedback high level tests can give to help refactoring.

So a single test can satisfy all four of these needs. But it doesn't need to so it's better to figure out where the focus is for the test you are writing at the time. Some tests might only be there for documentation. It's okay for tests to evolve to help refactoring only when you need that refactoring. But don't assume that once a test is passing that it automatically ticks all these boxes. It's just a check at that point.

Popular posts from this blog

When Agile is not common sense

It's been about 8 years since I got my Agile training from James Grenning and started an Agile Pilot with my team. Right from the start I was super keen and got people practising TDD and Pair Programming even though I didn't really know if that effort would pay off. I just knew we had lots of problems with the old way so I wanted to embrace the new.

So I'd run my own little sessions with the broader team in Sydney sharing Agile concepts and ideas. Giving us a chance to discuss things and keep learning long after James had left. But often I would present new ideas and a few people in the group would respond "oh of course, that's just common sense". Initially I thought it was great as they were embracing these ideas as valuable even though it was not what we were doing before. But after a while I started to get frustrated because they we're not really listening. If some small part of a concept fit nicely with their "common sense" then they took th…

Setting up Fitnesse on Ubuntu in 7 steps

Some pretty basic steps but just to make sure it's here for everyone to see. Setting up fitnesse and running the jar is easy enough. Just go to http://fitnesse.org/ and get started and do it on your desktop just to see it in action. But for me that wasn't good enough I wanted it to run as service on ubuntu.

I stole a few tricks from how ubuntu runs jenkins and setup fitnesse a similar way.

1. Create a user and group for fitnesse (optional)
I didn't do this because I wanted tomcat, jenkins and fitnesse all running as the same user. Call it laziness to avoid any permissions classing but it doesn't change the process that you need to create or choose what user you're going to make it run as. Don't make it run as your user or root!

2. Download the jar file and place it in /usr/share/fitnesse
Make the folder too of course. It can belong to root as long as the fitnesse user has read access

3. Create the folder to run in at /var/lib/fitnesse
Fitnesse user needs write…

RestFixture

So most of the tests I'm writing now in Fitnesse are using RestFixture. Being able to do all this black box style testing has helped me get a lot of tests up and running without having to change the existing code base. Now I've taken a step future with my own little fork so I can use scenarios and build nice BDD style scripts. But first I want to give me own quick guide to using RestFixture

Step 1: Installing
You can dive straight in by grabbing the latest jar files for RestFixture here https://github.com/smartrics/RestFixture/downloads
If you know what you're doing can get the nodep version to work nicely along side other libraries you may be including in Fitnesse. But I grabbed the 'full' version and unzipped it into a RestFixture folder alongside my FitNesseRoot folder.
Step 2: Write your first test
I took advantage of the built in Fitnesse api as a basic test and wrote a page called RestFixture with the following contents
!define TEST_SYSTEM {slim} !path RestFix…