Why run a test more than once? If it passed the first time, we don’t need to run it again – or do we? James Bach provides ten good reasons to run the same test more than once.
Common Sense
Common sense dictates that we don’t need to run the same test twice – once we’ve run a test, we know the results – and we therefore don’t need to run it again.
The definition of insanity is doing the same thing over and over again and inspecting different results.
Benjamin Franklin
James Bach writes a great article that points out that we can never do exactly the same thng every time, so running the same test again is not insane. He presents the following ten reasons to run the same test more than once. The commentary here is ours – check out his original post for more details, and other good writing on the topic.
- Recharge – By changing the code that is being tested, we recharge a test making it valuable again.
- Intermittence – Some bugs repeat as a function of otherwise uncontrolled variables.
- Retry – Sometimes we don’t run a test correctly, or can’t be certain that we ran it correctly.
- Mutation – By creating a test that is a variation on an existing test, we are repeating that subset of the test that is unchanged, but in a different context, established by the mutated version of the test.
- Benchmark – Performance measurement is most easily understood by establishing a baseline performance with a set of tests – then repeating those tests as the code is modified (or the platform it runs on is modified).
- Inexpensive – As long as there is some value to a test, and the cost of running it is low – what does it hurt?
- Importance – “This will never happen again” – best prevented with a regression test.
- Enough – James uses virus scanning as an example – honestly, this one doesn’t click for us. If you understand it, please add a comment.
- Mandated – The man always puttin’ me down, makin’ me run the same test over and over again.
- Indifference / Avoidance – What if your goal isn’t to find bugs, but to avoid them? What better way than to run the same test over again?
Sightings in the Real World
Continuous integration is an approach to testing/development that explicitly calls for the repeated running of tests as part of software development. In James’ framework, continuous integrations uses recharge, intermittence, and mutation as the drivers for repeating tests.
Combining unit tests and functional tests is an effective way to approach system testing. This takes advantage of mutation to provide effective coverage.
By contrast, pairwise testing is designed to achieve enough without repeating any tests.
