This post is the seventh and final post in a seven part series covering my seven tenets of software testing.

As I discussed in tenet five, all software has bugs, and the goal of testing is to find them.

One of the ironies of testing is that when a test case runs without error we give it a big green tick, and say that it has passed. By doing this, we are incorrectly reinforcing that a test is successful if it doesn’t find any bugs. If we want to find defects and improve the quality of our software, we want our tests to fail.

This is a subtle but very important difference in testing approach. A good analogy are tests performed by a doctor. When a doctor returns with test results for your sore leg, the last thing you want to hear is: “That pain in your leg, the tests didn’t find anything, so you are fine, just ignore it.”

There is a danger when we change our expectation away from “All tests must pass all the time”, to “We want tests to fail”. The danger is that our expectation will instead become: “Most of our tests should fail and it’s ok to have tests failing for weeks on end.”

Ideally, your goal should be to have a high fidelity test system, with a core set of automated regression tests, that suffers an occasional failure as developers evolve the application over time. In addition to the regression tests, you should be adding tests for things like known defects, and new tests to expand the test coverage. Ideally, you should expect any new test to fail the first few times it is run. When the issue you found is resolved, the test should execute without failure and then stay that way.

It is also important that your test suite has a high signal to noise ratio. As the number of tests increases, so will the amount of analysis that you need to perform when there are test failures.

Now if only I could charge for tests like a doctor does …