In the first post in my series on the seven tenets of software testing, I discussed one technique for identifying your most important tests. In this post I am going to discuss using combinations reducing the number of test cases that you need to execute.

If you want to impress all your friends with can refer to this topic as combinatorics. This is guaranteed to have nerds from miles around thinking you are “the man”, but it will also ensure that you will never be able to get a date with a member of the opposite sex again.

The problem

Let’s assume that we are building an application that we use determine the cost of a flight. We can choose from Sydney, Melbourne or Brisbane as destinations. We can choose from Business, Economy or Budget Economy as our type of ticket. Finally, we can choose Virgin Blue, Qantas or Jetstar as our airline. Let’s assume that there is a bug which occurs when Business and Jetstar are selected together, because Jetstar don’t have any business class seats. With all 27 possible combinations, we would get three test failures, as shown below.

With all 27 cases, there is obviously some duplication, as we get three failures from the same error. What can we do to intelligently reduce the number of tests? The trick we will use here is to test each unique pair of combinations at least once. Using a free tool like All pairs or TConfig to generate the pairs, reduces the list to the following 10 test cases.

So what exactly are we doing here? Well we are ensuring that we are testing each pair of values at least once. For example: Sydney and Business is one pair. Business and Jetstar is another. I personally used TConfig for this example and it produced the following output.

Degree of interaction coverage:            2
Number of parameters:                      3
Maximum number of values per parameter:    3
Number of configurations:                  10

1 | 1 1 1 2 | 1 2 2 3 | 1 3 3 4 | 2 1 2 5 | 2 2 1 6 | 2 3 1 7 | 3 1 3 8 | 3 2 1 9 | 3 3 2 10 | 2 2 3

All 2-way combinations covered. Initially I thought that the 10th test case was erroneous, and unnecessary. The 10th case is ensuring that all the 2-way combinations are covered. A 2-way combination means that it counts Sydney and Business as a different pair to Business and Sydney. This may or may not be an issue for your application, but it is good to know that the bases are being covered.

Ok so what is the trade-off with using pairs? Well if a bug only occurs when three or more things occur, such as a Qantas business class flight to Melbourne, then you are not guaranteed of finding it. You may get lucky if your test case just happens to match, but you are not guaranteed to find it.

So using pairs isn’t magic, it is simply a technique to ensure that you are intelligently reducing the number of tests that you have to perform, instead of just bumbling through and relying on blind luck.