This post is the first in a seven part series covering my seven tenets of software testing.
Why can’t I test everything?
Computer software is inherently complex, and even the simple act of adding two numbers together produces an infinite number of possibilities that would need to be tested to test all possible cases.
So what about something simpler, say entering a six character user name into a text box. Limiting ourselves to ASCII input still requires 127 x 127 x 127 x 127 x 127 x 127 = 4,195,872,914,689 test cases. If each test case takes us only 1 second to manually run, it will take 7.9 million years to test the user name field before we then moved onto the password textbox.
So if “complete” testing is out of the question, where should I start and how much testing should I do test before I can be confident that we can ship the product?
So where should I start?
The first step of the solution is to consciously accept that you can’t test everything, then begin to take proactive steps to decide what will and what wont be tested, as opposed to letting fate, or the project schedule decide for you.
The second step is to use the appropriate level of testing documentation, so you can spend as much time as possible executing tests, and not planning or updating them.
Most test design is performed in a waterfall style, where all the tests are planned, documented and then executed. This planning usually starts with the first requirement developing the tests in detail, then moves to the second, then the third and so on.
This approach has the danger that the early requirements have great tests, most requirements have ok tests, and he later requirements have crap tests, and get very little coverage as you run out of planning time.
Just as development has moved to an iterative or spiral lifecycle, so should your test planning. For your first iteration, hold a brainstorming session, with the Product Managers, developers, test team, or even just yourself. It doesn’t really matter you just want to think of as many things to test as possible in the shortest amount of time. (Hint: This makes a great testing interview question.) Having more people from different roles helps to get a greater cross section of ideas. At this stage, do not concern yourself with the effort involved to perform a particular test, you just want to capture as many potential tests as you can.
Once you have your hand written list of test cases, you should transfer them into a spreadsheet, adding columns for estimated execution time, and numerical ratings say from 1-5, (where higher is better) for the: priority (relative importance) of the test, the visibility of the area to users, the quality of the existing code etc. The columns are then added together (in a column called “importance rating” to provide an overall score for the test. If you are testing a new release of an existing product you will probably want to add additional columns that identify the relative quality of the existing teat area.
With your spreadsheet in hand, you should circulate it and solicit input around the priority of the tests. You may even find it easier to submit the spreadsheet with a column for each area, say Testing, Development, Product Management and asking each area to fill in their own set of numbers. This will allow you to apply a weighting to each of the areas, say increasing the PM vote to 150% and decreasing the development vote to say 75%.
With the ratings complete you can then add a column to the spreadsheet to sum the execution times. Then, sort the spreadsheet by the importance rating column, work out how long your schedule gives you to test and then draw a line across the sheet when the estimated execution time equals the time that you have available, and that is your level of test coverage.
The sample shown below assumes that you had five tests and only 10 minutes to test with. If you can execute all your test cases then great, go through another iteration and add more tests, if not, circulate the list to the key stakeholders and let them know this is the level of testing that you can currently perform, unless they add schedule, resources or both, a classic MSF tradeoff.
You can also apply combinatorics to reduce the number of tests, but that is a subject for another post.