Category Archives

Test Management

2010 Test Management Tool Survey

I have been thinking a lot about testing tools lately, and have my own (biased) ideas on what tools are great and which ones suck.

I thought that it would be interesting to create a survey and throw it out there to see what everyone else thinks and is using.

So … here is a link to my 2010 Test Management Tool  Survey http://www.surveymonkey.com/s/Y2X9SJR Please take the time to fill out the survey and tell your testing friends.

I will share the full results here on my blog, and if you all think that they all suck, I may even consider developing my own 🙂

Test Management0 comments

Using Monte Carlo simulation and bug fix rates to calculate when the product will ship

Generally there are two key factors at play here: requirement delivery and bug fix rates.

The simple way

The simple answer is to average the number of bugs fixed per day, and divide the total number of bugs by the average. That is approximately how many days until you reach zero bugs. So, if you are fixing 5 bugs per day and you have 200 active bugs, the earliest that you will ship is in 40 working days time. If you want to ship sooner, you will need to stop adding features and focus on fixing more bugs.

The same information can be used in reverse to calculate a maximum allowable bug count. Say you only have 40 days until your desired ship date, and you are fixing 5 bugs per day as in the previous example. If you active bug count is over 200 today, you will probably miss your target. This number continuously decreases so in 2 weeks time, with 30 working days to go, your bug count should be at the 150 mark if you are going to hit your ship date.

The advanced way

In the 1950’s there was a little project that was underway at the Los Alamos National Laboratory in New Mexico. The manhattan project, required some advanced simulations, and they named the method they used after the Monte Carlo casino, which can be found at Turn 4 of the modern Monaco Formula 1 Circuit. There are essentially 4 steps to performing a Monte Carlo simulation:

  1. Define the inputs in to the problem.
  2. Randomly generate inputs for your problem.
  3. Apply the random values you generated to your problem.
  4. Repeat step 3, enough times to get statistically valid results, and then combine and analyse them.

Applying this method to our problem the steps become the following, assuming a data set where the bugs fixed – bugs found for a 2 weeks period results in the following data : 2,3,1,4,0,2,2,1,2,3

  1. Determine the average and standard deviation of bugs being fixed over time. For our example the average is 2 and the standard deviation is 3.16.

  2. Using Microsoft Excel, use the following function to generate valid random data =NORMINV(RAND(),Average,StandardDeviation).

  3. In an Excel worksheet use the values that you generated in step 1, to workout the number of days to ship, I used an IF function and then COUNTIF to count if the days to ship was greater than zero.

  4. In excel, copy the above 100 times and then plot the results.

ESDRawPlot.gif
ESDShippingChance.gif

The first chart is the scatter plot of the simulations. The second one shows the standard deviation of the estimated ship dates, and shows the chance of shipping in X number of days in the future.

The interesting thing about this model, is that even for 20 defects, and an average of fixing 2 per day the minimum days to ship is 3, but the maximum is 37 !

Test Management3 comments

Revisting the testing tenets

Back in 2005 I wrote a series of seven blog posts called The Seven Tenets of Software Testing. These posts have been buried deep in this site, so I have added a new page – Tenets of software testing – that links to all these origional articles, hopefully making them easier to find if you are new to my software testing blog.

Advanced Testing Techniques Test Management Testing Tenets0 comments