Using Monte Carlo simulation and bug fix rates to calculate when the product will ship

Generally there are two key factors at play here: requirement delivery and bug fix rates.

The simple way

The simple answer is to average the number of bugs fixed per day, and divide the total number of bugs by the average. That is approximately how many days until you reach zero bugs. So, if you are fixing 5 bugs per day and you have 200 active bugs, the earliest that you will ship is in 40 working days time. If you want to ship sooner, you will need to stop adding features and focus on fixing more bugs.

The same information can be used in reverse to calculate a maximum allowable bug count. Say you only have 40 days until your desired ship date, and you are fixing 5 bugs per day as in the previous example. If you active bug count is over 200 today, you will probably miss your target. This number continuously decreases so in 2 weeks time, with 30 working days to go, your bug count should be at the 150 mark if you are going to hit your ship date.

The advanced way

In the 1950’s there was a little project that was underway at the Los Alamos National Laboratory in New Mexico. The manhattan project, required some advanced simulations, and they named the method they used after the Monte Carlo casino, which can be found at Turn 4 of the modern Monaco Formula 1 Circuit. There are essentially 4 steps to performing a Monte Carlo simulation:

  1. Define the inputs in to the problem.
  2. Randomly generate inputs for your problem.
  3. Apply the random values you generated to your problem.
  4. Repeat step 3, enough times to get statistically valid results, and then combine and analyse them.

Applying this method to our problem the steps become the following, assuming a data set where the bugs fixed – bugs found for a 2 weeks period results in the following data : 2,3,1,4,0,2,2,1,2,3

  1. Determine the average and standard deviation of bugs being fixed over time. For our example the average is 2 and the standard deviation is 3.16.

  2. Using Microsoft Excel, use the following function to generate valid random data =NORMINV(RAND(),Average,StandardDeviation).

  3. In an Excel worksheet use the values that you generated in step 1, to workout the number of days to ship, I used an IF function and then COUNTIF to count if the days to ship was greater than zero.

  4. In excel, copy the above 100 times and then plot the results.

ESDRawPlot.gif
ESDShippingChance.gif

The first chart is the scatter plot of the simulations. The second one shows the standard deviation of the estimated ship dates, and shows the chance of shipping in X number of days in the future.

The interesting thing about this model, is that even for 20 defects, and an average of fixing 2 per day the minimum days to ship is 3, but the maximum is 37 !

Test Management3 comments