This is the video of the presentation that I gave at the Microsoft Solutions Centre open day.
The full day of presentations is also available up at microsoft.com.
Agile Testing at the Microsoft SDC
Reply
Category Archives: Test Management
2010 Test Management Tool Survey
I have been thinking a lot about testing tools lately, and have my own (biased) ideas on what tools are great and which ones suck.
I thought that it would be interesting to create a survey and throw it out there to see what everyone else thinks and is using.
So … here is a link to my 2010 Test Management Tool Survey http://www.surveymonkey.com/s/Y2X9SJR Please take the time to fill out the survey and tell your testing friends.
I will share the full results here on my blog, and if you all think that they all suck, I may even consider developing my own 
Using Monte Carlo simulation and bug fix rates to calculate when the product will ship
Generally there are two key factors at play here: requirement delivery and bug fix rates.
The simple way
The simple answer is to average the number of bugs fixed per day, and divide the total number of bugs by the average. That is approximately how many days until you reach zero bugs. So, if you are fixing 5 bugs per day and you have 200 active bugs, the earliest that you will ship is in 40 working days time. If you want to ship sooner, you will need to stop adding features and focus on fixing more bugs.
The same information can be used in reverse to calculate a maximum allowable bug count. Say you only have 40 days until your desired ship date, and you are fixing 5 bugs per day as in the previous example. If you active bug count is over 200 today, you will probably miss your target. This number continuously decreases so in 2 weeks time, with 30 working days to go, your bug count should be at the 150 mark if you are going to hit your ship date.
The advanced way
In the 1950’s there was a little project that was underway at the Los Alamos National Laboratory in New Mexico. The manhattan project, required some advanced simulations, and they named the method they used after the Monte Carlo casino, which can be found at Turn 4 of the modern Monaco Formula 1 Circuit. There are essentially 4 steps to performing a Monte Carlo simulation:
- Define the inputs in to the problem.
- Randomly generate inputs for your problem.
- Apply the random values you generated to your problem.
- Repeat step 3, enough times to get statistically valid results, and then combine and analyse them.
Applying this method to our problem the steps become the following, assuming a data set where the bugs fixed – bugs found for a 2 weeks period results in the following data : 2,3,1,4,0,2,2,1,2,3
Determine the average and standard deviation of bugs being fixed over time. For our example the average is 2 and the standard deviation is 3.16.
Using Microsoft Excel, use the following function to generate valid random data =NORMINV(RAND(),Average,StandardDeviation).
In an Excel worksheet use the values that you generated in step 1, to workout the number of days to ship, I used an IF function and then COUNTIF to count if the days to ship was greater than zero.
In excel, copy the above 100 times and then plot the results.
The first chart is the scatter plot of the simulations. The second one shows the standard deviation of the estimated ship dates, and shows the chance of shipping in X number of days in the future.
The interesting thing about this model, is that even for 20 defects, and an average of fixing 2 per day the minimum days to ship is 3, but the maximum is 37 !
Revisting the testing tenets
Back in 2005 I wrote a series of seven blog posts called The Seven Tenets of Software Testing. These posts have been buried deep in this site, so I have added a new page – Tenets of software testing – that links to all these origional articles, hopefully making them easier to find if you are new to my software testing blog.
The 5 stages of the developer-test relationship
Over the years we have noticed a phenomenon where developers can become reliant on testing. This addiction is a good thing, as it leads to a much better end product being developed. As a by product we have identified 5 stages that developers go through in their test addiction.
Stage 1 : Denial or “We don’t need testing, we are shipping fine without it thanks.”
Developers in stage one are under the illusion that you can simply compile you code and then ship it. Most developers at this stage are seeking compliance with the works on my machine certification. The particularly dangerous developer at this stage is the one who takes offence when his god-like development skills are revealed as capable if causing bugs just like everybody else.
Stage 2 : Annoyance or “The ‘ Pointy-haired boss’ has hired some outside testing company”
Someone from on-high who the developers think of as a pointy haired boss has forced some testers onto the project. It may come as a surprise, but some managers aren’t stupid and don’t believe a developer who says that they are “almost finished” or “95% done”. At this stage, testers are seen as little more than an annoyance who is forcing developers to do extra work like daily builds, smoke testing and tracking bugs.
Stage 3 : Acceptance or “I suppose this isn’t a temporary thing is it?”
At the acceptance stage, the developers have accepted that this new trend of testing may actually catch on, and that it isn’t just a passing fad. Developers will now be used to looking at their issues queues in the bug tracking application, and realise that, yes, they do need to tell the test automatiors that they have renamed the login button on the home page of the application.
Stage 4 : Dependence or “I am at one with my testing brothers and sisters”
After the acceptance stage comes dependence. At this stage the developers on a project have come to rely on the feedback loop that comes from fully engaging with the testers on your team. They get their code verified BEFORE they check it in, they rely on the suite of automated tests to verify that their changes have not broken and they may even from time to time raise the odd defect themselves.
You feel guilty doing any coding without unit tests. You half consider adding tests to the changes you just made to your autoexec.bat! (Thanks Russ !)
Stage 5 : Withdrawal or “Oh my god, what do you mean you don’t have any testers?”
The withdrawal stage when a developer who is used to being at level 5, changes jobs and is plummeted back into a team at stage 1. I’m sorry, but the prognosis for this situation is not good, and the only real solution is to progress your team through all 4 stages right from the very beginning. Who knows, if you suggest it, they may think that you are beginning to sound like a pointy-haired boss.
If you want to get your team to stage 4, feel free to drop me a line.