My presentation at the Microsoft SDC open day alongside Devtest CEO Sarah Richey is now up on Microsoft’s site.
http://www.microsoft.com/australia/services/microsoftservices/sdc_openday.mspx
For my presentation you need to open the “How we do: Testing” video and I am in the last half following Sarah.
Generally there are two key factors at play here: requirement delivery and bug fix rates.
The simple answer is to average the number of bugs fixed per day, and divide the total number of bugs by the average. That is approximately how many days until you reach zero bugs. So, if you are fixing 5 bugs per day and you have 200 active bugs, the earliest that you will ship is in 40 working days time. If you want to ship sooner, you will need to stop adding features and focus on fixing more bugs.
The same information can be used in reverse to calculate a maximum allowable bug count. Say you only have 40 days until your desired ship date, and you are fixing 5 bugs per day as in the previous example. If you active bug count is over 200 today, you will probably miss your target. This number continuously decreases so in 2 weeks time, with 30 working days to go, your bug count should be at the 150 mark if you are going to hit your ship date.
In the 1950’s there was a little project that was underway at the Los Alamos National Laboratory in New Mexico. The manhattan project, required some advanced simulations, and they named the method they used after the Monte Carlo casino, which can be found at Turn 4 of the modern Monaco Formula 1 Circuit. There are essentially 4 steps to performing a Monte Carlo simulation:
1. Define the inputs in to the problem.
2. Randomly generate inputs for your problem.
3. Apply the random values you generated to your problem.
4. Repeat step 3, enough times to get statistically valid results, and then combine and analyse them.
Applying this method to our problem the steps become the following, assuming a data set where the bugs fixed – bugs found for a 2 weeks period results in the following data : 2,3,1,4,0,2,2,1,2,3
1. Determine the average and standard deviation of bugs being fixed over time. For our example the average is 2 and the standard deviation is 3.16.
2. Using Microsoft Excel, use the following function to generate valid random data =NORMINV(RAND(),Average,StandardDeviation).
3. In an Excel worksheet use the values that you generated in step 1, to workout the number of days to ship, I used an IF function and then COUNTIF to count if the days to ship was greater than zero.
4. In excel, copy the above 100 times and then plot the results.
The first chart is the scatter plot of the simulations. The second one shows the standard deviation of the estimated ship dates, and shows the chance of shipping in X number of days in the future.
The interesting thing about this model, is that even for 20 defects, and an average of fixing 2 per day the minimum days to ship is 3, but the maximum is 37 !
Over the years we have noticed a phenomenon where developers can become reliant on testing. This addiction is a good thing, as it leads to a much better end product being developed. As a by product we have identified 5 stages that developers go through in their test addiction.
Stage 1 : Denial or “We don’t need testing, we are shipping fine without it thanks.”
Developers in stage one are under the illusion that you can simply compile you code and then ship it. Most developers at this stage are seeking compliance with the works on my machine certification. The particularly dangerous developer at this stage is the one who takes offence when his god-like development skills are revealed as capable if causing bugs just like everybody else.
Stage 2 : Annoyance or “The ‘ Pointy-haired boss’ has hired some outside testing company”
Someone from on-high who the developers think of as a pointy haired boss has forced some testers onto the project. It may come as a surprise, but some managers aren’t stupid and don’t believe a developer who says that they are “almost finished” or “95% done”. At this stage, testers are seen as little more than an annoyance who is forcing developers to do extra work like daily builds, smoke testing and tracking bugs.
Stage 3 : Acceptance or “I suppose this isn’t a temporary thing is it?”
At the acceptance stage, the developers have accepted that this new trend of testing may actually catch on, and that it isn’t just a passing fad. Developers will now be used to looking at their issues queues in the bug tracking application, and realise that, yes, they do need to tell the test automatiors that they have renamed the login button on the home page of the application.
Stage 4 : Dependence or “I am at one with my testing brothers and sisters”
After the acceptance stage comes dependence. At this stage the developers on a project have come to rely on the feedback loop that comes from fully engaging with the testers on your team. They get their code verified BEFORE they check it in, they rely on the suite of automated tests to verify that their changes have not broken and they may even from time to time raise the odd defect themselves.
You feel guilty doing any coding without unit tests. You half consider adding tests to the changes you just made to your autoexec.bat! (Thanks Russ !)
Stage 5 : Withdrawal or “Oh my god, what do you mean you don’t have any testers?”
The withdrawal stage when a developer who is used to being at level 5, changes jobs and is plummeted back into a team at stage 1. I’m sorry, but the prognosis for this situation is not good, and the only real solution is to progress your team through all 4 stages right from the very beginning. Who knows, if you suggest it, they may think that you are beginning to sound like a pointy-haired boss.
If you want to get your team to stage 4, feel free to drop me a line.
Microsoft have released an alpha version of a best practice analyser for ASP.NET. Without using the tool yet, it looks like the tool scans your config files, and not your code or your aspx pages. You can download the tool here.
Found via : Joe Morel’s Blog – Best Practice Analyzer for ASP.Net Alpha Released!
From Scrum for Team System, which was found via Rob Caron’s blog post Scrum for Team System Released.
Scrum for Team System is a free Agile Software Development Methodology add-in for Microsoft Visual Studio Team System, developed by Conchango, in collaboration with Ken Schwaber and the Microsoft Technology Centre UK.
Scrum for Team System provides development teams with deep support for the use of Scrum, when running projects using Visual Studio Team System’s integrated suite of lifecycle tools.
Cruise on over to the Gotdotnet workspace for the latest MSF Agile information.
It’s always good to see a company practising what it preaches, and Microsoft have just performed what must be one of the most visible executions of trade-off through the decision to ship Longhorn without WinFS and optimise for schedule.
For a product like an operating system, it seems logical that there would be two key “phases” where trade-off would be lead by distinctly different priorities. Initially in the development cycle, the balance would be geared towards features over schedule, to help flush out the cool new features that will provide a compelling reason for customers to upgrade to the new version.
Once the compelling features have been designed and initially developed then the project should switch into ship mode and change the priority to hitting a schedule, otherwise there is a danger of becoming like Taligent and never shipping anything. (Don Box uses a term called: “The Taligent Effect”, which he discusses here.)
In modern motor racing, such as formula one, each car is wired with a multitude of sensors that detect almost every possible influence or reaction on the racing car that it is possible to measure. Ride height, suspension travel, wheel spin, steering input and many other measurements all get recorded and analyzed in an effort to squeeze some more performance out of the race car. During a race engineers will monitor the engine performance and fuel consumption in real-time to help the tactical decisions that are made during a race, or slow the car down to avoid an impending engine failure. After the race is long over the data is analyzed to help tune the setup of future events and help evolve the car in an effort to win future races.
Software development can benefit in exactly the same way from this type of analysis and monitoring. Unlike a race car it isn’t necessary to purchase specialised measuring equipment to record data, as it is mostly already being recorded. However, like the racing counterpart it is necessary to spend some effort to yield some useable information and valuable results.
One of the most valuable tools that is available to a Software Development team are the statistics and metrics that can be recorded during a project, as it progresses, or in a post mortem at the end of a project. These different types of measures provide tactical and strategic benefits to an organization during a project. Tactical metrics are produced at a regular interval throughout the life of a project, (usually daily) and provide a close to real-time view into the health of a project.
Tactical metrics are typically produced by the test manager and include the defect trend graph, The defect trend is the single most valuable indicator that I use on my projects, it is the closest thing to an electrocardiogram (ECG) that I have seen for a software project. This graph plots active defects over the life of the project and provides one of the most valuable insights into the progress of both the testing system and the quality of the product itself.
So what exactly is an active defect? An Active defect is any defect that has not been closed, regardless of where it is in the defect cycle. For example, if a fix has been completed and is ready to be verified by a tester, is still counted as an active defect.
Other graphs that are quite valuable include reports that identify the most unstable components in the system, and test execution reports that indicate the amount of test coverage.
Another key graph is the “worm”, which shows daily how you are actually progressing compared to the original plan.