Teknologika

Found via Alex Barnett’s MSDN blog, this flash file reminds me of one of my favourite bands The Art of Noise, and is DEFINITELY worth a look.

This post is the first in a seven part series covering my seven tenets of software testing.

Why can’t I test everything?

Computer software is inherently complex, and even the simple act of adding two numbers together produces an infinite number of possibilities that would need to be tested to test all possible cases.

So what about something simpler, say entering a six character user name into a text box. Limiting ourselves to ASCII input still requires 127 x 127 x 127 x 127 x 127 x 127 = 4,195,872,914,689 test cases. If each test case takes us only 1 second to manually run, it will take 7.9 million years to test the user name field before we then moved onto the password textbox.

So if “complete” testing is out of the question, where should I start and how much testing should I do test before I can be confident that we can ship the product?

So where should I start?

The first step of the solution is to consciously accept that you can’t test everything, then begin to take proactive steps to decide what will and what wont be tested, as opposed to letting fate, or the project schedule decide for you.

The second step is to use the appropriate level of testing documentation, so you can spend as much time as possible executing tests, and not planning or updating them.

Most test design is performed in a waterfall style, where all the tests are planned, documented and then executed. This planning usually starts with the first requirement developing the tests in detail, then moves to the second, then the third and so on.

This approach has the danger that the early requirements have great tests, most requirements have ok tests, and he later requirements have crap tests, and get very little coverage as you run out of planning time.

Just as development has moved to an iterative or spiral lifecycle, so should your test planning. For your first iteration, hold a brainstorming session, with the Product Managers, developers, test team, or even just yourself. It doesn’t really matter you just want to think of as many things to test as possible in the shortest amount of time. (Hint: This makes a great testing interview question.)
Having more people from different roles helps to get a greater cross section of ideas. At this stage, do not concern yourself with the effort involved to perform a particular test, you just want to capture as many potential tests as you can.

Once you have your hand written list of test cases, you should transfer them into a spreadsheet, adding columns for estimated execution time, and numerical ratings say from 1-5, (where higher is better) for the: priority (relative importance) of the test, the visibility of the area to users, the quality of the existing code etc. The columns are then added together (in a column called “importance rating” to provide an overall score for the test. If you are testing a new release of an existing product you will probably want to add additional columns that identify the relative quality of the existing teat area.

With your spreadsheet in hand, you should circulate it and solicit input around the priority of the tests. You may even find it easier to submit the spreadsheet with a column for each area, say Testing, Development, Product Management and asking each area to fill in their own set of numbers. This will allow you to apply a weighting to each of the areas, say increasing the PM vote to 150% and decreasing the development vote to say 75%.

With the ratings complete you can then add a column to the spreadsheet to sum the execution times. Then, sort the spreadsheet by the importance rating column, work out how long your schedule gives you to test and then draw a line across the sheet when the estimated execution time equals the time that you have available, and that is your level of test coverage.

The sample shown below assumes that you had five tests and only 10 minutes to test with.

If you can execute all your test cases then great, go through another iteration and add more tests, if not, circulate the list to the key stakeholders and let them know this is the level of testing that you can currently perform, unless they add schedule, resources or both, a classic MSF tradeoff.

You can also apply combinatorics to reduce the number of tests, but that is a subject for another post.

Balancing the level of detail and formality in your test documents can be somewhat tricky, especially to the tester who has not been battle hardened by the experience of several different projects.

Over the years I have seen and used many different methods for my testing documents and there are some things that work really well and some that seem like a good idea when you start out, but quickly can become very cumbersome to use and maintain.

In a perfect world there would be one format for a test plan, and one for a test case that everyone could use. Everyone would rejoice, and the Von Trapp family continue to sing new and exciting songs as part of their daily lives.

Appropriate levels of testing documentation

In my experience I have seen, (and used) two extremes of testing documentation. In the dotcom boom I managed a great technical test team where we had no formal test documentation to speak of. There was documentation, just not very much of it around the planning and execution of tests.

What gave the customer confidence, and kept them happy was the fact that there was a ton of reporting around the defects that we were finding, and through the verbal updates we would continuously provide, I now refer to this as “Post-it™ test planning”.

In direct contrast to Post-it™ test planning is the test planning method favoured by waterfall methodologies where step-by-step test documents are written so that anyone’s mother could test the application, with little or no prior knowledge.

The ideal level of formality is somewhere between the two extremes, shown in the following diagram.

A good approach is to iterate through your tests, adding detail as you go. Ideally you should be able to stop when you can answer yes to all the following, as required for your project or organisation

• Is the test repeatable by an of our testers, so that anyone can execute the test and get the same result?
• Is there enough detail so that I can satisfy an audit?
• Can I know with absolute certainty what has, and has not been tested?
• Are the test cases in a format that can be quickly and easily updated, e.g. Excel or a database?
• Can I easily answer any questions about current progress, or coverage levels?

To conclude I will present some fictitious test cases for Notepad to show the difference between the approaches.
A good test case example

Applicaiton: Notepad

Test Area: File Save Dialogue

Requirement: REQ0342

Test: Verify that notepad can successfully save a file when the length of the file name greater than 255 characters, plus the .txt at the end.

A bad test case example

Applicaiton: Notepad

Test Area: File Save Dialogue

Requirement: REQ0342

Step 1: Press the start button.

Step 2: Select run.

Step 3: Type “Notepad.exe” and press return.

Step 4: Verify that Notepad opens successfully.

Step 6: Type “This is a test” into the new document.

Step 7: Select the file menu then save.

Step 8: Enter a file name that contains 255 characters, then add the .txt to the end of the file name.

Step 9: Press save.

Step 10: Verify the file saved successfully, with no errors.

Both tests achieve the same results, but one is much more manageable, and requires significantly less time to write.

Microsoft Developer evangelist Andrew Coates has blogged about Financial Services Webcast Week.


A “webcast series focused on real-world developer solutions for the financial services industry. Hear from the Microsoft financial services team as well as experts who have deployed technologies and systems at some of the largest financial institutions.“


As Andrew notes they are being recorded, but there is something absurdly fun about getting up at 4 AM and watching a live US based webcast, or then again, maybe I finally have gone completely nuts …

In both this MSDN magazine article and this episode of the .net show, Don Box introduced four fundamental tenets for developing service based or connected systems.

• Boundaries are explicit
  
• Services are autonomous
  
• Services share schema and contract, not class
  
• Service compatibility is determined based on policy

That inspired me to develop my own list of guiding principles that apply to software testing. These tenets are documenting some key learning’s from over the years working as a Test Manager, Senior Consultant and Development Manager for various software development shops.

• You can’t test everything so you have to focus on what is important.
  
• If you are going to run a test more than once, it should be automated.
  
• Test the product continuously as you build it.
  
• Base your decisions on data and metrics, not intuition and opinion.
  
• To build it, you have to break it.
  
• Apart from Test-Driven Development, A developer should never test their own software.
  
• A test is successful when the software under test fails.

In a series of future posts I will be expanding on the tenets, explaining them in detail, providing links to reference materials; hopefully providing something helpful for you to use on your projects.

Origional post (Mon, 6 Sep 2004)
To fully utilise the power of side by side deployment of ASP.NET sites running on IIS, through pain and experience we that the best way to do this is to set up a virtual root for each daily build and then simply point the master site to the virtual root of the “live” build.

There is a significant lack of flexibility available when you are doing this. We have found out the hard way that you need to add an additional host header with a unique port number to deploy to a virtual root, then call the msiexec command line and specify the TARGETPORT of the website.

e.g: www.example.com:8888 <- website host header

msiexec.exe /i c:\WebSetup.msi TARGETPORT=8888

This will install a virtual site under the website that uses port 8888, you can then remove the host header and you are done.

Update (Fri, 7 Jan 2005)

David Every has written a detailed step-by-step article (www.15seconds.com/issue/041223.htm) on how to get IIS side-by-side up and running.

The unofficial mac weblog posted an entry about the continued lack of an Australian version of the iTMS

Expectations were high that Apple planned to launch the Australian iTunes Music Store at a special Apple event held shortly after Steve Jobs’s keynote speech. Apple Australia introduced the new iPod Shuffle, but gave not a word about the Australian iTMS.

Australia has a different copyright system than the one in place in the US, which is said to be the reason for the hold up down under. iTunes likely won’t come to the Australian market until all partners are on board and the final deals are in place.

I’m sure Aussies are hopping mad that those lucky Luxembourgians got the goods before they did. Any Australian readers out there want to give word on the digital download market there.

I felt that it was my “Civic duty” to add my 2 cents worth:

“The whole “legal issues” argument just doesn’t smell right. As Trevor J. pointed out, there are other sites such as bigpond music that simply copied the iTMS store concept. They had to build sites from scratch, sign the labels and encode content and they still managed beaten Apple to the punch.

It really shows how US centric Apple is, especially when it comes to online services such as online iPhoto printing, and local sherlock plug-ins.

I am not holding my breath to see localised versions of the Dashboard widgets such as Australian weather, or Australian stock quotes either.”

I think that I finally found a machine to replace my ageing PIII 500…

Starting at just A$799, Mac mini is the ideal desktop computer for anyone looking to get started with Mac OS X and use iLife 05 software for managing digital photo and music collections, editing movies and creating music. Just 5 cm tall and weighing only 1.3 kg, Mac mini redefines design for the sub-$1,000 desktop.

From the Microsoft press release: “Based on technology acquired from GIANT Company Software Inc. in December 2004, the beta release of Windows AntiSpyware will help reduce negative effects caused by spyware, including slow PC performance, annoying pop-up ads, unwanted changes to Internet settings and unauthorized use of private information. It is available for Microsoft Windows 2000 and later versions.” You can download the beta here.