I always enjoy sneaking behind the curtain at Microsoft, (even though I am now working at Microsoft Sydney’s SDC myself). David Weiss has posted a Tour of Microsoft’s Mac Lab at the Macintosh Business Unit (MacBU), which is well worth a look.

Steve McConnell has recently released his latest book, Software Estimation: Demystifying the Black Art. If this is like any of Steve’s previous works, it will be another must own book for the library.

A good estimation book has been sorely needed in our industry for a while, so this will hopefully be the start of an estimation revolution for most software projects.

Good software design is all about trade-offs. If you want your application to be portable across operating systems, you may have to sacrifice performance. If you want your application to be secure, it may not be as user friendly as an unsecured version. When designing test automation, these types of trade-offs are no different. In my experience, for automation, there are two key design goals.

• Maintainability. Automated tests must be easy to maintain. Your tests will need to change, and a good automation framework should support and facilitate change, not restrict it. If your tests are not easy to maintain, they will quickly become a time consuming burden.
• Performance. Performance matters when it comes to automated tests. The tests that you write will be executed over and over again and if you can reduce each test by a couple of seconds, it starts to add up pretty quickly.

Along with goals, there are some things that we definitely don’t want to try and achieve. I don’t particularly like the term, but I’ll call them non-goals.

• Portability. Portability could be a goal if you wished. Typically, automation will be built using one automated tool, on a single platform. I say platform, as opposed to an operating system because a common scenario will be running the same test on different operating systems, for example Windows XP and Windows Server 2003. If you were building something like the .Net Common Language Runtime (CLR), portability would be a goal. Particularly, if you wanted to run the same test on the regular CLR, the Compact Framework on a handheld, and Rotor running on Mac OS X and Free BSD. In the case of the CLR however, you wouldn’t be using a GUI testing tool.
• Language independence. Oh I wish this could be a design goal. Realistically though, your tests will all have to be written in one language, either VBA or some form of “vendorscript”, such as WinRunner’s Test Script Language (TSL).

High Level Design
Our design will be loosely based on nunit as follows:

A setup method will be called at the start of every test to cleanup if a previous test has failed, and then launch the application we are testing.

A teardown method that is called at the completion of a test. Teardown determines if the test has passed or failed, writes the test result and then shuts down the application we are testing.

A number of assert methods that will do our verification, and pass or fail a test appropriately.

In a previous post Assert me! I included a comment about how most test tools mark tests as passed by default.

As you can see this is not marked as a failure or not run, it is recorded as a pass. This is a common error that all automated test tool developers tend to make. The default result for any automated test should be to fail (or some other non-passing state), unless explicitly passed during test execution.

We will endeavour to change the default test result behaviour along the way.

In my internet travels I have not come across any good guides on how to go about building a test framework with tools such as TestPartner, WinRunner, Rational Robot or TestComplete.

To try and fill this gap I am going to start a new series of posts on how to build a GUI automation stack. As much as possible this will be generic, and not focus on any one tool, but provide general guidelines that you can use on any of the tools that are available.

Why do I need a GUI test framework, doesn’t the tool just work out of the box?

Well yes they do work out of the box, but if you want to use them for any serious enterprise level testing effort, you will need to invest some time up front to make life easier in the long run.

In this series of posts I want to provide some of that guidance by essentially providing a high level road map of the path steps you need to take. Then you can either take those steps yourselves, of hire me, or someone else to take those steps for you.

This series of posts will cover the following:

• Design goals for a GUI testing framework
• Mapping the GUI
• Lifecycle of an Automated test
• Handling errors and writing results
• Controlling the browser with the IE object model
• Improving test perfomance

Plus anything else I think of along the way … or anyone wants to ask about, just send a mail or leave a comment.