This post is the third in a series of posts, building a GUI test automation framework. In the first post, I outlined the key design goals of an automation framework. In the second second post I talked to the need for a control file. Since the original posts way back in early 2006, a lot has changed. As a result I am going to re-focus (and finish) these posts on how you can structure and develop a set of simple tests using WatiN.

Ok there are some pre-requites if you want to run these examples. I am going to assume that you are running Visual Studio 2005, SQL Express 2005, nUnit 2.4.0 and we are using the asp.net job site starter kit as our test application. I was planning to use the .net petshop 4.0 for this article, however it wouldn’t install with Windows Vista.

Ok assuming you are up and running, let’s get started.

The general structure that I like to follow for this type of project is to have a single solution broken up in to several projects. The standard projects that I like to have are Controls, Framework and Tests. In this example I am also going to add a fourth project to hold the site URLs.

Once I have these projects in place I set up the classes so they are all using single namespace that is then subdivided in a similar fashion such as: JobSite.Controls, JobSite.Tests and JobSite.URLs.

You can download the solution with the basic structure from here. I have set up a Codeplex project http://www.codeplex.com/WatinJobsiteSample/ to store the source code for the examples in the rest of this series of blog posts. In the next post we will get seriously back on track and start putting our test framework together.

This post is the second in a series of posts, building a GUI test automation framework. In the previous post, I outlined that maintainability was one of our key design goals. There are two key techniques I use to develop maintainable tests.

1. Reducing the number of lines of code in a test to the absolute minimum.

2. Mapping the application’s GUI so when a control changes, you only need to change it’s definition in one place.

For this post I will base my example on the new Login control that ships with ASP.net 2.0.

Record and playback is evil

Recorded tests shouldn’t be used in any serious test automation approach. To show why, I will record entering a username, password and pressing the login button in an unnamed commercial tool.


' Begin Record on Monday, 21 May 2006 at 6:45:39
' PM by BruceM

' Attach to Untitled Page - Microsoft
' Internet Explorer Window
Window("Untitled Page - Microsoft Internet Explorer Window").Attach

' Attach to Caption='Untitled Page'
HTMLBrowser("Caption='Untitled Page'").Attach
HTMLEditBox("Name=Login1$UserName").Click 47, 11
HTMLEditBox("Name=Login1$UserName").SetText "username"
HTMLEditBox("Name=Login1$Password").SetText "password"
HTMLButton("Name=Login1$LoginButton").Click

' End Record on Monday, 21 May 2006 at 6:45:51 PM
' by BruceM
First things first, the code is just plain ugly. Indenting and spacing is all over the place, the comments are bad and it follows no known standard that I know of.

Now let’s say that we have this code at the start of every test, and we have 400 tests in our test suite. Our lead UI developer has decided to replace the login control with their own and he renames the login controls as follows:
"UserName" now becomes "txtUsername"
"Password" becomes "txtPassword"
"LoginButton" now becomes "btnLogin"

To change the control names in all our tests, we would need to change 1600 lines of code, and that is painful, not to mention a complete waste of time that could be spent elsewhere.

Enter the GUI mapping file

To avoid this situation, the fix is technically quite simple. If we define variables for all the controls in a single file, which all the tests refer to, we only need to update the mapping file in a single location instead of each and every test.

Mercury’s WinRunner uses this approach for it’s GUI map files, which contain all the definitions in a single file.

If aren’t using WinRunner, (I’m not), and your tool does not have a GUI map, you can simply extend your tool and “roll your own”. I have used various solutions to build GUI maps from XML files and the MSXML DOM, to a class that is full of public constants similar to the following.

Public Const LOGIN_PAGE as string = "Caption='Untitled Page'"
Public Const LOGIN_USERNAME_TEXTBOX as string = "Name=Login1$txtUserName"
Public Const LOGIN_PASSWORD_TEXTBOX as string = "Name=Login1$txtPasword"
Public Const LOGIN_BUTTON as string = "Name=Login1$LoginButton"

This method is nice that with a good tool, you should now have code completion happening in your editor. You also don’t have a runtime performance issue looking up controls in an XML file which you get with the DOM method.

Our updated code is now as follows:

' Begin Record on Monday, 21 May 2006 at 6:45:39
' PM by BruceM

' Attach to Untitled Page - Microsoft
' Internet Explorer Window
Window("Untitled Page - Microsoft Internet Explorer Window").Attach

' Attach to Caption='Untitled Page'
HTMLBrowser(LOGIN_PAGE).Attach
HTMLEditBox(LOGIN_USERNAME_TEXTBOX).Click 47, 11
HTMLEditBox(LOGIN_USERNAME_TEXTBOX).SetText "username"
HTMLEditBox(LOGIN_PASSWORD_TEXTBOX).SetText "password"
HTMLButton(LOGIN_BUTTON).Click

' End Record on Monday, 21 May 2006 at 6:45:51 PM
' by BruceM

Common GUI map gotcha’s

Moving all your controls to a GUI map solves one problem but creates another. With all the controls now being defined by your test developers, you can have different developers define the same control using different standards. e.g.:

Public Const LOGIN_PASSWORD_TEXTBOX as string = "Name=Login1$txtPasword"
Public Const TEXTBOX_PASSWORD as string = "Name=Login1$txtPasword"

I recommend that you should develop a standard, and stick to it. The goal is that if anyone adds a new control, it will have the same variable name regardless of who defined it. A standard may be PAGENAME_CONTROLNAME_CONTROL_TYPE, or tlaPagenameControlName. (Where tla is the three letter acronym for the control type, e.g. txt for a textbox.)

If you can just use the same standard the developers are using for the application itself.

At this point you might be asking, if the test code produced is completely un-maintainable, why do the tools provide that feature. The answer is that tool vendors are in the business of selling tools, not selling maintainable automated tests. Providing record and playback in an automated test tool allows testers who can’t code to quickly produce an automated test, by simply ignoring the maintainable test goal. Personally I only use record and playback to know what command I should be using to interact with a particular control, before rewriting that command myself.

In my next post in this series, I will cover using functions to reduce the number of lines of code in each test.

Good software design is all about trade-offs. If you want your application to be portable across operating systems, you may have to sacrifice performance. If you want your application to be secure, it may not be as user friendly as an unsecured version. When designing test automation, these types of trade-offs are no different. In my experience, for automation, there are two key design goals.

• Maintainability. Automated tests must be easy to maintain. Your tests will need to change, and a good automation framework should support and facilitate change, not restrict it. If your tests are not easy to maintain, they will quickly become a time consuming burden.
• Performance. Performance matters when it comes to automated tests. The tests that you write will be executed over and over again and if you can reduce each test by a couple of seconds, it starts to add up pretty quickly.

Along with goals, there are some things that we definitely don’t want to try and achieve. I don’t particularly like the term, but I’ll call them non-goals.

• Portability. Portability could be a goal if you wished. Typically, automation will be built using one automated tool, on a single platform. I say platform, as opposed to an operating system because a common scenario will be running the same test on different operating systems, for example Windows XP and Windows Server 2003. If you were building something like the .Net Common Language Runtime (CLR), portability would be a goal. Particularly, if you wanted to run the same test on the regular CLR, the Compact Framework on a handheld, and Rotor running on Mac OS X and Free BSD. In the case of the CLR however, you wouldn’t be using a GUI testing tool.
• Language independence. Oh I wish this could be a design goal. Realistically though, your tests will all have to be written in one language, either VBA or some form of “vendorscript”, such as WinRunner’s Test Script Language (TSL).

High Level Design
Our design will be loosely based on nunit as follows:

A setup method will be called at the start of every test to cleanup if a previous test has failed, and then launch the application we are testing.

A teardown method that is called at the completion of a test. Teardown determines if the test has passed or failed, writes the test result and then shuts down the application we are testing.

A number of assert methods that will do our verification, and pass or fail a test appropriately.

In a previous post Assert me! I included a comment about how most test tools mark tests as passed by default.

As you can see this is not marked as a failure or not run, it is recorded as a pass. This is a common error that all automated test tool developers tend to make. The default result for any automated test should be to fail (or some other non-passing state), unless explicitly passed during test execution.

We will endeavour to change the default test result behaviour along the way.