I am pretty proud of our WatiN stack, it is in it’s third major version now and is an absolute joy to work with.

After seeing some other’s code samples online I thought that I should share some of the details so people can improve what they are doing. We recently hit a new one day record with one of our SDE/T’s writing 72 new test cases in a single day.

How it works

Our stack is broken down into four key layers, the test, a logical layer, a physical layer and then WatiN itself. If we are working on a Winforms application, we replace WatiN with an in-house developed stack that uses the Windows UI automation API’s. Our test data is stored in a SQL express database and we have a generated data access layer, which keeps the cost of creating CRUD methods very low. One of our tests looks like this:

[TestMethod]
public void VerifyCustomerCreation()
{
    Logical.Customer.CreateNewCustomer("Default Customer");
    Verification.Customer.VerifyCustomerCreated();
}

And that’s it. The tests are written in the business domain of the application that we are testing, not in a pile of clicks and set values. The next level down the “logical” layer is called by the tests and looks like this:

public static void CreateNewCustomer(string customerName)
{
    Physical.Customer.GoToNewCustomerPage();
    Physical.Customer.EnterNewCustomerInfo(customerName);
    Physical.Customer.SaveNewCustomer();
}

We build our logical layer up by combining smaller atomic actions, this allows us to heavily use the DRY (don’t repeat yourself) approach.

Our physical layer is where the rubber hits the road and it looks like this:

public static void EnterNewCustomerInfo(string customerName)
{
    ControlHandler conroller = new ControlHandler();
    TestData.Customer customerDetails = TestData.Customer.GetCustomerDetails(customerName);

    controller.SetValue(NewCustomerPage.txtFirstName, customerDetails.FirstName);

    controller.Invoke(NewCustomerPage.btnSave);
}

The control handler fully abstracts WatiN, or another automation tool so we interact with any control by typically only calling either SetValue, Invoke or GetValue.

To define our controls we have an in-house developed tool where we drag a magnifying glass (Spy++ style) onto a page and it then captures all the controls and gives us a generated static class.

This stack is the result of a lot of investment in our time and represents, in my opinion, state of the art when it comes to web test automation. But then again, I am a bit biased because I wrote it .

With the latest commit to the http://code.google.com/p/mcwatin/ repository can now open and close firefox all by itself.

The next feature to add is detection if a browser is open with jssh running and to re-use that browser instead of always launching a new one, or close as appropriate.

On thing that I have noticed is that the lack of a debugger in mono develop on the mac is really slowing me down as I can’t easily follow the code execution.

So I haven’t worked on my WatiN Mac os x port for a few days, but I got back into it last night. The main hurdle at the moment is that the System.Diagnostics.Process class is not fully baked on Mono 2.4 on the Mac platform, because it doesn’t implement the /proc filesystem that linux does.

The correct thing to do would be to learn how the Mach kernel process structure works and fix mono, but I haven’t programmed in non managed C since 1990 and I would be very dangerous, especially at such a low level in mono.

So the first approach was to use the parts of mono that were working, namley Proces.Start() to call a shell script that I wrote to /tmp, then redirect standard out to a file, then parse that file and act accordingly.

StreamWriter sw = new StreamWriter("/tmp/watin-listprocess.sh",false);
sw.WriteLine (@"ps x | grep" + processname + " | grep -v grep > /tmp/watin-processlist.out");
sw.Flush();
sw.Close();

Process shScriptProcess = new Process();
shScriptProcess.StartInfo = new ProcessStartInfo("/bin/sh","/tmp/watin-listprocesses.sh");
shScriptProcess.Start();

A hack, yes, in fact a very messy hack. So messy there had to be a better way.

And there is …

Enter Monobjc. Monobjc is a managed wrapper around the Mac os x Cocoa API’s. A quick search pointed to the NSTask and NSPipe classes as a way to start a process with NSTask, and capture standard out with NSPipe, then parse the result.

A much more technically correct solution.

public static void Main(string[] args)
{
// spin up the objective-c runtime
ObjectiveCRuntime.LoadFramework("Cocoa");
ObjectiveCRuntime.Initialize();
NSAutoreleasePool pool = new NSAutoreleasePool();

// Create our process
NSTask task = new NSTask();
NSPipe standardOut = new NSPipe();
task.StandardOutput = standardOut;
task.LaunchPath = @"/bin/ps";

// add some arguments
NSString argumentString = new NSString("-ax");
NSArray arguments = NSArray.ArrayWithObject(argumentString);
task.Arguments = arguments;

// We have liftoff
task.Launch();

// Parse the output and display it to the console
NSData output = standardOut.FileHandleForReading.ReadDataToEndOfFile;
NSString outString = new NSString(output,NSStringEncoding.NSUTF8StringEncoding);
Console.WriteLine(outString);

// Dipose our objects, gotta love reference counting
pool.Release();
}

So this needs to be fully baked in and then put into my Watin on mac os x port and then it should work stand alone, without shell script assistance.

Twelve moths ago, before my last project started I went on a journey. The intended destination was test automation nirvana. The way to get there, or so I thought was to combine WatiN with The Braidy Tester’s Automation Stack. The end result, I hoped would be the Deep Thought of test automation. It didn’t quite work out that way, and I would like to share some o the challenges we faced.

The Braidy Tester’s automation stack

I am not going to describe the architecture in detail of the Braidy Tester’s automation stack, as Michael Hunter has already covered that in great detail. What IS important however is the pieces that we chose. The key aspects of the stack that we decided implement were:

• Logical Functional Model The logical functional model is a user centric view of how the application is used. For example: Logical.Google.Search("Bruce McLeod");
• Physical Object Model The physical object model is an abstraction layer that hides the implementation and automation semantics details from the tests. For Example:Physical.Google.btnSearch.Invoke();
• Execution Behaviour Manager Execution behaviours are a way of loosely coupling execution actions. They allow executions to be composed, as well as executed in different sequences.
• Loosely Coupled Verification Instead of littering your tests with assert statements, loosely coupled verification takes a snapshot of the application, calculates an expected state based on your execution behaviour and then compares it to the actual state after the action is performed.
• Test Data Providers Test Data providers remove the need for test data to be hard coded into the test by moving it out, into a database in our case.

Things that worked brilliantly in the automation stack

The separation into the logical and physical layers worked exceptionally well. The test cases become very clean and concise. We added another lower level layer called the controller in place of the Application Internals model.

The controller essentially was an abstraction later over WatiN and our control definitions. This makes the entire stack easily portable from one automation technology to another, so the transition will be very easy when the new automation tools ship in the next version of Visual Studio Team Test, if we decide to change.

Implementing Execution Behaviours with Delegates

The idea behind Execution Behaviours is that you use attributes on a method that matches the delegate signature to perform one of the following actions:

• Execute all the specified actions in the listed sequence. ExecuteInSequence
• Execute one of the actions "randomly". ChooseAny
• Execute all the actions in the listed sequence until some condition is met. ExecuteUntil

Execute in sequence is straight forward and is used to compose actions.

[CompositeExecutionBehavior("OpenSearchPage", "WebSearch")]

public static void Search(TestDataProvider dataProvider)

{

   TestActionDelegate myDelegate = TestExecutionManager.ExecuteInSequence(Logical.Google. Search(TestDataProvider dataProvider);

}


With delegates being used and the possibility of execution sequence order changing, it becomes necessary to provide inherit record and replay for actions. We chose to use a version of the command pattern and use .net serialization to achieve this. We also had to package up the parameters into an array of objects to the could be serialized. The implementaiton of one of our actions in the physical layer would look something like this:

public static void OpenSearchPage(TestDataProvider dataProvider)

{

   // Add this action to the command list and serialize it

   object[] parameters = { dataProvider };

   ExecutionManager.Add(parameters);

   // Open the search page

   SearchModel searcher = new SearchModel();

   searcher.OpenSearchPage(dataProvider);

}

This then calls the physical layer that actually does the work. All this loosely coupled goodness is great, except that we only ever used the one type of execution behaviour, execute in sequence. As a result our code could have actually looked like this instead if it was hard coded in the logical model.

public static void Search(TestDataProvider dataProvider)

{

   SearchModel searcher = new SearchModel();

   searcher.OpenSearchPage();

   searcher.Search(dataProvider);

}

That code is much simpler and easier to maintain, and removes the method signature matching constraints that delegates impose. For us it would have been a much better way to go, simply because we would have removed the constraints required by functionality that we didn’t actually implement.

In the next post, I’ll run through some of the benefits and challenges we had implementing loosely coupled verification.

In my last post I gave a glimpse into our new WatiN stack, another feature is that we have implemented the command pattern, and serialise the commands out as XML. This lets us build tests that have randomised execution and still replay them when they fail. As a nice by-product, we also can write tests as XML if we wish.

After a month of pushing my gray matter as far as it will go (which isn’t all that far compared to some of the people I know) I have implemented the next generation of automated testing on my project.

No it isn’t using excel to generate tests like some people claim, although that does have it’s place. I have implemented a stack based on the design used in the expression team as described by Michael Hunter aka “The Braidy Tester”.

This transforms a test case that was like this:

using (IE ie = new IE("http://www.google.com"))
{
    ie.TextField(Find.ByName("q")).TypeText("WatiN");
    ie.Button(Find.ByName("btnG")).Click();
    Assert.IsTrue(ie.ContainsText("WatiN"));
}

to

Logical.Search(DataProvider.GetProvider("SearchForWatin"));

The next layer down is:

public static void Search(DataProvider dataProvider)
{
    object[] parameters = {dataProvider};
    executioner.Add(parameters);
    SearchModel searchModel = new searchModel();
    searchModel.OpenSearchPage();
    VerificationManager.ActionStarting(parameters);
    searchModel.Search(dataProvider);
    VerificationManager.ActionEnding(parameters);
}

then onto the “guts” of the test in the physical model.

public void Search(DataProvider dataProvider)
{
    SearchTDO search = DataProvider.GetSearchTDO();
    controller.SetValue(Controls.Search.Google.txtSearch,
         search.SearchText);
    controller.Invoke(Controls.Search.Google.btnSearch);
}

In case you didn’t notice, WatiN is fully abstracted away out of the tests, as is the test data, verification and the intent of how things are done. This increases the complexity significantly, but it also allows us to handle changes VERY easily.

Tonight I noticed a cool project on codeplex called WAX http://www.codeplex.com/wax, whilst this isn’t something that I will use myself, I have done something similar in the past, and generating tests in excel can save a LOT of time.

The project that I am currently working on uses infopath forms. This was causing us a few hassles on the testing front, especially now we are using WatiN extensively. So after a few failed attempts at using the new .Net 3.0 automaton API’s and not really wanting to learn MSAA, I decided to create a specalised port of WatiN to test infopath and ItiN was born.

ItiN uses a hybrid approach allowing manipulation of an InfoPath document directly using XPath and the XML dom, or the view using traditional WatiN automation.

The full set of HTML controls supported by WatiN is not required for ItiN, so only: buttons, check boxes, radio buttons, textboxes and select lists are supported.

The ItiN framework I is open source on Codeplex here http://www.codeplex.com/itin.

A sample script using the framework that works against the 2003 sample IssueTrackerSimple form saved in c:\ is as follows.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using ItiN;
namespace Itin.Tests
{
[TestMethod()]
public void IssueTrackingSample()
{
// Open test form
string infopathFileName = @"c:\IssueTrackingSample.xml";
InfopathTester FormTester = new InfopathTester(infopathFileName);
FormTester.SetInfopathNamespace("xmlns:iss='http://schemas.microsoft.com/office/infopath/2003/sample/IssueTracking'");
FormTester.SetFormValue(@"//iss:title", "Issue Title");

// Click the send email button which does nothing as I don’t have email configured for infopath
FormTester.Button(ItiN.Find.ByValue("Send as E-mail")).ClickNoWait();
FormTester.SaveDocumentAs(@"c:\SavedForm.xml");
FormTester.CloseAndQuit();

// TODO: Implement code to verify target
Assert.Inconclusive("TODO: Implement code to verify target");
}
}
}

In a previous post, I talked in a closing comment about how some tool vendors are trying to piggyback on WatiN to promote their inferior tools. Well there have been a couple of comments on my blog since from SWEA evangelist, Alex at a Hotmail address.

SWEA is much better product than WATIN. It can save days on test development and maintenance. Any small UI change will require change in the WATIN scripts, but SWEA scripts are more resilient to the UI changes – the control binding is separated from the script code. WATIN is just thin wrapper around HTML DOM, but SWEA is complex automation framework.

You can also try SWExplorerAutomation (SWEA) from webiussoft. SWEA is not free but provides more functionality (Recorder, Data Extractors, internal support for AJAX, more resilient to UI changes scripts) than WATIN.

Ok first things first. This is my blog and not a public forum. If you want to advertise on my blog, you can pay for your adds via Google. It is for this reason that I have deleted these comments.

Let’s debunk these claims one at at time.

Claim #1 SWEA is much better product than WATIN.

Dictionary.com defines a product as “the totality of goods or services that a company makes available.” WatiN is free and not produced by a company. So I will concede that point SWEA is a better “product” than WatiN, simply because WatiN is not a product at all.

Claim #2 SWEA scripts are more resilient to the UI changes … any small UI change will require change in the WATIN scripts

Now that is complete BULLSHIT. Good script design and development practices have significantly more impact on how resilient a script is to UI changes than the underlying tool does. WatiN supports controls defined as .net regular expressions. Using WatiN Attributes in a control class, allows controls to be defined in a single location and called consistently regardless of how WatiN finds the control. Couple that with Visual Studio 2005’s refactoring support and I am working in UI change nirvana with WatiN. If anyone needs to know how good script design should be done, please feel free to follow my blog post series and learn how to do it properly.

Claim #3 SWEA is not free …

That one is true.

Claim #4 SWEA is not free but provides more functionality (Recorder, Data Extractors, internal support for AJAX …

You can get WatiNRecorder from here, so that is an outright lie.

I honestly don’t know what a “Data Extractor” is, nor why I would need one.

WatiN does support AJAX, and yes there are some issues, however they are well know and being improved every day, so that is another lie as well.

Claim #5 WATIN is just thin wrapper around HTML DOM, but SWEA is complex automation framework.

WatiN is a thin wrapper around Internet Explorer, not just the HTML DOM. One of the key reasons WatiN and Watir do so well are their elegance, simplicity and speed. I personally would not be quoting complexity as a feature or any application.

Honestly, SWEA may be an OK product, however if it was so good, webiussoft wouldn’t need to leave comments on every WatiN related blog post they can find, pleading for people to try their products instead.

I’ll leave the last word on this to a comment one one of my posts from Petey in regard to test tool companies that resort to blog spam marketing : “Don’t forget SWEA. It is so annoying!”

If that isn’t a great reason to not try a tool, I don’t know what is.

This post is the third in a series of posts, building a GUI test automation framework. In the first post, I outlined the key design goals of an automation framework. In the second second post I talked to the need for a control file. Since the original posts way back in early 2006, a lot has changed. As a result I am going to re-focus (and finish) these posts on how you can structure and develop a set of simple tests using WatiN.

Ok there are some pre-requites if you want to run these examples. I am going to assume that you are running Visual Studio 2005, SQL Express 2005, nUnit 2.4.0 and we are using the asp.net job site starter kit as our test application. I was planning to use the .net petshop 4.0 for this article, however it wouldn’t install with Windows Vista.

Ok assuming you are up and running, let’s get started.

The general structure that I like to follow for this type of project is to have a single solution broken up in to several projects. The standard projects that I like to have are Controls, Framework and Tests. In this example I am also going to add a fourth project to hold the site URLs.

Once I have these projects in place I set up the classes so they are all using single namespace that is then subdivided in a similar fashion such as: JobSite.Controls, JobSite.Tests and JobSite.URLs.

You can download the solution with the basic structure from here. I have set up a Codeplex project http://www.codeplex.com/WatinJobsiteSample/ to store the source code for the examples in the rest of this series of blog posts. In the next post we will get seriously back on track and start putting our test framework together.