Twelve moths ago, before my last project started I went on a journey. The intended destination was test automation nirvana. The way to get there, or so I thought was to combine WatiN with The Braidy Tester’s Automation Stack. The end result, I hoped would be the Deep Thought of test automation. It didn’t quite work out that way, and I would like to share some o the challenges we faced.

The Braidy Tester’s automation stack

I am not going to describe the architecture in detail of the Braidy Tester’s automation stack, as Michael Hunter has already covered that in great detail. What IS important however is the pieces that we chose. The key aspects of the stack that we decided implement were:

• Logical Functional Model The logical functional model is a user centric view of how the application is used. For example: Logical.Google.Search("Bruce McLeod");
• Physical Object Model The physical object model is an abstraction layer that hides the implementation and automation semantics details from the tests. For Example:Physical.Google.btnSearch.Invoke();
• Execution Behaviour Manager Execution behaviours are a way of loosely coupling execution actions. They allow executions to be composed, as well as executed in different sequences.
• Loosely Coupled Verification Instead of littering your tests with assert statements, loosely coupled verification takes a snapshot of the application, calculates an expected state based on your execution behaviour and then compares it to the actual state after the action is performed.
• Test Data Providers Test Data providers remove the need for test data to be hard coded into the test by moving it out, into a database in our case.

Things that worked brilliantly in the automation stack

The separation into the logical and physical layers worked exceptionally well. The test cases become very clean and concise. We added another lower level layer called the controller in place of the Application Internals model.

The controller essentially was an abstraction later over WatiN and our control definitions. This makes the entire stack easily portable from one automation technology to another, so the transition will be very easy when the new automation tools ship in the next version of Visual Studio Team Test, if we decide to change.

Implementing Execution Behaviours with Delegates

The idea behind Execution Behaviours is that you use attributes on a method that matches the delegate signature to perform one of the following actions:

• Execute all the specified actions in the listed sequence. ExecuteInSequence
• Execute one of the actions "randomly". ChooseAny
• Execute all the actions in the listed sequence until some condition is met. ExecuteUntil

Execute in sequence is straight forward and is used to compose actions.

[CompositeExecutionBehavior("OpenSearchPage", "WebSearch")]

public static void Search(TestDataProvider dataProvider)

{

   TestActionDelegate myDelegate = TestExecutionManager.ExecuteInSequence(Logical.Google. Search(TestDataProvider dataProvider);

}

With delegates being used and the possibility of execution sequence order changing, it becomes necessary to provide inherit record and replay for actions. We chose to use a version of the command pattern and use .net serialization to achieve this. We also had to package up the parameters into an array of objects to the could be serialized. The implementaiton of one of our actions in the physical layer would look something like this:

public static void OpenSearchPage(TestDataProvider dataProvider)

{

   // Add this action to the command list and serialize it

   object[] parameters = { dataProvider };

   ExecutionManager.Add(parameters);

   // Open the search page

   SearchModel searcher = new SearchModel();

   searcher.OpenSearchPage(dataProvider);

}

This then calls the physical layer that actually does the work. All this loosely coupled goodness is great, except that we only ever used the one type of execution behaviour, execute in sequence. As a result our code could have actually looked like this instead if it was hard coded in the logical model.

public static void Search(TestDataProvider dataProvider)

{

   SearchModel searcher = new SearchModel();

   searcher.OpenSearchPage();

   searcher.Search(dataProvider);

}

That code is much simpler and easier to maintain, and removes the method signature matching constraints that delegates impose. For us it would have been a much better way to go, simply because we would have removed the constraints required by functionality that we didn’t actually implement.

In the next post, I’ll run through some of the benefits and challenges we had implementing loosely coupled verification.

In my last post I gave a glimpse into our new WatiN stack, another feature is that we have implemented the command pattern, and serialise the commands out as XML. This lets us build tests that have randomised execution and still replay them when they fail. As a nice by-product, we also can write tests as XML if we wish.

After a month of pushing my gray matter as far as it will go (which isn’t all that far compared to some of the people I know) I have implemented the next generation of automated testing on my project.

No it isn’t using excel to generate tests like some people claim, although that does have it’s place. I have implemented a stack based on the design used in the expression team as described by Michael Hunter aka “The Braidy Tester”.

This transforms a test case that was like this:

using (IE ie = new IE("http://www.google.com"))
{
    ie.TextField(Find.ByName("q")).TypeText("WatiN");
    ie.Button(Find.ByName("btnG")).Click();
    Assert.IsTrue(ie.ContainsText("WatiN"));
}

to

Logical.Search(DataProvider.GetProvider("SearchForWatin"));

The next layer down is:

public static void Search(DataProvider dataProvider)
{
    object[] parameters = {dataProvider};
    executioner.Add(parameters);
    SearchModel searchModel = new searchModel();
    searchModel.OpenSearchPage();
    VerificationManager.ActionStarting(parameters);
    searchModel.Search(dataProvider);
    VerificationManager.ActionEnding(parameters);
}

then onto the “guts” of the test in the physical model.

public void Search(DataProvider dataProvider)
{
    SearchTDO search = DataProvider.GetSearchTDO();
    controller.SetValue(Controls.Search.Google.txtSearch,
         search.SearchText);
    controller.Invoke(Controls.Search.Google.btnSearch);
}

In case you didn’t notice, WatiN is fully abstracted away out of the tests, as is the test data, verification and the intent of how things are done. This increases the complexity significantly, but it also allows us to handle changes VERY easily.