WatiN based unit tests using the ASP.NET Development web server

One of the more interesting and less documented features of VSTS is how to handle the ever changing port of your local development server when running unit tests. This really becomes an issue when you want to write WatiN tests that can run out of the box on any developers machine or a server that is performing continuous integration.

As part of this week’s Teched, Mitch Denny has organized a Smith Family community project to develop an application for the smith family. My contribution was to port WatiN to .net 2.0 (again), so it didn’t have my recent enhancements.

The next step was to write a test that used WatiN, but would work with the TFS continuous integration tool Chris Burrows, without a hardcoded test server. Whilst not the exact code, Chris showed me how to utilise the AspNetDevelopmentServer attribute from the Microsoft.VisualStudio.TestTools.UnitTesting.Web namespace to work with the local ASP.NET Development web server. I have written up a short example below that shows how it is done.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.UnitTesting.Web;
using WatiN.Core;

[TestClass] public class UnitTest1 { private TestContext m_testContext; public TestContext TestContext { get { return m_testContext; } set { m_testContext = value; } }

public string DevServerURL
{
    get
    {
        Uri webServer = m_testContext.Properties["AspNetDevelopmentServer.localhost"] as Uri;
        return webServer.AbsoluteUri;
    }
}

[TestMethod]
[AspNetDevelopmentServer("localhost", @"C:\Docu ... \WebSite1")]
public void TestMethod1()
{
    using (IE ie = new IE(DevServerURL))
    {
        Assert.IsTrue(ie.MainDocument.ContainsText("This is a test."));
    }
}

}
To make this work you will need to add a COM reference to Microsoft Internet Controls, and a reference to your Watin.Core.dll. To our complete surprise, once we had defined the initial test as part of the team build, it ran first go on the server, sight unseen and passed.

Adding Authenticaton to WatiN

Well after a week or so of not touching WatiN, I had another crack at getting authentication to work. In the short term I have abandoned the “technically correct” solution of trying to get impersonation to work, and instead I have ported over the AutoIT code from watir. To do this I created two new classes AutoIT and AutoItLogonParamaters. The main methods in the AutoIT class are:

public void logon(AutoITLogonParameters loginParamaters)
{
     autoit.WinWait(loginParamaters.windowTitle, "", _timeout);
     autoit.WinActivate(loginParamaters.windowTitle, "");
     autoit.Send(loginParamaters.userName, 0);
     autoit.Send("{TAB}", 0);
     autoit.Send(loginParamaters.password, 0);
     autoit.Send("{ENTER}", 0);
}
The logon parameters class just contains properties to get and set three strings so I can pass in one parameter instead of three. The next trick was to work out how to get this code to work on another thread. This required “drilling a hole” through the IE class, passing the login paramaters class into the popupwatcher class and adding the following to the MyEnumThreadWindowsProc method.
private bool MyEnumThreadWindowsProc(IntPtr hwnd, IntPtr lParam)
{
   if (IsDialog(hwnd))
   {
IntPtr handleToDialogText = NativeMethods.GetDlgItem(hwnd, 0xFFFF); string alertMessage = GetText(handleToDialogText); alertQueue.Enqueue(alertMessage);

    // get the window title, and check if it matches our login window
    if (GetText(hwnd).StartsWith("Connect to"))
    {
        if (logonParameters != null)
        {
            //  TODO: We should be able to use win32 for all this like
            //  NativeMethods.SetWindowText(hwnd, "Hello");
            WatiN.Core.AutoIT auto = new AutoIT();
            auto.logon(logonParameters);
        }
        else
        {
            throw new MissingFieldException("A window title is missing.");
        }
    }
    NativeMethods.SendMessage(hwnd, NativeMethods.WM_CLOSE, 0, 0);

} return true; } This works exactly as I expected and the following test passed with no problem.

IE ie = new IE(new AutoITLogonParameters("Connect to localhost", "uid", "pwd"));
ie.GoTo("http://localhost");
Assert.IsTrue(ie.ContainsText("this is a test"), "Text not found");
ie.Close();
The main problem with this approach is that it requires the AutoIT dll to be bundled with the code, and it needs to be registered on the machine before it can be used. My next step is to investigate the Win32 calls similar to what already exists to close the popup windows, removing the need for AutoIT entirely.

Refining the code – updates to my previous WatiN example

In my previous post I mentioned that further investigation was required to refine the script to remove the thread.sleep statements. Well post investigation I can share some insights on exactly what is happening and what changes are required.

In addition to the ReadyState property, Internet Explorer exposes a Busy property as well. When a normal page has completed loading the ReadyState will return READYSTATE_COMPLETE and Busy will return false. Internally WatiN uses this to let the browser to dictate when the test script should continue through the Wie.aitForComplete method. This method internally calls two private methods WaitWhileIEBusy() and WaitWhileIEStateNotComplete(). These methods wait for their corresponding methods to return the expected value, then the script continues.

The challenge comes in when a dialog is displayed. When a modal dialog is displayed, IE set’s ie.busy = true, and returns READYSTATE_COMPLETE. If a test is calling WaitForComplete, then the script will hang forever. WatiN caters for this with the WaitForComplete method on it’s HTMLDialog class, which ignores the busy property. Armed with this information, our script now looks like this.

 [TestMethod]
 public void AddNewUser()
 {
    using (IE ie = new IE(baseURL))
    {
        // Click on the 'Manage customers' button to open the customer 
        // management screen 
        ie.MainDocument.Button(Find.ByName(btnManageCustomers)).Click();
// Click on the add customer image button (an INPUT tag with type image) // We need to use ClickNoWait as the Create customer dialog is a modal dialog // and the readystate does not return to complete until the dialog is closed. ie.MainDocument.Input(Find.ById(btnSelectTicketType_id)).ClickNoWait();
// Find the new 'Create new customer' dialog that opens in the collection // of dialogs and wait a maximum of 2 seconds for the dialog HtmlDialog dialog = ie.HtmlDialog(Find.ByTitle("Select ticket type"),2);
// Select Melbourne, Qantas and Business class // We need the dialog.WaitForComplete() as these contols load dynamically based on // the previous selection and we are in dialog mode dialog.MainDocument.SelectList(Find.ById(ddlDestination_Id)).Select("Melbourne"); dialog.WaitForComplete(); dialog.MainDocument.SelectList(Find.ById(ddlCustomerAirline_Id)).Select("Qantas") ; dialog.WaitForComplete(); dialog.MainDocument.SelectList(Find.ById(ddlCustomerClass_Id)).Select("Business"); dialog.WaitForComplete(); dialog.MainDocument.Button(Find.ById(btnOk_Id)).ClickNoWait(); ie.WaitForComplete();
// Test that the text 'Melboune' is in the page Assert.IsTrue(ie.MainDocument.ContainsText("Melbourne"));

// Test that the text 'Qantas' is in the page Assert.IsTrue(ie.MainDocument.ContainsText("Qantas"));
// Test that the text 'Business class' is in the page Assert.IsTrue(ie.MainDocument.ContainsText("Business class")); }

When to Automate Testing?

It seems that when to automate and how much automated testing you should have is one of the more popular blog posting topics these days. David Weiss has just put up an excellent post When to Automate Testing?, for his take on the subject. This adds to the list of posts that already exist, including my own, shown in chronological order.

Brian Marick – When Should a Test be Automated? (pdf)

Sara Ford – Automation Testing versus Manual Testing Guidelines

Bruce McLeod – If you are going to run a test more than once, it should be automated.

Keith Stobie – Reality Check: % of test cases automated

Bj Rollison – For those of you dreaming the 100% automation dream…please wake up!

David Weiss – When to Automate Testing?

Steve Rowe – Too Much Test Automation?

If there is another post that I have missed or should be added to the list, just leave a comment on this post and I’ll add it.

Tenet: If you are going to run a test more than once, it should be automated.

This post is the second in a seven part series covering my seven tenets of software testing.

Original post (31 Jan 2005)

Automation explained Test automation means different things to different people in different contexts. When I talk about automated testing I am referring to; A method of executing a test without human intervention, that would otherwise require it. In practical terms that may mean a nUnit test, a GUI test using a commercial testing tool, or a test written using an application’s internal scripting language. The technology is not the key concern, the fact that the test can be run 100% without any human involvement is the key.

Why automate? The primary reason to automate tests is time. As a tester, you always need more time.

Automation can provide immense reductions in the amount of time required to execute tests. My first introduction to automated testing, in 1997, managed to condense 5 days of manual testing effort into 1 hour of automated execution for a 97.5% reduction in execution time.

Think about that for a moment, by automating our tests, we have achieved the equivalent of adding 40 additional testers to the team, for a fraction of the price. In addition I had taken the drudgery from my teams work day, increasing morale. More importantly, it lets your testers perform more ad-hoc testing which is much more effective than performing the same manual test over and over again.

So that’s it then, should we just retrench all our testers, and use automation instead? Well, no.

Despite a recent example where Microsoft retrenched 62 Longhorn testers, citing automation as the cause. Test automaton is generally used to help increase the amount of test coverage that can be achieved with a given schedule and resources, not reduce testing head-count.

When implemented correctly, with enough hardware, automating your tests allows you to execute all your tests, at least once a day, every day. When combined with a daily build, you have one (if not the), most powerful testing tool on the planet.

Build Verification or smoke testing Every night after a successful compile, the daily build should automatically be packaged and deployed into a test environment, and smoke tested with an automated build verification test (BVT). The term “smoke” test is derived from the idea of quickly plugging in some electronics to test them, checking that no smoke comes out. While ideally you would run all your tests, typically the smoke test is a carefully selected subset of your full automated tests that can be run in about 10-20 minutes or so.

Automated regression testing regression n. “[To] relapse to a less perfect or developed state.”

Regression testing has the goal of ensuring that the quality of an application doesn’t decline as features are added. Regression testing has a significant challenge. When an application needs it most, there is less time and resources available to perform all the tests that were executed when the product first shipped.

Without automation, the typical approach is to perform localised regression testing, which is limited to directly testing the area around the changes.

With an automation suite in hand, it is simply a matter of executing all the tests that were developed previously. This allows the maintenance programmers to make frequent releases, and allows the quality of the application to improve over time.

This is particularly important in light of trend that Fred Brooks suggests in his classic work, the mythical man-month, where each defect that is fixed has a 20-50% chance of introducing another.

The largest automation project that I have personally worked on was a huge effort which my client ran their $1M+ investment of tests on a rack of 25 dedicated machines that pounded away relentlessly, shortening their regression testing cycle by 75%, and that was partially automated to begin with!

Update (Tue, 8 Feb 2005) Sara Ford, a tester in the Visual Studio team at Microsoft has blogged about this topic as well here.

Update (Wed, 2 Aug 2006) A recent post by Bj Rollison, For those of you dreaming the 100% automation dream…please wake up! makes a very good point that, an goal 100% automated tests is completely unrealistic. I agree with his post, feel that I need to add a clarification to this one.

With any form of testing, you have to focus on what is important. Blindly trying to automate everything just to reach an arbitrary automation goal is contrary to that tenet. Does that mean that this tenet is wrong then? No, I don’t think so. There is a significant difference between should be automated and must be automated. In my experience, most projects need a heck of a lot more automation than they have. If this tenet was called, “you must automate what you can”, the whole point of this tenet would have been lost. Personally, the highest I have ever achieved on a project was 95% automated, and that lasted for all of 1 day, before we added more tests.

The #1 priority for a tester on any project, should always be to finding and log issues, however you find them. However, investing in the right amount of automation can make that a whole lot more effective.