When Microsoft announced that Whidbey would include unit testing, a lot of people thought that it would mean the death of nUnit as we know it. Well one year later, it seems that rumours of the death of nUnit at the hands of VSTS are very much exaggerated. In-fact I don’t know of a single developer that is using the Microsoft.VisualStudio.TestTools.UnitTesting framework. In this post I offer to the Visual Studio team, some of my thoughts why, in my experience, their stuff is not being adopted with hope that they will correct the situation.
1. You should be able to run nUnit tests in Visual Studio 2005.

Before you run out and say, but you can run unit tests in Whidbey, I am referring to tests that have been written using the nunit framework, using the Test View and Test Manager. You should be able to open a project with nUnit tests and boom, they appear as a test asset like any other in the Test view window. I can’t think of any good reason, commercial or other why you can’t do this, especially as nUnit uses the zlib/libpng license.

2. Unit testing should be in all versions of Visual Studio.

Unit testing is such a fundamental part of modern software testing that it should be a ubiquitous feature. Users of VS professional are not going to upgrade to the VSTS developer SKU for this feature when nunit is available instead. That’s right Mr. architect, no testing for you either.

3. Microsoft’s stuff is better, but no one knows it.

It supports data driven tests, it has implementaiton-to-test and test-to-method code generators. It creates accessors for private methods, but most developers don’t use this stuff. Hardcore TDD guys use resharper, and everyone just creates their test classes by hand.

4. You can’t run tests without a full VSTS license.

You can use MSTest.exe to run tests without Visual Studio, but you can’t escape the fact that to get MSTest you need VSTS. You should be able to call tests programatically without needing the full version of Visual Studio installed.

Essentially, these tools show that no matter how good the tools, if you don’t market them correctly they simply won’t be adopted.

I came a cross a post by the C# Test Manager Rusty Miller where he describes some of the tools that were used to test C# for the Visual Studio 2005 release.

Developing your own, specialised test tools and frameworks is one of the best parts of my job and I am always looking for directions we can to push the envelope, and innovate to make our job easier. It is also a big reason why I think that document heavy approaches like ISO testing standards are have little or no place in the modern software development and testing arena.

Although I don’t yet have time to start using WATIN full time at work, I managed to convert a few more tests over to compare how the tests work. I am doing this in my spare time (i.e lunch and after work), to building A proof of concept and see how the framework will go in real world testing situations. For this post I have removed all references to our real application and mocked up a sample application.

The part of our app that I have simulated has a model dialog, a group of drop down’s in that each refresh when the one above it changes, and an image button implemented on an ‘input’ tag to launch the dialog. The sample dialog looks like this mock-up.

The test case was a very simple one, click the image, select a value in each of the drop-down’s, click OK then validate the page contains some text. There were a few interesting challenges, but nothing that I couldn’t solve by adding some code to the Watin framework and recompiling. Here is an altered version of the resultant code.

 [TestMethod]
 public void AddNewUser()
 {
    using (IE ie = new IE(baseURL))
    {
        // Click on the 'Manage customers' button to open the customer
        // management screen
        ie.MainDocument.Button(Find.ByName(btnManageCustomers)).Click();

        // Click on the add customer image button (an INPUT tag with type image)
        // We need to use ClickNoWait as the Create customer dialog is a modal dialog
        // and the readystate does not return to complete until the dialog is closed.
        ie.MainDocument.Input(Find.ById(btnSelectTicketType_id)).ClickNoWait();

        // Find the new 'Create new customer' dialog that opens in the collection
        // of dialogs and wait a maximum of 2 seconds for the dialog
        HtmlDialog dialog = ie.HtmlDialog(Find.ByTitle("Select ticket type"),2);

        // Select Melbourne, Qantas and Business class
        // We need the thread.sleep as these contols load dynamically based on
        // the previous selection and we are in dialog mode
        dialog.MainDocument.SelectList(Find.ById(ddlDestination_Id)).Select("Melbourne");
        Thread.Sleep(2000);
        dialog.MainDocument.SelectList(Find.ById(ddlCustomerAirline_Id)).Select("Qantas") ;
        Thread.Sleep(2000);
        dialog.MainDocument.SelectList(Find.ById(ddlCustomerClass_Id)).Select("Business");
        Thread.Sleep(2000);

        dialog.MainDocument.Button(Find.ById(btnOk_Id)).ClickNoWait();
        ie.WaitForComplete();

        // Test that the text 'Melboune' is in the page
        Assert.IsTrue(ie.MainDocument.ContainsText("Melbourne"));

        // Test that the text 'Qantas' is in the page
        Assert.IsTrue(ie.MainDocument.ContainsText("Qantas"));

        // Test that the text 'Business class' is in the page
        Assert.IsTrue(ie.MainDocument.ContainsText("Business class"));
    }

So what were the traps?

Internet Explorer implements a ReadyState property which indicates that
the browser is loading, rendering the page, etc. ReadyState typically returns an enum value of READYSTATE_COMPLETE when the browser is loaded and ready for more input. Where the challenge comes in is that his behaviour changes when when a modal dialogue is shown The browser does not return READYSTATE_COMPLETE, yet it is waiting for user input. To work around this issue Watin implements a ClickNoWait() method to use instead of the Click() method under these circumstances.

The problem I ran into (apart from an un-handled exception being thrown from the click method) was that the applicaiton dynamically loads the value of one dropdown after a value is selected in the one above it. This required a Thread.Sleep call to pause the script for 2 seconds, after each drop-down list is selected. I don’t think that this solution is very elegant, and it adds some 6 additional seconds to the execution time for this test. This definitely warrants some further investigation into what the IE ReadyState is when the dialog is shown, writing another wait method for the Watin ie object, that waits for that state to change.

Anyway so far Watin is meeting my expectations and still looks like it will be able to handle just about everything I can throw at it.

If you are developing websites or web automation the Microsoft IE Developer Toolbar is a great tool. It is ideal for identifying controls in a HTML page when rendered at runtime which is great for debugging layout issues.

If you are looking (as I have been for a while) for a solution on how to write automated web GUI tests for internet explorer in .net then look no further. WATIR has been ported to C# as the WATIN framework. I have compiled the framework in .net 2.0 with minimal fuss, and it feels like it should have shipped out of the box with Visual Studio Team Edition for Software Testers.

So is it any good? I have spent about 2 hours so far converting one of my simple tests across, and this thing rocks. The tests are simple and execute very quickly. The following sample, copied from the WATIN homepage, searches for this site in google and checks for the text “Saving the world, one test case at a time” in the browser window.

    [Test]
    public void Google()
    {
      using (IE ie = new IE("http://www.google.com"))
      {
        ie.MainDocument.TextField(Find.ByName("q")).TypeText("Teknologika");
        ie.MainDocument.Button(Find.ByName("btnG")).Click();

        Assert.IsTrue(ie.MainDocument.ContainsText("test case"));
      }
    }

The best part of all, is that if you use NUnit and Visual C# Express Edition, your total cost is $0.

Google have been publishing some of their internal tech talks for a while now, and I recently came across this little gem; “Predicting bugs in code changes using SCM information “. The concept proposed by this video is that you can use the combination of your source control check-in logs and defect tracking history to identify where the areas of lowest quality code are and focus your testing efforts on them.

In Code Complete, Steve McConnell espouses some statistics that: “80% of errors are found in 20% of a project’s code.”, and “50% of errors are found in 10% of a project’s code.”.

This leads me to a logical conclusion that with a little bit of thought and planning with VSTS, you could identify the 20% of code that has the most defects logged against it and focus your testing efforts there. However like everything, once you start measuring you begin squeezing the balloon, and effecting the system that you are measuring.