Assert me!

Following on from my previous post referring to Joe Schmetzer’s work on unit testing anti-patterns, I felt the need to share my observations from some recent code reviews that I have performed.

Always include at least one assert in each test

If I hadn’t seen a unit test with no asserts myself, I would find it hard to believe. Each nUnit test must contain at least one assert statement. Without the assert, you will get a test pass regardless of what the code does. For example the following test:

[Test]
public void TestName()
{
    Console.WriteLine("\nThis is a test with no asserts.");
}
Will produce the following output in nunit-console.
.
This is a test with no asserts.

Tests run: 1, Failures: 0, Not run: 0, Time: 2.541 seconds. As you can see this is not marked as a failure or not run, it is recorded as a pass. This is a common error that all automated test tool developers tend to make. The default result for any automated test should be to fail (or some other non-passing state), unless explicitly passed during test execution.

Microsoft have the right idea for their generated test stubs in Whidbey. In VS 2005 it places an Assert.Inconclusive statement in the body of any test it auto-generates. That is a good first step, but ideally it should be the default behaviour for any test with no asserts, and not require that the Assert.Inconclusive statement to even be there.

Back to the code review, to be fair to the developer involved, the test code was testing an asset object which is only one letter different to an assert statement, so it was pretty easy to miss.

Where multiple asserts are used in a test, include a descriptive string.

If your test has more than one assert statement, each statement needs to have a nice, descriptive string explaining what is being validated. The reason for this is if you have a test with a bunch of asserts that fails, you will have to debug through the test code to find out which assert failed. For example the following test:

[Test]
public void TestIntegers()
{
    int FirstInteger = 1;
    int SecondInteger = 1;
    Assert.AreEqual(FirstInteger, 1);
    Assert.AreEqual(SecondInteger, 2 );
}
will produce the following output in nunit-console.
.F
Tests run: 1, Failures: 1, Not run: 0, Time: 2.606 seconds

Failures: 1) Teknologika.Tests.BlogPostTests.TestIntegers : expected:<1> but was:<2> The challenge here of course is which test failed? In the contrived example above, it is obviously the second one. If it was an actual test, however, we would most likely have to debug through the test code to find out. If we add some comments to the test, as follows:

[Test]
public void TestIntegers()
{
    int FirstInteger = 2;
    int SecondInteger = 1;
    Assert.AreEqual(FirstInteger, 1, "Failed : Verifying that FirstInteger is 1." );
    Assert.AreEqual(SecondInteger, 2, "Failed : Verifying that SecondInteger is 2." );
}
Then we get a much more meaningful output, which let’s us focus straight on the assert that is failing.
.F
Tests run: 1, Failures: 1, Not run: 0, Time: 2.791 seconds

Failures: 1) Teknologika.Bulldozer.Tests.BlogPostTests.TestIntegers : Failed : Verifying that FirstInteger is 2. expected:<1> but was:<2> Refactor your tests into smaller more atomic tests

The previous example is hiding something. There is more that one failure in the TestIntegers test, but we are only seeing the first one, as nUnit aborts the test once the first failure is reached.

If this was a real test, and the each of the failures were being caused by different things we would probably fix the first one, and re-run the tests only to have the test fail on the second error.

Incidentally, for my contrived example, where both our asserts have the same expected and actual values, if we hadn’t added a description, our test results would be identical, so you might still think that it was broken. Even though we have fixed the first error.

So if we refactor our test into two smaller atomic tests like this:

[Test]
public void TestFirstInteger()
{
    int FirstInteger = 2;
    Assert.AreEqual(FirstInteger, 1, "Failed : Verifying that FirstInteger is 1." );
}

[Test] public void TestingSecondInteger() { int SecondInteger = 1; Assert.AreEqual(SecondInteger, 2, "Failed : Verifying that SecondInteger is 2." ); } then our results show the full story.
FF
Tests run: 2, Failures: 2, Not run: 0, Time: 3.209 seconds

Failures: 1) Teknologika.Bulldozer.Tests.BlogPostTests.TestFirstInteger : Failed : Verifying that FirstInteger is 1. expected:<1> but was:<2>

2) Teknologika.Bulldozer.Tests.BlogPostTests.TestingSecondInteger : Failed : Verifying that SecondInteger is 2. expected:<2> but was:<1>

The bottom line here is that spending a small amount of time when writing your tests can make things a whole lot easier when it comes to investigating your test failures.

Testing Testing Tenets0 comments