Infographics are all the rage these days, so I decided to knock up a quick graphic for a post that I did ages ago about the 5 stages of the developer tester relationship.
I have also included a 1920 X 1024 PDF if you want to print it out for your wall.
Testing in Australia has a problem. The Australian IT industry still views testing as a job that anyone can perform with minimal technical skills. A quick search on a leading recruitment site has a number of advertisements for Testers with requirements similar to the following:
- At least 3 years experience in a Tester or Test Analyst role with demonstrated experience in planning and executing functional, systems and regression tests.
- Excellent written and verbal communications skills will be a must for this role – the right candidates must be able to develop clear and concise test cases and test scripts
- Some previous experience using Test Director or Quality Center – any exposure to automated test tools (particularly QTP) would be a benefit
- Experience of various software testing approaches and SQL in oder to extract data for tests.
- A methodical nature and high levels of attention to detail.
(Extra points to those of you who noticed the spelling error in the advertisement = “oder” instead of “order”.)
This ad is typical of those here in Australia for testing positions. No programming skills required, no industry specific knowledge, just X number of years in testing, good communication skills and some experience with a particular test tool vendor’s product.
I’m a big fan of analogies so let’s imagine for a second that a car company uses non technical testers to evaluate new cars under development. I can imagine a dialogue between a test driver and an engineer going something like this:
“The new car feels a bit funny at the front”
“What do you mean by funny?”
“Well I turn the round thinngy in the there …”
“oh you mean the steering wheel”
“yeah, the steering feel. I turn the steering feel to this way, but the car doesn’t turn as much as I thought it would.”
“It’s a wheel, not a feel.”
“oh sorry, wheel”
“Anyway, so let me see if I understand you. When you turn the wheel, and the front of the car doesn’t turn as much as you expect it to.”
“Exactly”
By comparison, a test driver that understands how a car works can have a much more meaningful conversation with the engineer:
“It has mid-corner understeer, after turn in.”
“Ok we can solve that by increasing camber slightly or adjusting the rear springs.”
“Let’s try the spring adjustment, as changing camber will also effect tire wear.”
The technical tester is much more effective than the non technical equivalent as they have similar, equivalent or superior knowledge to the engineer but simply a different focus and specialist skills. Whilst my example is fictional, the exact same difference in conversation should be expected between a “Tester” and a SDE/T. What is an SDE/T? An SDE/T is one of the common, technical testing roles at Microsoft. Their testing careers page has the following description:
Software Design Engineer in Test
Tests and critiques software components and interfaces in more technical depth, writes test programs to assure quality, and develops test tools to increase effectiveness.
At Devtest, this is what we do as well using, the same tools, knowledge and experience, as all the other “developers” on the project. The key difference, however is we have a different focus, live in different .Net namespaces (lately System.windows.Automation) and have different goals. However we aren’t any less capable at writing code that the “developers” on the project.
What we constantly have to battle is a prejudice that if you have test somewhere in your job title we are somehow lesser beings, and are not valued or required on most projects. We are seen by people starting out in the industry as un-cool and very few people would consider testing as a career, this is just crazy, we use C#, visual studio, TFS and other cool tools, just like everyone else, we just have a different focus. The “tester is dead” long live the SDE/T.
Bj Rollison has blogged about the percieved value of code coverage and the information, percieved or otherwise that it provides. This post is definatley worth a read and has me thinking about an example that I commonly use drawn from Code Complete when takking about the subject, but that is a topic for future blog post.
The project that I am currently working on uses infopath forms. This was causing us a few hassles on the testing front, especially now we are using WatiN extensively. So after a few failed attempts at using the new .Net 3.0 automaton API’s and not really wanting to learn MSAA, I decided to create a specalised port of WatiN to test infopath and ItiN was born.
ItiN uses a hybrid approach allowing manipulation of an InfoPath document directly using XPath and the XML dom, or the view using traditional WatiN automation.
The full set of HTML controls supported by WatiN is not required for ItiN, so only: buttons, check boxes, radio buttons, textboxes and select lists are supported.
The ItiN framework I is open source on Codeplex here http://www.codeplex.com/itin.
A sample script using the framework that works against the 2003 sample IssueTrackerSimple form saved in c:\ is as follows.
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using ItiN;
namespace Itin.Tests
{
[TestMethod()]
public void IssueTrackingSample()
{
// Open test form
string infopathFileName = @"c:\IssueTrackingSample.xml";
InfopathTester FormTester = new InfopathTester(infopathFileName);
FormTester.SetInfopathNamespace("xmlns:iss='http://schemas.microsoft.com/office/infopath/2003/sample/IssueTracking'");
FormTester.SetFormValue(@"//iss:title", "Issue Title");
// Click the send email button which does nothing as I don’t have email configured for infopath
FormTester.Button(ItiN.Find.ByValue("Send as E-mail")).ClickNoWait();
FormTester.SaveDocumentAs(@"c:\SavedForm.xml");
FormTester.CloseAndQuit();
// TODO: Implement code to verify target
Assert.Inconclusive("TODO: Implement code to verify target");
}
}
}
The Continuous Integration and Testing Conference is coming to Sydney. Joe Schmetzer highly recommended the UK incarnation of the conference he attended, so I will definatley be looking to attend this one.
The Open Information Foundation, co-founded by Jeffrey Fredrick and Paul Julius, presents CITCON Asia/Pacific 2007 in Sydney, Australia.
CITCON (Continuous Integration and Testing Conference) brings together people from every corner of the software development industry to discuss Continuous Integration and the type of Testing that goes along with it.
- What: OpenSpace event discussing all aspects of CI and Testing, together
- Where: TBD, Sydney, Australia
- When: July 27 & 28, 2007
In a previous post, I talked in a closing comment about how some tool vendors are trying to piggyback on WatiN to promote their inferior tools. Well there have been a couple of comments on my blog since from SWEA evangelist, Alex at a Hotmail address.
SWEA is much better product than WATIN. It can save days on test development and maintenance. Any small UI change will require change in the WATIN scripts, but SWEA scripts are more resilient to the UI changes – the control binding is separated from the script code. WATIN is just thin wrapper around HTML DOM, but SWEA is complex automation framework.
You can also try SWExplorerAutomation (SWEA) from webiussoft. SWEA is not free but provides more functionality (Recorder, Data Extractors, internal support for AJAX, more resilient to UI changes scripts) than WATIN.
Ok first things first. This is my blog and not a public forum. If you want to advertise on my blog, you can pay for your adds via Google. It is for this reason that I have deleted these comments.
Let’s debunk these claims one at at time.
Claim #1 SWEA is much better product than WATIN.
Dictionary.com defines a product as “the totality of goods or services that a company makes available.” WatiN is free and not produced by a company. So I will concede that point SWEA is a better “product” than WatiN, simply because WatiN is not a product at all.
Claim #2 SWEA scripts are more resilient to the UI changes … any small UI change will require change in the WATIN scripts
Now that is complete BULLSHIT. Good script design and development practices have significantly more impact on how resilient a script is to UI changes than the underlying tool does. WatiN supports controls defined as .net regular expressions. Using WatiN Attributes in a control class, allows controls to be defined in a single location and called consistently regardless of how WatiN finds the control. Couple that with Visual Studio 2005’s refactoring support and I am working in UI change nirvana with WatiN. If anyone needs to know how good script design should be done, please feel free to follow my blog post series and learn how to do it properly.
Claim #3 SWEA is not free …
That one is true.
Claim #4 SWEA is not free but provides more functionality (Recorder, Data Extractors, internal support for AJAX …
You can get WatiNRecorder from here, so that is an outright lie.
I honestly don’t know what a “Data Extractor” is, nor why I would need one.
WatiN does support AJAX, and yes there are some issues, however they are well know and being improved every day, so that is another lie as well.
Claim #5 WATIN is just thin wrapper around HTML DOM, but SWEA is complex automation framework.
WatiN is a thin wrapper around Internet Explorer, not just the HTML DOM. One of the key reasons WatiN and Watir do so well are their elegance, simplicity and speed. I personally would not be quoting complexity as a feature or any application.
Honestly, SWEA may be an OK product, however if it was so good, webiussoft wouldn’t need to leave comments on every WatiN related blog post they can find, pleading for people to try their products instead.
I’ll leave the last word on this to a comment one one of my posts from Petey in regard to test tool companies that resort to blog spam marketing : “Don’t forget SWEA. It is so annoying!”
If that isn’t a great reason to not try a tool, I don’t know what is.
This post is the third in a series of posts, building a GUI test automation framework. In the first post, I outlined the key design goals of an automation framework. In the second second post I talked to the need for a control file. Since the original posts way back in early 2006, a lot has changed. As a result I am going to re-focus (and finish) these posts on how you can structure and develop a set of simple tests using WatiN.
Ok there are some pre-requites if you want to run these examples. I am going to assume that you are running Visual Studio 2005, SQL Express 2005, nUnit 2.4.0 and we are using the asp.net job site starter kit as our test application. I was planning to use the .net petshop 4.0 for this article, however it wouldn’t install with Windows Vista.
Ok assuming you are up and running, let’s get started.
The general structure that I like to follow for this type of project is to have a single solution broken up in to several projects. The standard projects that I like to have are Controls, Framework and Tests. In this example I am also going to add a fourth project to hold the site URLs.
Once I have these projects in place I set up the classes so they are all using single namespace that is then subdivided in a similar fashion such as: JobSite.Controls, JobSite.Tests and JobSite.URLs.
You can download the solution with the basic structure from here. I have set up a Codeplex project http://www.codeplex.com/WatinJobsiteSample/ to store the source code for the examples in the rest of this series of blog posts. In the next post we will get seriously back on track and start putting our test framework together.
As you can tell from my blog, I am a big fan of WatiN. One thing that WatiN doesn’t do is test non IE stuff. Well as part of .net 3.0 Microsoft has now shipped API’s that allow testing of Windows apps, including both legacy Win32 and WPF applications. There is an MSDN article on how to do this here.
So what does this mean? It essentially means that commercial tools are dead in the long term. If it is possible for a single developer to build tools that work better than commercially available tools costing hundreds or thousands of dollars, which it currently is. It won’t be long until they are not purchased any longer.
Another interesting phenomenon is that companies are trying to advertise their tools by adding comments to blogs, such as this one saying things like, you should try tool xxxx.
Sorry but when your commercial tool costs more and does less than the open source alternative, it is time to look for a new product to develop.
Another major irony is that if the open source tool has a bug, or doesn’t have a feature I need, then you can simply open up the code and fix it or add the feature. Most commercial vendors take 4-6 months or more to investigate, test and resolve issues.
So if you haven’t yet, download a copy of WatiN, WatiNRecorder, nUnit and C# Express, then see what the future is like and join in by submitting back features.
One final word, Webiussoft and Inscif, please stop trying to promote your commercial tools by adding comments to blogs about WatiN, we aren’t changing !
Over the years we have noticed a phenomenon where developers can become reliant on testing. This addiction is a good thing, as it leads to a much better end product being developed. As a by product we have identified 5 stages that developers go through in their test addiction.
Stage 1 : Denial or “We don’t need testing, we are shipping fine without it thanks.”
Developers in stage one are under the illusion that you can simply compile you code and then ship it. Most developers at this stage are seeking compliance with the works on my machine certification. The particularly dangerous developer at this stage is the one who takes offence when his god-like development skills are revealed as capable if causing bugs just like everybody else.
Stage 2 : Annoyance or “The ‘ Pointy-haired boss’ has hired some outside testing company”
Someone from on-high who the developers think of as a pointy haired boss has forced some testers onto the project. It may come as a surprise, but some managers aren’t stupid and don’t believe a developer who says that they are “almost finished” or “95% done”. At this stage, testers are seen as little more than an annoyance who is forcing developers to do extra work like daily builds, smoke testing and tracking bugs.
Stage 3 : Acceptance or “I suppose this isn’t a temporary thing is it?”
At the acceptance stage, the developers have accepted that this new trend of testing may actually catch on, and that it isn’t just a passing fad. Developers will now be used to looking at their issues queues in the bug tracking application, and realise that, yes, they do need to tell the test automatiors that they have renamed the login button on the home page of the application.
Stage 4 : Dependence or “I am at one with my testing brothers and sisters”
After the acceptance stage comes dependence. At this stage the developers on a project have come to rely on the feedback loop that comes from fully engaging with the testers on your team. They get their code verified BEFORE they check it in, they rely on the suite of automated tests to verify that their changes have not broken and they may even from time to time raise the odd defect themselves.
You feel guilty doing any coding without unit tests. You half consider adding tests to the changes you just made to your autoexec.bat! (Thanks Russ !)
Stage 5 : Withdrawal or “Oh my god, what do you mean you don’t have any testers?”
The withdrawal stage when a developer who is used to being at level 5, changes jobs and is plummeted back into a team at stage 1. I’m sorry, but the prognosis for this situation is not good, and the only real solution is to progress your team through all 4 stages right from the very beginning. Who knows, if you suggest it, they may think that you are beginning to sound like a pointy-haired boss.
If you want to get your team to stage 4, feel free to drop me a line.
Richard Griffin has ported WatirRecorder++ to work with WatiN. For more details please visit the post on Richard’s blog.