As you can tell from my blog, I am a big fan of WatiN. One thing that WatiN doesn’t do is test non IE stuff. Well as part of .net 3.0 Microsoft has now shipped API’s that allow testing of Windows apps, including both legacy Win32 and WPF applications. There is an MSDN article on how to do this here.
So what does this mean? It essentially means that commercial tools are dead in the long term. If it is possible for a single developer to build tools that work better than commercially available tools costing hundreds or thousands of dollars, which it currently is. It won’t be long until they are not purchased any longer.
Another interesting phenomenon is that companies are trying to advertise their tools by adding comments to blogs, such as this one saying things like, you should try tool xxxx.
Sorry but when your commercial tool costs more and does less than the open source alternative, it is time to look for a new product to develop.
Another major irony is that if the open source tool has a bug, or doesn’t have a feature I need, then you can simply open up the code and fix it or add the feature. Most commercial vendors take 4-6 months or more to investigate, test and resolve issues.
So if you haven’t yet, download a copy of WatiN, WatiNRecorder, nUnit and C# Express, then see what the future is like and join in by submitting back features.
One final word, Webiussoft and Inscif, please stop trying to promote your commercial tools by adding comments to blogs about WatiN, we aren’t changing !
March 23rd, 2007 at 9:37 am
Bruce,
About your article: +1
About that last sentence: ++1
Jeroen
March 23rd, 2007 at 9:52 pm
Don’t forget SWEA. It is so annoying!
June 12th, 2007 at 3:52 am
Teknologika, it is true than promoting commercial tool by adding useless comments to blogs is not a good way. And after thinking about it, I stopped doing this.
If I found interesting posts about testing and if and only if I really have something good to add or a question to ask then I add a comment.
At the same time I have a family and bills to pay, so is it fair that WatiN is free (Life is not fair anyway).
How can we compete with a free tool ?
Is free software a reaction to a market based economy and capitalism ?
Note I am not against open source.
I am thinking about way to give our customers access to the source code of our product for 2008.
Can’t wait for your answer.
June 12th, 2007 at 5:09 am
Frederic,
I am glad to see that you understand you were doing yourself more harm than good with those comments. Now hopefully you will actually put what you said into practice.
How do you compete with open source?
Well you need to have a better product, with better features at a price point that people will pay for. Also you can leverage open source to your competitive advantage as well.
Take WatiN for example, there is nothing stopping a commercial vendor using WatiN as the core playback engine and spending their time to build tools around it. Instead of spending your time working on a low level engine, you can spend your time instead on making the technology accessible to non-developers.
Open source is more than giving customers access to source, it’s about letting the end users define and shape the form that a product will take instead if having the destiny decided for them.
The “killer feature” for me about WatiN has been that it lets me use the tools that I use every day (Visual Studio 2005), and I can add features or make patches to the core code, instead of spending hour after hour working with customer support to get a patch to a problem released at some point in a future release.
Regards,
Bruce McLeod
May 12th, 2009 at 4:45 pm
Nice article, but I really can’t agree with it especially with “commercial tools are dead in the long term”.
Even if, I am also a big fan of opensource testing tools, I had the “chance” to participate in projects of implementing automated software testing in very large companies(telecom, banks) and I really understood their need of a comercial testing tool because of the suport, and reliability they ofer.
These companies had large testing departments but small or non existing development teams, so “simply open up the code and fix it or add the feature” its imposibile or in the best case scenario very expensive.
Till now I haven’t still meet this situation: “Most commercial vendors take 4-6 months or more to investigate, test and resolve issues.” and I think that there is no guarantee that a comunity of an open source tool could solve an issue faster.
So in conclusion I think that: <<commercial tools are dead in the long term, for software development companies which have the knowledge and the people to suport or to build their own open source testing tool ”
Regards,
John
May 12th, 2009 at 5:42 pm
Sorry John but I couldn’t disagree more. One of the reasons we changed from commercial tools was the lack of support and reliability. In our case the commercial tool was causing no less than 7 IE crashes, the sent a developer our from the US to fix the problems and after 4 weeks onsite she only managed to solve 3 of them.
The thing about open source tools is that I can debug them and solve the problems myself. To date there is no issue that I have been unable to solve myself, and that includes building my own UI automation library from scratch to test windows applications.
The other thing that I didn’t say at the time when I wrote this post back in 2007 as I was under NDA, is the inclusion of the testing tools in Visual Studio 2010. So the way of the future is a) open source or b) use Microsoft’s tools.
May 13th, 2009 at 1:00 am
Ok, so I will agree with you that not all commercial tools are good, and not all testing tool producers are capable. The programmer that couldn’t fix the problems is not relevant, because we can asume that someday everybody could face a problem with his tool and not be able to fix it…
If that test tool would have been open source, and you had the posibility to debug and solve the problem yourself doesn’t automatically mean that you were necessarily able to do it. When I say you I don’t mean your person, I’m just reffering to a general programmer/tester from a company… and we are not all, geniuses.
And you didn’t said nothing about open source tools in non-development software companies, were I consider the commercial alternative the best one.
So again a I must say that the future is a) open source, b) use Microsoft’s tools, c)good commercial tools and services.
Regards, and sorry for my late answer(2009), I didn’t look at the date of your blog post before sending my previous message.
May 13th, 2009 at 10:36 am
John,
Comments and friendly debate is always welcome, the date doesn’t matter.
I agree that not all problems are solveable, but, when you are charging $10,000 plus per client and the software cannot perform the basic function that you are paying for, you expect that the problem will be solved.
Commercial tools definatley COULD exist in this space, however, IMHO, the current tool vendors lag behind the mainstream technology, have products that themselves are very buggy and do not justify the price that is charged for them.
Having to re-run 30% of your 1200 tests every day because of a tool bug that cannot be fixed by the vendor is completely unacceptable, and when the open-source replacement has worked flawlessly ever since, I would rather spend that $10,000 on a good testing services company to write tests in an open source tool instead.
But, as you point out, I am not an average tester, but I have different expectations than most of what a tester should be … http://www.teknologika.com/blog/the-tester-is-dead-long-live-the-sdet/
May 13th, 2009 at 5:08 pm
I think we both talk from our experience, and I must say that, I am very frustrated that, even if I searched the web desperately in order to find an open source tool that could perform load testing for something else than web protocols, I couldn’t find one.
So because I’m not a programmer, I won’t develop my own performance testing tool or add the features I want to an existing one, and I will have to use expensive commercial tools with or without their flaws.
What I’m trying to say is that even if there are hundreds of open source tools, for a regular testing effort they are, so often(at least for me), unsuitable.
I admit that for testing a web application open source is the best way to go, but in other areas… it is more to be done.
Also there are a lot of commercial tools out there, so finding a tool that suits your needs is not that dificult but it is time consuming.
Sadly, for me, the commercial tools are also stringly oriected on web, and only the most expensive ones cand be used for other tehnologies.