Test Automation May Not Be Right For You
Summarize With:
The wonderful book, 'Implementing Automated Software Testing' authored by Elfriede Dustin, Thom Garrett, and Bernie Gauf, mentions a worldwide survey they conducted on the subject. The online survey got over 700 responses and among a host of interesting results they included in their book was a startling finding. They reported that 72% of the survey respondents believed that automation was useful and even that their management agreed but despite that they either had not implemented it at all or had had only limited success.
Ways of Implementing Automated Software Testing
The book came out in 2009 and how things seem to have changed since then. Anecdotal evidence, at the very least, would seem to show that most software development efforts today consider automation initiatives somewhere along the way. Much has been written about why one should choose to automate testing - there are great technical as well as commercial arguments. That being said, though, as a testing and test automation company every so often we come across a situation when we recommend to the client that test automation may not be the best bet for them or for that specific development effort. Let's look at what these situations are where test automation may not be the best fit?
- Very less or almost no regression testing: Don't be surprised — there are a lot of situations where future iterations add on to what has gone before without really touching the old code base in any significant way. For example, remember the waterfall model - it's still very pertinent in many situations. There are many small, self-contained software projects or projects where the requirements are extremely clearly defined where this model plays well. Here since the development for each phase is completed and then accepted after testing, conventional wisdom is that the regression effort is reduced. In such situations, there is not much value in automating the testing since you are unlikely to gain any significant advantage of time once you factor in the effort of developing the test suite.
- Tests that will be run only once: In some ways, this is an extension of the earlier thought. Like everything else, there are choices to be made while automating your testing. If there are tests that are likely to be run on only 1 iteration or very few times in the overall effort then save your time, energy and money — don't automate them. Instead, prioritize those test cases that are likely to come up time and again in the development cycle.
- GUI is not yet frozen or there are frequently functional changes: In many ways, this is the opposite of the first condition. Here the requirements are very dynamic. The fact is that building the test automation suite is an effort and once the suite is built making tweaks to it to allow for changes that have crept into the project scope is neither easy nor advisable. Our advice is always to get the test automation team involved at an early enough stage in the project to be able to design, plan for and build an effective test automation suite — if such planning is not possible then chances are high that the test automation suite will be incomplete or ineffective. In that case, why do it at all?
- No skilled resources: We know, this is obvious. That being said you would be surprised how many test teams struggle to provision the right kind of people on test automation teams. Chances are for a test automation effort to be successful, developers will have to join the team — in the end, your automation suite is also a product and hence developing it is like any other product development effort. And it's not just developing it either, consider that this 'product' will need support and maintenance over its lifecycle. If you cannot find the resources with the right skills for this then you may be better off not automating.
- Seeking 100% Automation: There has been enough said about this. If you are seeking to automate your testing in quest of the mythical '100% Automated' one-horned beast, then just don't. It's not possible, not desirable and just plain wrong to start with inflated expectations. Set out on the automation path with tempered expectations or not at all.
Conclusion
In closing let us just say there is a time and a place for automated testing — just not every time and place. Brad Siims, co-founder of SandVine, explains the value of automated testing elegantly, 'With manual testing you can't reproduce problems as readily and the breadth of test-case coverage isn't as high. With more automation, you can focus your QA team on making a difference, versus manual re-cabling to run nightly tests.' Do you agree?