Test Automation – At Home in an Agile Environment

'A 'passing' test doesn't mean 'no problem.' It means no problem ' observed'. This time. With these inputs. So far. On my machine.' — Michael Bolton

In 2013 as many as 88% of the organizations responding to a Version One 'State of Agile Development' survey confirmed that they were practicing agile development and that number has only gone up since. Agile, obviously, is defined by an approach of short sprints, iterative development and short release cycles. Given the apparent time pressure on the test cycles testing expert Bolton's tune would ring true to many in software product development today (sorry couldn't resist the lame pun). The objective is faster testing and more code coverage so less 'technical debt' is passed on. So what's the way out? Many have considered test automation to be the answer

The faster cycles in Agile development mean the time available to test is shorter — an excellent case for automating the testing. Each successive release also means more features added and hence more code to be tested — more test cases to be covered in the same or less time. This would be practically impossible to do without automation testing. The iterative development approach also means a need for more robust regression tests to check that new releases don't break stuff already fixed in previous versions — again a strong case for a well put together automation suite. So, that seems to be quite categorical — agile product development absolutely needs test automation.

It seems important, thus, to start at the beginning and make test automation a consideration when the product is being designed - essentially design tests when the product and its features are being designed. This would allow for the test automation strategy to be based on what the product is expected to do rather than on specific iterations of the code. This could also allow designing automated tests that test at layers below the GUI and such which are impacted more at each iteration.

Assuming that test automation has the benefit of having been part of the product planning an Agile, read iterative, approach can also work in building a complete regression suite. Essentially this would mean building automation only for those features carried over into the current version from the previous version. The focus would be on those features that have become stable. Over the course of a few sprints as the features add up the automation of their unit tests would too leading to a regression suite that offers more or less complete coverage. A practical variant of this method is to divide the creation of the suite into parts & approach each of them separately, for eg. —the critical suite which must pass every single iteration, the 'must-have' suite that must pass all major release iterations and the 'nice to have' suite that can be run ad-hoc.

There are also movements out there that look at this differently. A case in point is the 'Test First' approach — in some ways this turns the traditional build first, test later approach on its head. This approach proposes to have the tests in place first and use them to validate if the code that has been created achieves what it is supposed to — different from the approach where the tests are used to determine if anything is not working the way it should. Clearly the planning burden here is high — the test automation team has to be firmly integrated into the product planning process at the very start to be able to make this work. A lot of testing professionals have their eye on this interesting approach to see how it pans out.

The test automation case is not without challenges though — the chief being when the releases are coming so thick and fast which target code base do you base the automation suite on? The other recurring theme seems to be an incomplete strategy — many test automation plans stop at the automation of the unit tests. A more complete automation strategy that addresses unit tests, integration tests, systems tests and obviously regression tests would likely have a much greater chance of delivering the promised benefits. Key to addressing both these challenges seems to be the ability of the product leadership to integrate the test automation team into the early stages of the product design and planning cycle.

---

Conclusion

---

In closing let us accept that Test Automation seems to have a crucial role to play in Agile Development — like everything else in software engineering though it need to be approached in a considered and organized fashion. Wasn't it Louis Srygley who said 'Without requirements or design, programming is the art of adding bugs to an empty text file.'