The Role of Software Testing in a DevOps World
Summarize With:
By Rajiv Jain (CEO, ThinkSys)
Mark Berry said, 'Programmers don't burn out on hard work, they burn out on change-with-the-wind directives and not 'shipping''. I don't know about the first but the desire to 'ship' seems to be a powerful motivation for the push towards the adoption of DevOps practices among software development teams and companies everywhere. That and the business benefits obviously. I remember a Puppet Labs survey from a couple of years ago that showed that of the organizations surveyed, those that adopted DevOps shipped code up to 30 times more frequently with lead times of as little as a few minutes.
They also experienced only half as many failures and had the ability to recover from those failures up to 12 times faster. These are not just numbers, I recall reading about one of the early DevOps adopters, Etsy. Apparently they update their site every 20 minutes without any service disruption to over 20 million users — that's a phenomenal competitive edge to possess. Small wonder then that the DevOps tide is rising.
Importance of Software Testing in Devops World
- Across most SaaS-based product development efforts, the approach seems to be to bring Development and Operations together in, a largely, automation-driven effort to keep pushing releases out, more or less continuously. Given the sheer number, it's perhaps not even fair to term these as releases any longer! Coming from a company that has a significant Software Testing practice though an interesting facet of this partnership between Development and Operations occasionally occurs to me. It seems to me that a third, equally interested party, Testing and QA, may not be getting the attention it deserves in this discussion. In fact, the case has sometimes been made that the greater degree of automation and the souped-up release cycles imply a reduced role for software testing. I couldn't disagree more.
- Let's go back to the Puppet Labs survey I quoted earlier — notice that one of the key benefits stated was the reduced rate of failures and the ability to recover from those failures quickly. Clearly the aim is not to ship code, it's to ship market-ready product on a 'Continuous' basis. To me it seems that would be impossible to achieve without an even greater emphasis on testing. Testing, while Agile had brought out the need to involve testing strategy at a very early stage of the product planning. How else to keep up with the short release cycles? DevOps with its Continuous Integration and Continuous Delivery is, in this context, like Agile on steroids and the need, thus, is even greater to get testing involved early in the product definition and planning stage. In many ways, the approach now has to be to plan to prevent defects rather than detect them.
- In DevOps testing comes in early, plans for the way the product is going to pan out and prepares itself accordingly so that as code starts rolling out it gets tested. This is a significant change, at least of mindset. Earlier the development used to get done and the testing would start — today these seem to have to go, more or less, in parallel. Many have called this Continuous Testing — a perfectly valid term.
- Another significant change that this 'Continuous' model is engineering is the integration of functions. DevOps teams now tend to contain people from Development, Operations and, increasingly, Testing. Without that level of integration, getting quality code out in such short time frames would be impossible. This integration is also throwing up new roles for everyone, including testing. Carl Schmidt, CTO of Unbounce explains it well, 'I'm of the mindset that any change at all (software or systems configuration) can flow through one unified pipeline that ends with QA verification. In a more traditional organization, QA is often seen as being gatekeepers between environments. However, in a DevOps-infused culture, QA can now verify the environments themselves, because now infrastructure is code.'
- That statement points to a third significant change for testing in the DevOps way - what to test? Now that the Operations, i.e the nuts and bolts that get the SaaS product out to the millions of subscribers, is a part of the effort of getting the product out is it not necessary to test that too? There are also some subtle changes of emphasis that emerge here — functional testing is always important but now there is a premium placed on testing for load conditions and for performance. The environment is also, as much a part of the SaaS product now as the code. There is a role here that testing can excel in — clearly they know more than most about the difficulties of taking the application code, deploying it in test environments, testing it and then moving it out to production. The process may have become shorter and sharper but the skills are the same.
Conclusion
I like a quote, sadly unattributed, that I read somewhere, it goes, 'Software testers do not make software, they only make them better.' In the context of DevOps it would seem that finally they are getting an opportunity to not only make software but also to make it better!