Software Testing is at the heart of new software development methodologies such as Behavior Driven Development, Test Driven Development and of course, Agile. In a previous blog on the role of testing in Behavior driven development we touched upon two topics, Acceptance Tests vs Acceptance Criteria and how BDD has changed the approach towards these testing stages. In this blog, we take a look at these similar sounding and yet very different concepts.
What is Acceptance Criteria?
It thus becomes essential to first define what the product is expected to do and the conditions it must satisfy to be accepted by a user. In order to achieve this, testers need to flesh out comprehensive ‘user stories’, then iterate criteria specific to each of these user stories and define the value proposition, characteristics of the solution and user flow. Testers then need to develop test cases based on these user stories and define conditions that need to be satisfied for the product to be “acceptable” by a user. These set of conditions that define the set of standards that the product or piece of software must meet are called ‘Acceptance Criteria’.
Loosely speaking, Acceptance Criteria documents the expected behavior of a product feature. It also takes into account cases that could have been missed by the testing team while developing test cases. Defining the Acceptance Criteria is the first testing step that comes after writing user stories. Usually, the Acceptance Criteria is concise, largely conceptual, and also captures the potential failure scenarios.
Acceptance Criteria are also called ‘Conditions of Satisfaction’. These consist of a set of statements that specify the functional, non-functional and performance requirements at the existing stage of the project with a clear pass or fail result. Defined Acceptance Criteria outline the parameters of the user story and determine when a user story is completed.
Acceptance Criteria should always be written before development commences so that it can successfully capture the customer intent rather than iterate functionalities in relation to the development reality. Acceptance Criteria thus should be written clearly in a simple language that even non-technical people, such as the customer and product owner, can understand. The idea behind writing an Acceptance Criteria is to state the intent but not the solution and hence it should define ‘what’ to expect rather than ‘how’ to achieve or implement a particular functionality.
What is Acceptance Tests?
Acceptance Testing is the process that verifies if the installed piece of code or software works as designed for the user. It is a validation activity that uses test cases which cover scenarios under which the software is expected to be used and is conducted in a ‘production-like’ environment on hardware that is similar to what the user or the customer will use.
Acceptance Tests assert the functional correctness of the code and hence contain detailed specifications of the system behavior in relation to all meaningful scenarios. Unlike Acceptance Criteria that defines the expected behavior of a particular feature, Acceptance Tests ensure that the features are working correctly and defines the behavior of the system and hence demand more detailed documentation. Acceptance Tests check the reliability and availability of the code using stress tests.
This also checks the scalability, usability, maintainability, configurability and security of the software being developed and determines whether a developed system satisfies the Acceptance Criteria and checks if the user story is correctly implemented.
Acceptance Tests can be written in the same language as the code itself. So, these tests can be written in Gherkin language used commonly in Behavior Driven Development.
While Acceptance Criteria are developed prior to the development phase by the product owners or business analysts, Acceptance Tests may be implemented during product development. They are detailed expressions that are implemented in the code itself by the developers and the testers.
Acceptance Testing is usually performed after System Testing before the system is made available for customer use. To put it simply, Acceptance Tests ensure that the user requirements are captured in a directly verifiable manner and also ensure that any problems that were not identified during integration or unit tests are captured and subsequently corrected.
There are two kinds of Acceptance Testing, namely:
a) Internal Acceptance Testing:
Performed in-house by members who are not involved in the development and testing of the project to ensure that the system works as designed. This type of testing is also called Alpha Testing.
b) External Acceptance Testing:
– This testing is of two types:
a) Customer Acceptance Testing – Where the customer does the testing.
b) Beta Testing or User Acceptance Testing – Where the end users test the product.
Conclusion:
In conclusion, we can say that amongst other things, the main difference between Acceptance Criteria and Acceptance Testing lies in the fact that while the former defines ‘what needs to be done’ the latter defines ‘how it should be done’. Simply put, Acceptance Tests complete the story started by Acceptance Criteria and both together make sure that the story is complete and of high functional value.