What I’ve learned about Acceptance Testing
The typical problems some teams have to face when implementing acceptance testing are: Very long builds and flaky tests.
Both problems are usually caused because developers tend to omit unit and integration tests and move them up to the acceptance test tool. Notice that these extra ‘technical tests’ add up more time to the build (seconds rather than milliseconds). As they are not in their natural test level, they usually require more complex setup (more stubs) and they are more likely to leave the system in an inconsistent state and break test interdependency which ends up in more flaky tests.
One of the desperate solutions I’ve seen in different projects is to parallelize the build by running groups of tests in different threads. This usually adds up more problems such as typical concurrency issues. I am not saying that this can’t work, I am saying that it’s difficult to get it done and may take a long time to do it right, longer than fixing the root cause.
The root cause of the problem is the wrong assumption that acceptance testing tools are general purpose automation tools. No, they are collaboration and communication tools between the customers and the team. They help to understand a business problem by providing examples of how to use it, they become live documentation and synchronize documentation when new understandings are discovered. It is crucial to keep them clear, readable, very well organized and easy to understand by customers, QA and other developers, especially new starters!
I remember when I started to work (more than ten years ago) that my team spent six months implementing a product based on the documentation written by BAs. When the product was delivered and tested by the customer we discovered a very important misunderstanding of the business that cost a lot of money. We spent 6 months developing the wrong thing. On top of that, the team could then understand many of the business rules much better than in the beginning so the initial documentation had rotten and was completely deprecated.
What I am trying to say in my previous paragraph is that, if you think about it, implementing the business rules (the domain) is the most difficult part of any software application because they are different in every project and they need to be discovered. The interface or the integration with other third party services such as databases or other web services are trivial technical challenges solved by many frameworks. Understanding what the application should do helps us to build the right software. Solving the technical issues helps us to build the software right.
Taking this into account we can divide tests in two groups:
- External tests which verify that the software does what the customer wants (the right software)
- Internal tests, which verify that the software works without technical errors (the software right)
Acceptance tests belong to the external tests group. They check the quality of the product from the customer point of view.
What to test, what not to test in the acceptance test layer
Acceptance tests usually test whole features of the system from end to end as a black box: interface + domain + repositories. Each test takes seconds to run! This is very expensive in terms of time. We should be very careful about what to test and what not test at acceptance level. My recommendations are:
- Only key examples of business scenarios
As I mentioned before, we should test only business scenarios. Not all of them, only the key examples.
E.g.: Give 50 pounds to customer if account has enough funds; Do not give $50 pounds if not enough funds.
- Test complete features, do no test technical issues
As I mentioned above, acceptance tests are about keeping live documentation about how to use the system. We should explain WHAT the system should do from the customer point of view. We should exclude technical and trivial tests such as what the system should do if an invalid input or database failure. That’s trivial! We should be testing technical stuff with unit tests and integration tests because these ones are much faster to execute, they are easier to setup and they provide quicker feedback. E.g.: If pounds is null, display error
- The interface can be excluded from tests
If you actually understood how difficult is discovering what the software must do (business rules), you might have reached the conclusion that in order to achieve this goal it should be only necessary to test the business model. The interface has nothing to do with the business model so we can exclude it and save precious time. On top of that, testing the interface is sometimes pointless because it doesn’t guarantee anything: an html page that passes tests could be displayed wrongly on a different browser or system. It also can be tedious to setup: it requires to start a browser, some times to restart a server, etc. It’s a high cost and it doesn’t intervene in the business model. There are better tools to do this.
Tag your acceptance tests
Even though implementing acceptance tests correctly by having business requirements only, we can end up with a huge build that takes ages to execute.
In the beginning of the project, the continuous integration server takes less than 10 minutes to execute the whole suite of tests (unit + integration + acceptance) and it’s ok that each developer waits for the whole build to complete. But when it grows and takes longer than that, you have a problem.
The important thing here is to find a balance between what tests must be run always and what tests can be delayed. If you count on a good suite of unit and integration tests, you should be able to commit with confidence after these ones pass. Remember that these tests are fast and they shouldn’t take more than 10min to run. The hard part is the acceptance tests.
The policy we used in some of my projects was:
- We tagged every suite of tests and run locally only the ones related to the feature we were developing. E.g.: if I was developing something related to products, I run only tests tagged as products. You can use tags in the properties pages in Fitnesse; Cucumber uses the format @tag.
- We tagged tests by importance and we told the Continuous Integration server to run only the most important features after every commit and run groups of less important features every 30min, every hour and so on. Developers could commit after the important tests passed. This post by Richard Paul talks about this.
“Work In Progress” tag
It is a good idea to tag features under development as “work in progress”. When you do this, you tell the Continuous Integration server not to run “work in progress” scenarios so you can commit without breaking the build. This can be used along with the Feature Toggling suggested by Martin Fowler so you can disable unfinished code.