Do you feel that your end-to-end testing delivers value?
End-to-end tests can take a long time to set up, can be complicated, and are often flaky. Why is this? Lots of moving parts (environment requirements), lots of interfaces between components, and lots of rigidity and potential brittleness in our toolchains when things change.
It’s not just that e2e tests are not just hard to set up; they are also challenging to maintain.
This blog post by Javier Lopez (medium members only) got me thinking about this subject. I’m distilling down what I believe the core elements to replacing an end-to-end test should be. That blog post was itself inspired by this one by JS Rainberger.
The end-to-end test will take several components, glue them together in a way that you use and test the ‘happy path’ when things just work. How often is this representative of your actual system? And how often do we get to a point where the test is so complicated to maintain that it just fails too regularly?
Several points here:
- What are the distinguishing features of good tests?
- What do we do when tests fail regularly?
- What value do our end-to-end tests give?
- What alternative is there?
Javier proposes using FIRST principles for deciding on good tests:
- Fast
- Isolated/Independent
- Repeatable
- Self-validating
- Thorough
What do we do when tests fail regularly? We either ignore them or turn them off. How often do we check them?
So then our value for end-to-end tests is low. They are only ‘happy path’ tests that validate when everything is working.
What is the alternative?
Contract Testing and Test Doubles.
The summary of this is: let’s make sure that the component that we are working on has a contract with the rest of the services in our application and let’s make sure it tests itself.
Javier found this brilliant quote from Kent Beck on StackOverflow:
Kent Beck : “I get paid for code that works, not for tests”
When we create a component, we ensure that its interfaces are doubled for the purposes of testing. That means we create three things:
- The code itself in component form
- The interface towards other components
- An interface “double” which returns dummy results for testing purposes
The exciting thing about the Kent Beck quote, and the subsequent comments, is that there is no ‘perfect’ solution for testing. It’s something we must do, but we don’t know how much of it we must do. But by sticking to FIRST principles and keeping testing local – we break the dependency between systems which makes testing them so tricky.