Contract testing scales better than end-to-end testing

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

This is an interesting write up from Nubank. They had end-to-end tests that would call a given API endpoint, which might be connected to a service that then called some other service, which might itself call another service. Because of the multiple connections, the testing could be slow, and the slowness got worse as they scaled up. Worse, because multiple apps were involved, when there was a bug it was difficult to know in which of those apps the bug had occurred. So they switched to Contract testing, which only looked at the inputs and outputs to one API. But here they had to go back to using mocks, so they lost the ability to be sure that the real world system was working, but again, they’d already lost that ability due to the sheer scale of their system.

Another aspect of their project is that they had already used Clojure Schema, and so they found they already had contracts for everything:

“And so we decided to create Sachemour very own contract testing framework, to deprecate End-to-End testing in staging as a practice. One interesting thing about this project is that we realized we already had the contracts in our microservices. Clojure has this library Schema that allows you to richly describe data structures. “

It was already common practice to write schemas for every HTTP endpoint and Kafka topic. What we did was to build a tool that collects those schemas and checks if they are compatible. 

Post external references

  1. 1
    https://building.nubank.com.br/why-we-killed-our-end-to-end-test-suite/
  2. 2
    https://github.com/plumatic/schema
Source