Shift Left on a Budget: Cost-Savvy Testing for Microservices

Developer velocity is like weather: Everyone talks about it, the basic mechanics have been understood for decades, but can anyone really control it? The reason that features come out later than expected is an example of the John Kennedy quote: “Victory has 100 fathers and defeat is an orphan.” Everyone has blame for work taking too long, but no one wants to claim responsibility.

Currently, the methodology for teams’ testing doesn’t work: It’s a poor developer experience and it’s causing pain every time we try to release code. We know this is a problem, but we can’t agree on the exact causes and solutions. Some of the current proposed changes — expansion of testing and replication of environments — are expensive, ineffective measures. Here are some cost-efficient solutions.

Why It’s Harder To Ship Code: Testing

As platform engineers worked to measure DORA metrics and thereby quantify the challenges of getting code out to production, meetings among platform engineers identified some common patterns hurting releases. While CI/CD pipelines have automated the release of code, too often these pipelines jam because the code being merged encounters problems during automated testing.

When we speak to dev teams, they admit that code that’s merged to staging is often low-confidence without a ton of evidence that it will run on staging. The result is CI/CD pipelines that run automatically but can still take days to actually release code, with developers waiting on long-running processes to get their first realistic feedback on how their code works.

The Problem With Current Testing Approaches 

As we adopt microservice architecture, the ability to run reliable integration tests reduces, as we have many service dependencies and a more complex architecture that is harder and harder to simulate. The result is a reliance — or overreliance — on end-to-end (E2E) and unit tests, leading to the challenge of improving coverage without escalating costs.

E2E tests, where all the services that would handle a production request run on a staging environment, offer an incredibly reliable way to know your code is working. The trouble is that in a normal architecture these end-to-end tests can only be run fairly late in the process on a final staging environment, where all the changes from other teams also need to be tested.

To make E2E tests run earlier, teams adopt many expensive measures:

• Adding an environment before staging is expensive in terms of time, as platform/QA teams must then maintain two environments, and critical code releases must go through two stages.
• Replicating the whole staging environment for each developer or dev team presents two problems: the infrastructure expense of running a large number of services around the clock just for occasional testing, and the difficulty of keeping “staging clones” correctly updated.

In response to these limitations, some teams choose to implement more of the testing they already utilize in the hopes of making their testing phases more reliable.

Why Brute-Force Testing Isn’t the Answer

Running a large number of early integration tests or an ever-expanding list of brittle, late-feedback E2E tests have a number of drawbacks as solutions for testing reliability. Integration tests that already failed to catch most problems before a merge to staging aren’t likely to fix our problems if we simply use more of them. And as implied above, E2E test feedback comes too late to make a nice, quick feedback loop for developers. Any approach that relies on expanding the number of tests also adds to one central problem: Tests are themselves code, and expanding a codebase means increasing maintenance work.

Developers are, naturally, reluctant to write a ton of new