Flaky tests

A flaky test gives different results for the same code. Common causes in API testing are timing — asserting before an asynchronous job finishes — shared state between tests, data left over from earlier runs, real network calls, unseeded random data, and dependence on the current date or time zone.

How it works

Flakiness is diagnosed by rerunning the same commit until the failure reproduces and comparing what differed. It is fixed by removing the source of variation: wait for a condition instead of a fixed time, give each test its own data, seed generators, freeze clocks, and replace live dependencies with a fake or a sandbox you control.

The misconception

That an automatic retry fixes a flaky test. Retries hide the symptom and keep the cause; they also hide real intermittent bugs in the product — a race condition looks exactly like a flaky test. Quarantine and fix, rather than retry and forget.

Try it here

POST /protocol/v1/jobs returns 202 and the job progresses on each poll of GET /protocol/v1/jobs/{id} — a good place to practise waiting for a condition instead of sleeping. Reset the sandbox between runs to keep state from leaking.

Sources

Other terms

Test fixture — The known data and state a test starts from, set up the same way on every run.
Mock API — A stand-in for a real API that returns realistic responses without the real system behind it.
API sandbox — An isolated copy of an API where calls have no real-world effect and data can be reset.
Mocks, stubs and fakes — Three kinds of test double — stand-ins for a real dependency — that differ in what they do and what they check.
Contract testing — Testing that an API and its consumers agree on the shape of requests and responses, without running them together.
Smoke testing — A small, fast set of checks that the most important paths work at all, run before anything more thorough.

All glossary terms · Testing techniques · All 39 mock REST APIs

Last updated 16 October 2026