Log in
← Blog

Flaky Tests: The Seven Usual Causes and How to Fix Each

A test that passes and fails without any code change teaches everyone to ignore failures. The seven causes behind most flaky tests, how to spot each one, and how to fix it for good.

Vihari Panyam

July 28, 2026 · 5 min read

A debugger paused at a breakpoint in a concurrent test, with a list of running threads below

In short

  • A flaky test passes and fails on the same code. Google reported that about 1.5% of its test runs were flaky, and almost 16% of its tests had some flakiness.
  • Most flakiness comes from a handful of causes: timing, shared state, test order, external services, time and dates, randomness, and the environment.
  • Retries hide the problem. Quarantine the test, find the cause, fix it, and track flakiness like any other bug.
On this page

The pipeline goes red. Someone clicks "re-run". It goes green. Nobody looks into it, because it does this every few days. Three weeks later a real bug turns the same test red, someone clicks "re-run", and it’s still red, so they re-run it again, and then they mute the test. The bug ships.

That’s the real cost of a flaky test. Not the minutes lost re-running it, but the habit it builds of ignoring red. Google wrote about this in 2016: across its tests, about 1.5% of runs reported a flaky result, almost 16% of tests showed some flakiness, and about 84% of the pass-to-fail transitions they observed involved a flaky test. At that rate, red stops meaning anything.

01Cause 1: timing and waiting

The test clicks a button before it’s ready, or checks text before the page finished loading. It passes on a fast machine and fails on a busy CI runner.

Spot it: fails more on CI than locally; failures mention "element not found" or "timeout".

Fix it: wait for the condition you need, not for a fixed time. Remove every sleep(2000). Tools like Playwright retry assertions until they pass or time out, so assert on the visible result rather than checking once.

// Flaky: checks once, possibly before the cart updates
await page.click('text=Add to cart');
expect(await page.textContent('#cart-count')).toBe('1');

// Stable: waits until the cart shows 1 (or fails after the timeout)
await page.getByRole('button', { name: 'Add to cart' }).click();
await expect(page.getByTestId('cart-count')).toHaveText('1');
typescript

02Cause 2: shared state between tests

Two tests use the same user account. One changes its address while the other is checking out. Each passes alone; together they fail sometimes.

Spot it: fails only when the suite runs in parallel, or only when certain tests run together.

Fix it: give every test its own data. Create a fresh user, cart or record at the start of each test, with a unique name, and don’t rely on anything another test created.

03Cause 3: test order

A test only passes because an earlier test happened to log in, or set a feature flag, or created the product it needs.

Spot it: run the suite in random order, or run the single test alone. If it fails alone, it depends on another test.

Fix it: each test sets up everything it needs itself. Use setup hooks for shared preparation, never "test 1 creates the data test 5 uses".

04Cause 4: external services

The test calls a real payment sandbox, email provider or third-party API that is sometimes slow or down.

Spot it: failures cluster at certain times of day, or the error comes from another company’s domain.

Fix it: in most tests, replace the external service with a stub that returns known responses. Keep a small, separate set of tests that check the real integration, run less often, and are clearly labelled.

05Cause 5: time and dates

A test that checks "delivery in 3 days" fails over weekends. A test that creates a booking "for tomorrow" fails at 11:30pm when the server is in another time zone. A test passes every day except the last day of the month.

Spot it: failures on particular days, at particular hours, or around daylight-saving changes.

Fix it: control the clock. Most frameworks let you set a fixed time for a test. Never compare against "now" without fixing what "now" is.

06Cause 6: randomness and ordering

The app returns search results in no guaranteed order, and the test expects a particular product first. Or the test data uses random values that occasionally hit an edge case.

Spot it: the failing value is different each time.

Fix it: sort before comparing, or check membership instead of position. If you use random data, log the seed so a failure can be repeated exactly.

07Cause 7: the environment

Not enough memory on the CI runner, a different browser version, a screen size that hides a button, a test database that wasn’t cleaned up.

Spot it: fails on one runner type, or started failing after an infrastructure change with no code change.

Fix it: pin browser and tool versions, run tests in containers with fixed resources, and reset the database or use a fresh one per run.

08Why automatic retries make it worse

Retrying a failed test automatically turns the pipeline green, which feels like a fix. It isn’t. It hides the flakiness, and it also hides real intermittent bugs in the app, such as race conditions that customers will hit. Use retries to collect information, not to pass builds: if a test needed a retry, record it and count it.

Flaky test checklist

  • No fixed sleeps; tests wait for conditions
  • Every test creates its own data and cleans up
  • The suite passes in random order and in parallel
  • External services are stubbed outside a small, labelled integration set
  • Tests control the clock instead of using the real time
  • Random data is seeded and the seed is logged
  • Retries are recorded and counted, not used to hide failures
  • Flaky tests are quarantined, owned and fixed within a set time

Frequently asked questions

What is a flaky test?

A test that sometimes passes and sometimes fails on exactly the same code and configuration. The failure is caused by the test or its environment, not by a change in the code being tested.

What causes flaky tests?

Most often timing problems, shared state between tests, dependence on test order, unreliable external services, dates and time zones, randomness, and differences in the test environment.

Should I just retry flaky tests?

Retries can keep a pipeline moving in the short term, but they hide the cause and can hide real intermittent bugs. Record every retry, and fix or remove the flaky test.

How do I find which tests are flaky?

Re-run the same commit several times and compare results, run the suite in random order, and track tests that fail and then pass without code changes. Many CI tools report this automatically.

Can a flaky test be a real bug?

Yes. Sometimes the test is fine and the app has a race condition or timing bug that only shows up occasionally. That is why every flaky test deserves investigation, not just a retry.

Written by

Vihari Panyam

QualityAIQ

Sources and further reading

  1. Google Testing Blog: Flaky Tests at Google and How We Mitigate Them (2016)
  2. Martin Fowler: Eradicating Non-Determinism in Tests
  3. Playwright: Test retries
  4. Playwright: Best practices

More in Test Automation