Detecting Flaky Spatial Tests with Rerun Statistics

9 min read

A flaky test passes and fails against unchanged code and unchanged data. This guide sits beneath spatial test observability and metrics and treats flakiness as a measurable property rather than an annoyance to be retried away: recording rerun outcomes as a metric, separating the causes that are specific to spatial work from ordinary timing instability, and running a quarantine policy that keeps the suite’s verdict meaningful while the cause is found.

Spatial suites are unusually prone to a specific kind of flakiness, and it is not the timing kind. The dominant causes are result ordering and engine version, and neither is fixed by a retry — a retry simply reruns until the ordering happens to be the one the assertion expected, which converts a deterministic bug into an intermittent one.

Root cause: retrying hides the signal it was meant to manage

An automatic retry is attractive because it removes an immediate obstruction. It also destroys the information that would have identified the cause, and it does so silently: the second run’s success replaces the first run’s failure, and nothing records that a failure happened at all.

The consequence compounds. A suite with retries enabled reports a pass rate that is not the pass rate of any individual run, so the trend is invisible; a test that fails one run in three looks identical to one that has never failed. By the time the instability is bad enough to survive the retries, it has usually been present for months.

Ten runs, reported two ways Ten consecutive runs of a test that fails on its first attempt three times and passes on the retry each time. In the upper strip, where only the final outcome is recorded, all ten runs appear as passes and the instability is entirely invisible. In the lower strip, where every attempt is recorded, the three first-attempt failures are visible and yield a first-attempt pass rate of seventy per cent, which can be trended over time and alerted on. A closing note records that the retry removed the obstruction and the diagnostic information in the same action. Final outcome only 10 / 10 pass — nothing to see Every attempt recorded retriedretriedretried first-attempt pass rate 70% — a series that can be trended and alerted on The retry removed the obstruction and the information in the same action. Recording the attempt costs nothing and keeps the second.

Cause reference

Cause Signature Retry helps? Fix
Result ordering Fails on set comparison; passes when sorted Only by accident Compare sets, or sort by a stable key
Engine version Fails on one runner image, passes on another No Pin the engine; assert the version
Shared state under parallelism Fails only with -n > 1; moves between workers Masks it Per-worker schema, per-worker temp path
Missing grid file Fails on a subset of runners consistently No Guard at start-up
Timing or resource Fails under load; passes when the runner is idle Yes, genuinely Raise the timeout, or reduce the work
Unseeded randomness Fails unpredictably, never reproduces Masks it Seed the generator; record the seed

Only one row in that table is a case where a retry is a legitimate remedy rather than a concealment, and it is the least common in spatial suites.

Step-by-step implementation

The instrumentation targets pytest 7+ with a rerun plugin, and records rather than hides.

Step 1 — Record every attempt, not the final outcome

The report hook fires per attempt, which is exactly the granularity needed.

# conftest.py
import json, os
from collections import defaultdict

ATTEMPTS = defaultdict(list)

def pytest_runtest_logreport(report):
    if report.when == "call":
        ATTEMPTS[report.nodeid].append("pass" if report.passed else "fail")

def pytest_sessionfinish(session, exitstatus):
    unstable = {nid: outcomes for nid, outcomes in ATTEMPTS.items()
                if len(set(outcomes)) > 1}
    Path(os.environ.get("FLAKE_REPORT", "flakes.json")).write_text(
        json.dumps({"unstable": unstable, "total": len(ATTEMPTS)}, indent=2)
    )

A node whose attempt list contains both outcomes is unstable by definition — no heuristics, no thresholds.

Step 2 — Attach the environment to every record

Flakiness that correlates with a runner image or an engine version is not flakiness at all, and the correlation is only visible if the fingerprint is recorded alongside.

import shapely, pyproj

def environment_fingerprint() -> dict:
    return {
        "geos": shapely.geos_version_string,
        "proj": pyproj.proj_version_str,
        "image": os.environ.get("IMAGE_DIGEST", "unknown"),
        "workers": os.environ.get("PYTEST_XDIST_WORKER_COUNT", "1"),
    }

Step 3 — Reproduce before quarantining

The three spatial causes are all deterministic given the right conditions, so a reproduction attempt is short and usually conclusive.

# Ordering: does it pass when the result is sorted?
pytest -q tests/test_join.py::test_cardinality -p no:randomly

# Parallelism: does it only fail with workers?
pytest -q tests/test_join.py::test_cardinality            # serial
pytest -q tests/test_join.py::test_cardinality -n 4        # parallel

# Engine: does it pass on the pinned image and fail on the candidate?
docker run --rm -v "$PWD:/w" -w /w gis-test:pinned pytest -q tests/test_join.py

Step 4 — Quarantine with an expiry, not indefinitely

A quarantined test that never leaves quarantine is a deleted test with extra steps. Attach an owner and a date.

import pytest

@pytest.mark.quarantine(owner="mapping-platform", until="2026-09-30",
                        reason="ordering under xdist; see INC-4812")
def test_join_cardinality(parcels, zones):
    ...

A collection hook that fails the run when a quarantine marker’s date has passed is what makes the expiry real rather than decorative.

Three questions before considering a retry A triage sequence for an unstable spatial test. The first question asks whether the failure occurs only when the suite runs in parallel; a yes identifies shared state between workers and the remedy is per-worker schemas and temporary paths. The second asks whether the failure correlates with particular runner images; a yes identifies an engine version difference or a missing datum grid file and the remedy is pinning plus a start-up guard. The third asks whether the test passes when the result set is sorted before comparison; a yes identifies result-ordering dependence and the remedy is to compare sets or sort by a stable key. Only when all three answers are no is the cause genuine timing or resource pressure, which is the one case where an automatic retry is a legitimate remedy rather than a concealment. 1 · Fails only with -n > 1? run it serially and compare yes → shared state between workers per-worker schema and temp path; a retry masks it 2 · Correlates with a runner image? group the attempts by image digest yes → engine version or a missing grid pin the engine; guard the grid at start-up 3 · Passes when the result is sorted? compare sets instead of sequences yes → result ordering no ORDER BY guaranteed what the assertion assumed All three no → genuine timing or resource pressure — the one case where a retry is a remedy rather than a concealment. In spatial suites this is the least common of the four, which is why an automatic retry is the wrong default here.

Verify the fix

Run the suite repeatedly against unchanged input and confirm the report is empty:

for i in $(seq 1 10); do pytest -q tests/ -p no:randomly; done; jq '.unstable | keys' flakes.json

Ten identical runs producing an empty unstable list is weak evidence; ten runs with -n 4 producing an empty list is much stronger, because parallelism is where the spatial causes surface. Running both is the cheap version of a stability check.

A quarantine policy that keeps the suite honest

Quarantine exists so that one unstable test does not block everybody while its cause is found. It becomes corrosive when it is used as a disposal route, and three rules prevent that.

Quarantine excludes from the gate, not from the run. The test still executes and still records attempts; it simply does not fail the build. That preserves the very data needed to diagnose it, which a skip destroys.

Every quarantine has an owner and a date. Without both it is permanent. A collection-time check that fails the run when a date has passed is what converts the date from a comment into a commitment.

The quarantine list is reviewed, and its length is a metric. A list that grows quarter over quarter is a suite losing coverage, and the number is the honest summary of that. Watching it alongside the pass rate keeps the trade-off visible.

The complementary practice is worth stating: a test that has been stable for a long period and then becomes unstable is far more likely to indicate a real change than one that has always been marginal. Recording when instability began — which the attempt series gives free — separates the two, and the first case usually points at the environment or an upstream input rather than at the test.

Measuring the suite, not only the tests

Individual test stability is the obvious metric and not the most useful one. Two suite-level numbers say more about whether the gate can be trusted, and both fall out of the same attempt records.

First-attempt suite pass rate. The fraction of runs in which every test passed on its first attempt. This is the number that describes what an engineer actually experiences, and it degrades long before any individual test looks bad — twenty tests each failing one run in fifty produces a suite that fails roughly one run in three.

Time lost to instability. Attempts multiplied by their duration, summed over a period. Expressed in hours per month it converts a vague irritation into a figure that can be weighed against the effort of fixing the cause, which is usually the argument that gets the work scheduled.

Both are worth putting next to the pass rate on whatever page the team already looks at. The second in particular changes conversations: “the suite is a bit flaky” is easy to defer, and “instability cost eleven engineer-hours last month” is not.

Individually rare, collectively frequent A comparison between per-test and suite-level instability. Twenty tests are shown, each failing approximately one run in fifty, which individually looks stable enough that none would be investigated. The combined probability that at least one of them fails in a given run is calculated at roughly one in three, so the suite as a whole appears unreliable while no single test does. The conclusion drawn is that the suite-level first-attempt pass rate is the number describing what an engineer actually experiences, and it degrades well before any individual test's rate looks bad enough to prompt action. Twenty tests, each failing ~1 run in 50 individually: 98% pass rate each — none looks worth investigating collectively: the suite passes first-attempt ~67% of runs roughly one run in three needs a retry, and nobody can say which test to blame the number to watch suite first-attempt pass rate Per-test rates stay comfortable while the suite becomes unusable, which is why per-test thresholds never trigger the work. The suite-level rate is what an engineer experiences, and it is the one that justifies fixing anything. It is also trivially derived from the same attempt records the per-test rates come from.

Failure modes and edge cases

  1. Retries enabled by default. Every spatial cause is concealed and the pass rate becomes meaningless. If retries are used at all, record the attempts and alert on the first-attempt rate.
  2. Randomised test order without a recorded seed. Order-dependent failures become unreproducible. Randomise deliberately, print the seed, and make it settable.
  3. Quarantine implemented as skip. A skipped test records nothing, so the cause is never diagnosed and the test is silently dead.
  4. Judging stability from a serial run. Most spatial flakiness only appears under parallelism. A stability check that never runs with workers proves very little.
  5. Attributing engine flakiness to timing. A failure that correlates with an image digest is deterministic; treating it as timing produces a retry policy that fails one run in three forever.
  6. No expiry on the quarantine. The list becomes the place tests go to be forgotten, and the suite’s coverage declines invisibly.

Conclusion

Flakiness in a spatial suite is usually deterministic behaviour observed through a non-deterministic lens — an ordering, an engine, a shared resource — and a retry converts it from a diagnosable bug into a permanent tax. Recording every attempt with its environment fingerprint, triaging through the three spatial causes before reaching for timing, and quarantining with an owner and an expiry keeps the suite’s verdict meaningful, which is what spatial test observability and metrics is for.