Parallelizing Spatial Tests with pytest-xdist

9 min read

Spatial suites are slow because geometry operations are CPU-bound, and the cheapest way to reclaim wall-clock is to run them across cores with pytest-xdist. This guide sits beneath async execution for large datasets and shows how to parallelize a spatial suite without introducing the flakiness that naïve parallelism causes: worker-safe fixtures, partitioning a large geometry collection so each worker validates a disjoint slice, avoiding double-threading between xdist and GEOS, and keeping results deterministic when the order tests complete is no longer fixed. The distinction from async execution is that xdist parallelizes across processes for CPU-bound geometry work, where asyncio parallelizes I/O within one process — they solve different halves of the “large dataset” problem.

Why naïve parallelism makes spatial tests flaky

Three things break when you add -n auto to a spatial suite that was written serially. Shared mutable fixtures — a single database, a temp file, a cached GeoDataFrame — become race conditions when multiple workers touch them at once. Result ordering stops being stable, so any assertion that relied on the incidental order of a spatial join now fails intermittently. And CPU oversubscription appears when xdist launches one worker per core while GEOS or a NumPy backend also spawns threads, so the machine thrashes and the suite gets slower, not faster. Fixing all three is what makes parallelism a speedup rather than a new source of flakes.

The three resources that collide first

Parallel spatial tests do not fail randomly — they fail on the same three shared resources, in the same order, and each has a different remedy. Recognising which one you are hitting from the symptom saves an afternoon of bisecting worker counts.

Shared schema, shared temp path, oversubscribed threads Three cards. The first, shared database schema, has the symptom that tests observe rows created by other workers so failures move between workers between runs; the remedy is a schema or database named for the worker identifier. The second, shared temporary output path, has the symptom of truncated or interleaved files and driver write errors; the remedy is a per-worker temporary directory. The third, thread oversubscription, explains that each worker's numerical libraries default to one thread per core, so eight workers on an eight-core machine request sixty-four threads; the symptom is throughput falling as workers are added, and the remedy is to set the thread-count environment variables to one before workers are spawned. A footer notes that the first two produce wrong results while the third only produces slow ones, so the third is the easiest to misread. 1 · Shared schema SYMPTOM workers see each other’s rows the failing test moves each run REMEDY one schema per worker, named from the worker id gives wrong results 2 · Shared temp path SYMPTOM truncated or interleaved files driver errors on write REMEDY tmp_path per worker; never a fixed filename gives wrong results 3 · Thread oversubscription SYMPTOM 8 workers × 8 threads = 64 throughput falls as -n rises REMEDY pin thread env vars to 1 before workers spawn gives slow results The first two corrupt correctness; the third only costs time — which is why it survives longest, quietly cancelling the speed-up parallelism was added for.

The third resource deserves emphasis because it is the one that hides. Every worker process independently initialises NumPy, and through it whatever BLAS implementation is installed, which by default sizes its thread pool to the machine’s core count. On an eight-core runner with -n 8 that is sixty-four threads contending for eight cores, and the result is a suite that gets slower with more workers while every test still passes. Setting the thread-count environment variables to one before the workers spawn — in the process that launches pytest, not in a fixture, because the libraries read them at import time — restores linear scaling.

Configuration reference

Concern Serial default xdist-safe form
Worker count 1 -n auto (or pinned -n 4)
Shared DB one schema per-worker schema via worker_id
Temp files fixed path tmp_path_factory per worker
Geometry partition whole set slice by worker_id hash
GEOS threads implicit OMP_NUM_THREADS=1 to avoid oversubscription
Result order insertion explicit sort before assert

Step-by-step implementation

The pattern targets pytest 7+, pytest-xdist 3+, GeoPandas 0.14+, and keeps every worker isolated.

Step 1 — Isolate shared resources by worker

The worker_id fixture that xdist injects is the key to isolation: give each worker its own database schema and temp directory so nothing is shared.

# conftest.py
import pytest

@pytest.fixture(scope="session")
def db_schema(worker_id):
    # worker_id is "master" when serial, "gw0"/"gw1"/... under xdist
    return f"test_{worker_id}"

@pytest.fixture
def work_dir(tmp_path_factory, worker_id):
    return tmp_path_factory.mktemp(f"geo_{worker_id}")

Step 2 — Partition the geometry collection

For a large validation over one collection, slice it deterministically by worker so each row is validated exactly once and no work is duplicated.

import os, geopandas as gpd

def worker_slice(gdf: gpd.GeoDataFrame, worker_id: str) -> gpd.GeoDataFrame:
    n = int(os.environ.get("PYTEST_XDIST_WORKER_COUNT", "1"))
    if worker_id == "master" or n <= 1:
        return gdf
    k = int(worker_id.removeprefix("gw"))
    return gdf.iloc[k::n]          # every n-th row, offset by worker index

Step 3 — Prevent thread oversubscription

Pin the geometry engine to one thread per worker so xdist’s process parallelism and GEOS/NumPy thread parallelism do not multiply.

OMP_NUM_THREADS=1 pytest -n auto -m "not slow"

Step 4 — Keep assertions order-independent

Because tests now finish in nondeterministic order, any assertion over a joined or grouped result must impose an explicit sort first — the same determinism rule the spatial assertion types work insists on.

def test_join_is_deterministic(points, zones):
    joined = gpd.sjoin(points, zones, predicate="within").sort_index()
    assert list(joined["index_right"]) == expected

Choosing a distribution mode for spatial work

pytest-xdist offers several ways to hand tests to workers, and the default is a poor fit for spatial suites. The reason is fixture cost: building a spatial index, loading a projection grid, or standing up a database is expensive per worker, and a scheduler that scatters related tests across all workers forces every worker to pay that cost.

Mode How tests are assigned Fixture cost Right for spatial suites when
--dist load Next free worker takes the next test Every worker builds every session fixture Tests are independent and fixtures are cheap
--dist loadfile All tests in a file go to one worker Module fixtures built once per file The usual best default — fixtures are per-module
--dist loadscope All tests in a class or module scope go together Scope fixtures built once Fixtures are class-scoped and heavy
--dist loadgroup Tests marked into the same group go together You control the grouping explicitly A shared resource spans several files
Fixture cost under load versus loadfile distribution Two panels. In the load panel, four tests from one file are shown assigned to four different workers, and each worker carries its own copy of the expensive spatial index fixture, giving four index builds. In the loadfile panel, the same four tests are all assigned to a single worker which builds the index once, while other workers handle other files. A caption records that the gap widens as the worker count rises, so the default scheduler degrades precisely on the larger machines where parallelism was supposed to help. --dist load — index built 4× worker 0 worker 1 worker 2 worker 3 index build index build index build index build test_a test_b test_c test_d fixture cost = 4 × base --dist loadfile — index built 1× worker 0 worker 1 worker 2 worker 3 index build test_a b c d other file other file other file fixture cost = 1 × base per file The gap widens with the worker count, so the default scheduler degrades exactly on the larger machines where parallelism was meant to pay off. Rule of thumb: if a session or module fixture takes longer than a test, the scheduler — not the worker count — is what limits the suite. Measure it directly with --durations and compare the fixture setup line against the slowest test before adding workers.

There is a second, subtler reason to prefer file-level distribution in spatial suites: determinism of failure attribution. Under the default scheduler, which worker runs which test changes between runs depending on timing, so a test that fails because of a leaked resource fails in a different place each time and looks like flakiness. Pinning tests to workers by file makes the assignment stable, which turns an intermittent mystery into a reproducible failure — and reproducibility is worth more than the last few per cent of load balance.

Verification pattern

Confirm the speedup is real and the suite is still deterministic by running it serially and in parallel and comparing outcomes — same pass count, lower wall-clock.

pytest -q -m "not slow"                 # serial baseline
OMP_NUM_THREADS=1 pytest -q -n auto -m "not slow"   # parallel: same result, faster

If the parallel run reports different failures than the serial run, an unisolated fixture or an order-dependent assertion is leaking — fix it before trusting the parallel gate.

Knowing when to stop adding workers

Parallelism has a ceiling set by whichever resource saturates first, and adding workers past it makes the suite slower rather than faster. For spatial suites that resource is almost never the CPU — it is memory, because each worker holds its own copy of the geometry it is testing.

Wall-clock against worker count, with the memory ceiling A curve of total suite duration versus number of xdist workers. Duration drops sharply from one to four workers as work is spread out, flattens between four and six as the fixed fixture cost begins to dominate, and then climbs from seven workers onward as combined worker memory exceeds the container limit and the runner starts swapping. A vertical dashed line marks the memory ceiling. An annotation identifies the flat region as the useful operating point and warns that choosing the worker count from the core count alone lands to the right of it. time workers 1 2 4 5 6 7 8 memory ceiling workers start swapping useful range — the flat part Pick the worker count from measured memory per worker and the container limit, never from the core count — for geometry-heavy suites the two rarely agree.

The practical procedure is short. Measure peak resident memory for a single worker running the heaviest module, divide the container limit by that figure, subtract one for the controlling process, and use the result as the cap. Then confirm empirically: run the suite at that count and at one above it, and keep the lower number if the times are close. A suite pinned just below its memory ceiling is stable; one pinned at the core count on a memory-constrained runner produces intermittent worker crashes that read as flaky tests and get retried rather than diagnosed.

Failure modes and edge cases

  1. Shared database schema. Two workers writing the same schema corrupt each other’s fixtures; derive the schema name from worker_id.
  2. Thread oversubscription. -n auto plus multi-threaded GEOS launches cores² threads and slows down; set OMP_NUM_THREADS=1.
  3. Order-dependent assertions. A join whose row order was incidentally stable serially flakes under xdist; sort explicitly before asserting.
  4. Uneven partitions. Slicing a collection where geometry complexity is clustered (all large polygons in one slice) unbalances workers; interleave with a stride rather than contiguous chunks.
  5. Session fixtures rebuilt per worker. A scope="session" fixture runs once per worker, so an expensive build multiplies; cache it to disk keyed by content hash, or accept the per-worker cost.

One habit prevents most of the trouble above: run the suite serially in CI at least once a day, on a schedule, alongside the parallel run. A divergence between the two results is the cleanest possible signal that a test depends on isolation it is not declaring, and it localises the problem to the parallel machinery instead of leaving an engineer to guess whether the data changed. When the serial run is green and the parallel run is red, the fault is never in the assertion.

Conclusion

pytest-xdist turns a slow, CPU-bound spatial suite into a parallel one, but only after isolating shared resources by worker_id, partitioning the geometry deterministically, pinning engine threads to avoid oversubscription, and making every assertion order-independent. Done that way, the suite runs several times faster with identical results. For the complementary I/O-parallel approach, return to async execution for large datasets.