Parallelizing Spatial Tests with pytest-xdist

3 min read

Spatial suites are slow because geometry operations are CPU-bound, and the cheapest way to reclaim wall-clock is to run them across cores with pytest-xdist. This guide sits beneath async execution for large datasets and shows how to parallelize a spatial suite without introducing the flakiness that naïve parallelism causes: worker-safe fixtures, partitioning a large geometry collection so each worker validates a disjoint slice, avoiding double-threading between xdist and GEOS, and keeping results deterministic when the order tests complete is no longer fixed. The distinction from async execution is that xdist parallelizes across processes for CPU-bound geometry work, where asyncio parallelizes I/O within one process — they solve different halves of the “large dataset” problem.

Why naïve parallelism makes spatial tests flaky

Three things break when you add -n auto to a spatial suite that was written serially. Shared mutable fixtures — a single database, a temp file, a cached GeoDataFrame — become race conditions when multiple workers touch them at once. Result ordering stops being stable, so any assertion that relied on the incidental order of a spatial join now fails intermittently. And CPU oversubscription appears when xdist launches one worker per core while GEOS or a NumPy backend also spawns threads, so the machine thrashes and the suite gets slower, not faster. Fixing all three is what makes parallelism a speedup rather than a new source of flakes.

Configuration reference

Step-by-step implementation

The pattern targets pytest 7+, pytest-xdist 3+, GeoPandas 0.14+, and keeps every worker isolated.

Step 1 — Isolate shared resources by worker

The worker_id fixture that xdist injects is the key to isolation: give each worker its own database schema and temp directory so nothing is shared.

# conftest.py
import pytest

@pytest.fixture(scope="session")
def db_schema(worker_id):
    # worker_id is "master" when serial, "gw0"/"gw1"/... under xdist
    return f"test_{worker_id}"

@pytest.fixture
def work_dir(tmp_path_factory, worker_id):
    return tmp_path_factory.mktemp(f"geo_{worker_id}")

Step 2 — Partition the geometry collection

For a large validation over one collection, slice it deterministically by worker so each row is validated exactly once and no work is duplicated.

import os, geopandas as gpd

def worker_slice(gdf: gpd.GeoDataFrame, worker_id: str) -> gpd.GeoDataFrame:
    n = int(os.environ.get("PYTEST_XDIST_WORKER_COUNT", "1"))
    if worker_id == "master" or n <= 1:
        return gdf
    k = int(worker_id.removeprefix("gw"))
    return gdf.iloc[k::n]          # every n-th row, offset by worker index

Step 3 — Prevent thread oversubscription

Pin the geometry engine to one thread per worker so xdist’s process parallelism and GEOS/NumPy thread parallelism do not multiply.

OMP_NUM_THREADS=1 pytest -n auto -m "not slow"

Step 4 — Keep assertions order-independent

Because tests now finish in nondeterministic order, any assertion over a joined or grouped result must impose an explicit sort first — the same determinism rule the spatial assertion types work insists on.

def test_join_is_deterministic(points, zones):
    joined = gpd.sjoin(points, zones, predicate="within").sort_index()
    assert list(joined["index_right"]) == expected

Verification pattern

Confirm the speedup is real and the suite is still deterministic by running it serially and in parallel and comparing outcomes — same pass count, lower wall-clock.

pytest -q -m "not slow"                 # serial baseline
OMP_NUM_THREADS=1 pytest -q -n auto -m "not slow"   # parallel: same result, faster

If the parallel run reports different failures than the serial run, an unisolated fixture or an order-dependent assertion is leaking — fix it before trusting the parallel gate.

Failure modes and edge cases

1. Shared database schema. Two workers writing the same schema corrupt each other’s fixtures; derive the schema name from worker_id.
2. Thread oversubscription. -n auto plus multi-threaded GEOS launches cores² threads and slows down; set OMP_NUM_THREADS=1.
3. Order-dependent assertions. A join whose row order was incidentally stable serially flakes under xdist; sort explicitly before asserting.
4. Uneven partitions. Slicing a collection where geometry complexity is clustered (all large polygons in one slice) unbalances workers; interleave with a stride rather than contiguous chunks.
5. Session fixtures rebuilt per worker. A scope="session" fixture runs once per worker, so an expensive build multiplies; cache it to disk keyed by content hash, or accept the per-worker cost.

Conclusion

pytest-xdist turns a slow, CPU-bound spatial suite into a parallel one, but only after isolating shared resources by worker_id, partitioning the geometry deterministically, pinning engine threads to avoid oversubscription, and making every assertion order-independent. Done that way, the suite runs several times faster with identical results. For the complementary I/O-parallel approach, return to async execution for large datasets.

Related

• Running Async Spatial Tests with pytest-asyncio
• How to Structure pytest-geo for Large Shapefiles
• Parent: Async Execution for Large Datasets