How to Structure pytest-geo for Large Shapefiles

7 min read

When validating multi-gigabyte .shp datasets in automated pipelines, a naive pytest configuration routinely triggers memory exhaustion, I/O contention, and non-deterministic timeout failures. The specific problem this guide solves is structural: how to load, index, and assert against a shapefile that does not fit comfortably in a CI runner’s RAM, using fiona, pyogrio, and Shapely 2.x inside session-scoped fixtures. It is a practitioner-level expansion of the base tier described in Understanding the GIS Test Pyramid — the layer where fast, deterministic checks must run before any heavy integration job is allowed to start. The objective is not merely to run spatial assertions, but to architect a harness that respects filesystem constraints, enforces strict validation boundaries, and scales across parallel runners without re-reading the dataset on every test.

Root Cause: Why Large Shapefiles Break a Default pytest Setup

A shapefile is not one file but a bundle (.shp geometry, .shx index, .dbf attributes, .prj CRS). The failure modes that appear at scale are all consequences of how that bundle is read, not of the assertions themselves:

1. Per-function full reads. A fixture that calls geopandas.read_file() with default (function) scope deserialises every geometry into a GeoDataFrame once per test. For a 500 MB dataset across 40 tests, that is 40 full materialisations and 40 GEOS geometry trees — linear memory and I/O blow-up.
2. Eager geometry construction. Building Shapely geometries for every feature converts compact WKB into Python objects with significant per-object overhead. The working set is far larger than the file on disk.
3. Shared file handles under parallelism. Once you add pytest-xdist, multiple workers opening the same .shp/.shx pair race on OS-level file descriptors and any on-disk spatial index, producing flaky, order-dependent failures.

The peak resident memory of an eager load is approximately

where $N$ is the feature count, $v_i$ the vertex count of feature $i$, and $c$ the per-coordinate object overhead. A bounding-envelope index instead stores four floats per feature, giving

which is independent of geometry complexity. The architectural goal is to push the suite from $M_{\text{eager}}$ toward $M_{\text{index}}$ and to read full geometries only in bounded windows.

Architectural Blueprint and Directory Layout

A scalable spatial test repository must isolate heavy binaries from version control, enforce deterministic fixture lifecycles, and align with the conventions in Geospatial QA Fundamentals & Architecture. The following layout decouples test logic from raw data while keeping reproducibility strict:

project-root/
├── conftest.py                 # Root-level fixtures, session-scoped setup
├── pyproject.toml              # pytest config, xdist, timeout, markers
├── tests/
│   ├── geo/
│   │   ├── conftest.py         # Spatial-specific fixtures, lazy loaders
│   │   ├── test_topology.py
│   │   ├── test_schema.py
│   │   └── test_crs_alignment.py
│   └── unit/
│       └── test_transforms.py
├── data/                       # .gitignored, mounted via CI artifact/cache
│   └── large_dataset/
│       ├── boundaries.shp
│       ├── boundaries.shx
│       ├── boundaries.dbf
│       └── boundaries.prj
└── ci/
    └── spatial_cache_policy.yaml

Large shapefiles should never be committed to Git; they are provisioned via CI artifact storage, cloud object mounts, or deterministic synthetic generators that produce statistically representative geometries. The tests/geo/conftest.py layer becomes the single source of truth for spatial fixture injection, while the root conftest.py handles session-level resource pooling and teardown.

Parameter Reference: Windowed Reads and Fixture Scopes

These are the function signatures and flags that control how much of the file enters memory. Pin the libraries in your test container so behaviour is reproducible across local and CI runs.

The window size $w$ in a pyogrio read bounds peak memory at $M \approx O(w)$ rather than $O(N)$: choosing $w$ so that a single window comfortably fits the runner’s RAM is what makes the suite stable regardless of total dataset size.

Step-by-Step Implementation

Step 1 — Resolve the dataset path once (session scope)

Pin the artifact location in a single session fixture so no test hard-codes a path. With pytest 7+, session scope guarantees the resolution runs once per run.

# tests/geo/conftest.py
import pytest
from pathlib import Path

@pytest.fixture(scope="session")
def large_shapefile_path():
    """Resolve path to the large shapefile from the CI artifact cache."""
    path = Path("/opt/ci/data/large_dataset/boundaries.shp")
    if not path.exists():
        pytest.skip("large dataset not mounted; run with the integration cache")
    return path

Step 2 — Build a bounding-envelope index without materialising geometries

Stream features with fiona 1.9+ and hand the geometries to a Shapely 2.x STRtree. The tree stores only envelopes, so resident memory follows $M_{\text{index}}$, not $M_{\text{eager}}$.

# tests/geo/conftest.py
import fiona
from shapely.geometry import shape
from shapely import STRtree

@pytest.fixture(scope="session")
def spatial_index(large_shapefile_path):
    """
    Build a Shapely STRtree over feature envelopes, once per session.
    Returns (tree, geoms, props) so query results map back to attributes.
    """
    geoms, props = [], []
    with fiona.open(large_shapefile_path) as src:
        for feat in src:                       # streaming: one record at a time
            geoms.append(shape(feat["geometry"]))
            props.append(dict(feat["properties"]))
    tree = STRtree(geoms)                       # indexes bounding envelopes only
    return tree, geoms, props

Step 3 — Read full geometries only in bounded windows

When a test genuinely needs a GeoDataFrame, use pyogrio 0.7+ windowed reads instead of loading the whole file. Each window is a copy-button-ready, self-contained read.

# tests/geo/conftest.py
import pyogrio

@pytest.fixture(scope="session")
def feature_windows(large_shapefile_path):
    """Yield bounded GeoDataFrame windows so peak memory stays O(window)."""
    info = pyogrio.read_info(large_shapefile_path)   # header only
    total, window = info["features"], 10_000

    def _windows():
        for start in range(0, total, window):
            yield pyogrio.read_dataframe(
                large_shapefile_path,
                skip_features=start,
                max_features=window,
            )
    return _windows

Step 4 — Write assertions against the session-scoped index

Topology checks run against the index built in Step 2, so the file is read once for the whole module. This is the base-tier work that must pass before the heavier integration layer of the GIS test pyramid runs — and it mirrors the predicates used when validating polygon topology with GeoPandas.

# tests/geo/test_topology.py
from shapely.validation import explain_validity

def test_polygon_validity(spatial_index):
    """Every feature in the session index must be a valid geometry."""
    _tree, geoms, props = spatial_index
    invalid = [
        (props[i].get("id", i), explain_validity(g))
        for i, g in enumerate(geoms)
        if not g.is_valid
    ]
    assert not invalid, f"Invalid geometries detected: {invalid[:5]}"

Step 5 — Configure isolation and timeouts for parallel runs

Parallelising introduces race conditions on shared file handles. Use pytest-xdist with --dist loadscope so session-scoped fixtures are reused per worker, and enforce a hard timeout so a hung GDAL read fails deterministically rather than wedging the runner. Calibrate any geometric thresholds with spatial tolerance thresholds before adding assertions here.

# pyproject.toml
[tool.pytest.ini_options]
addopts = "-n auto --dist loadscope --timeout=300 --strict-markers"
markers = [
    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
    "integration: requires the full dataset cache",
]

Verification Pattern

Confirm the structure works before trusting it in CI. Run the geo suite under xdist and assert that memory stays bounded by selecting only the fast index-backed tests first:

# Fast base-tier gate: index build + topology, parallel, hard-timed
pytest tests/geo -m "not slow" -n auto --dist loadscope --timeout=300 -q

A quick in-test guard verifies the index never silently degrades into an eager full load — the envelope index should be far smaller than the source file:

# tests/geo/test_memory_budget.py
def test_index_is_envelope_only(spatial_index, large_shapefile_path):
    tree, geoms, _ = spatial_index
    assert len(tree.geometries) == len(geoms)        # STRtree built once
    # 4 floats * 8 bytes per feature is the envelope budget; assert we are
    # nowhere near the on-disk size, i.e. geometries were not duplicated.
    assert large_shapefile_path.stat().st_size > 4 * 8 * len(geoms)

Failure Modes and Edge Cases

Large-shapefile suites fail in spatial-specific ways that a generic memory profile will not reveal:

1. Anti-meridian features. Polygons crossing ±180° longitude produce envelopes that span nearly the whole globe, making the STRtree return almost every candidate on a bounding-box query. Detect these (bounds width near 360°) and split them before indexing, or the base tier silently becomes $O(N^2)$.
2. .shx index corruption or absence. A missing or stale .shx forces GDAL into a full sequential scan, defeating windowed reads. Validate the bundle’s file signatures before ingestion and regenerate the index in a fixture if needed.
3. Mixed Z/M coordinates. A shapefile declared 2D but carrying Z or M values yields geometries whose has_z is inconsistent across features; equality and area assertions then diverge between fiona records and pyogrio frames. Normalise to a single dimensionality in the loader.
4. Empty or null geometries in the .dbf. Features with attributes but no geometry surface as None from fiona; calling shape(None) raises mid-index-build. Filter and count them explicitly rather than letting the fixture crash.
5. CRS unit mismatch. A .prj in degrees with thresholds written in metres makes every tolerance check meaningless. Normalise to a canonical EPSG code in the loader, the same discipline enforced when automating CRS validation in CI pipelines.

By combining lazy fixture loading, a session-scoped envelope index, windowed pyogrio reads, and hardened xdist isolation, teams can validate enterprise-scale shapefiles without compromising pipeline velocity or runner stability. For where this base-tier harness sits relative to integration and system checks, return to Understanding the GIS Test Pyramid.

Related

• Setting up spatial tolerance thresholds in assertions
• Running async spatial tests with pytest-asyncio
• Validating polygon topology with GeoPandas
• Parent: Understanding the GIS Test Pyramid