Setting Up Spatial Tolerance Thresholds in Assertions

9 min read

Deterministic equality (==, geom_a.equals(geom_b) with no slack) is the wrong default for geospatial validation: floating-point coordinate storage, projection transforms, and topology-preserving algorithms all perturb vertices below the threshold of cartographic significance. A tolerance threshold is the explicit distance, area, or relative-error bound under which two geometries are treated as equal. This page sits beneath Spatial Assertion Types Explained and shows exactly how to choose, configure, and verify those thresholds for Shapely 2.x and PostGIS so that your suite catches real regressions instead of phantom drift. Get this wrong and you ship flaky CI: tests that pass on your laptop and fail on a runner with a different GEOS build.

Why naive equality fails at the engineering level

Three independent mechanisms guarantee that bit-exact geometry comparison will eventually fail, even when the data is correct:

  1. Floating-point precision drift. Coordinates are IEEE 754 doubles. Every affine transform, reprojection, or serialization round-trip (GeoJSON → PostGIS → Shapely) accumulates rounding error. A delta of 1e-9 degrees is mathematically non-zero but spatially meaningless — roughly 0.1 mm at the equator. The Python floating-point arithmetic notes explain why 0.1 + 0.2 != 0.3 holds for coordinates too.
  2. Cross-engine algorithmic divergence. GEOS (behind Shapely and GeoPandas) and PostGIS implement predicates such as ST_Equals and ST_Intersects with different snap-to-grid heuristics and default precision models. Comparing a Shapely result against a PostGIS fixture without an explicit tolerance produces results that differ between local dev and CI.
  3. Topology slivers and micro-gaps. Buffering, union/difference, and raster-to-vector conversion routinely emit sub-millimetre artifacts. Strict equality flags these as failures and buries the genuine defects under noise.

The fix is a tolerance strategy that is CRS-aware, scoped per assertion type, and stored as version-controlled configuration rather than scattered magic numbers.

Tolerance is a relative-error bound, not a magic number

For two coordinates the comparison reduces to whether the Euclidean distance falls under a bound. For an absolute bound τ\tau in CRS units:

∥pactual−pexpected∥2≤τ\lVert p_{\text{actual}} - p_{\text{expected}} \rVert_2 \le \tau

For features whose scale varies (large parcels vs. survey points) a relative bound is safer, mirroring math.isclose:

∣a−b∣≤max⁡(τrel⋅max⁡(∣a∣,∣b∣), τabs)\lvert a - b \rvert \le \max\bigl(\tau_{\text{rel}} \cdot \max(\lvert a \rvert, \lvert b \rvert),\ \tau_{\text{abs}}\bigr)

For polygon equality, distance is the wrong metric entirely — use a symmetric-difference area bound, where A△A_\triangle is the area of actual.symmetric_difference(expected):

A△max⁡(Aactual, Aexpected)≤τarea\frac{A_\triangle}{\max(A_{\text{actual}},\ A_{\text{expected}})} \le \tau_{\text{area}}

The critical constraint is units. A geographic CRS such as EPSG:4326 measures distance in degrees; 0.001 there is ~111 m at the equator, not 1 mm. Apply absolute metre thresholds only after projecting to a metric CRS (EPSG:3857 or the appropriate UTM zone), or convert the bound to degrees first.

Mapping thresholds to assertion type

Each assertion family needs a different tolerance shape and range. The ranges below assume a projected, metre-based CRS unless noted.

Assertion type Tolerance strategy Typical range CRS unit note
Coordinate equality Absolute distance (τ\tau) 0.001 – 0.01 m project to metric CRS first
Topological equality (equals_exact) Vertex snap distance 1e-6 – 1e-4 m matches GEOS precision model
Area / overlap Symmetric-difference area ratio 0.1% – 1% of area unit-free ratio, CRS-agnostic
Proximity / buffer containment Relative to buffer radius 0.5% – 2% of radius scales with feature
Network connectivity Vertex snap tolerance 0.01 – 0.1 m aligns dangling endpoints
Choosing a tolerance bound by assertion type, then gating metric bounds on CRS units A top-down flowchart. The actual-versus-expected geometry pair branches by assertion type into point, network, and polygon. Point selects an absolute distance bound and network selects a snap bound; both are metric and pass through a Geographic CRS check that reprojects to a metric CRS when needed. Polygon selects a unit-free symmetric-difference area ratio and skips the CRS check. Every path reaches a within-bound decision that resolves to pass or fail. actual vs expected Assertion type? Point / coordinate Network / topology Polygon / overlap distance ≤ τ snap ≤ τ_snap area ratio ≤ τ_area Geographic CRS? reproject to metric CRS within bound? pass fail point network polygon unit-free yes no · units valid yes no Four tolerance shapes and where each one belongs Four cards. Absolute bound, written as the absolute difference being at most epsilon, applies to coordinate equality and buffer distance, is expressed in CRS units, and breaks down when coordinate magnitudes vary across the dataset. Relative epsilon, written as the absolute difference divided by the expected value, applies to area and length comparisons, is dimensionless, and breaks down as the expected value approaches zero. Snap grid, written as a grid size, applies to topological equality and repair, is expressed in CRS units, and breaks down when the grid is coarser than the smallest genuine feature. Ratio band, written as a lower and upper bound on the ratio of measured to expected, applies to scale-varying polygons, is dimensionless, and breaks down when the reference value can be zero. Absolute bound |measured − expected| ≤ ε use for: coordinate equality · buffer distance unit: CRS units — carries a unit, always breaks when: coordinate magnitudes vary widely Relative epsilon |measured − expected| / |expected| ≤ ε use for: area · length · perimeter deltas unit: dimensionless breaks when: expected value approaches zero Snap grid set_precision(geom, grid_size) use for: topological equality · repair · dedup unit: CRS units breaks when: grid coarser than the smallest feature Ratio band 1 − δ ≤ measured / expected ≤ 1 + δ use for: scale-varying polygons across projections unit: dimensionless breaks when: the reference value can be zero

Step-by-step implementation

The pattern below targets Shapely 2.x, GeoPandas 0.14+, and pytest 7+. Thresholds load from config so CI and local runs share one source of truth.

Step 1 — Externalize the thresholds

Keep tolerances in version-controlled YAML, not inline constants, so a GEOS upgrade is a one-line config change reviewable in a PR.

# spatial_tolerances.yaml
coordinate_abs_m:   0.005      # 5 mm absolute, metric CRS
topology_snap_m:    1.0e-6     # GEOS equals_exact tolerance
area_ratio:         0.001      # 0.1% symmetric-difference area
buffer_rel:         0.01       # 1% of buffer radius

Step 2 — Load config and normalize the CRS

import yaml
from pathlib import Path
from pyproj import CRS

def load_tolerances(path: str = "spatial_tolerances.yaml") -> dict:
    return yaml.safe_load(Path(path).read_text())

def assert_metric_crs(crs: CRS) -> None:
    # Absolute metre thresholds are invalid on geographic (degree) CRS.
    if crs is None or crs.is_geographic:
        raise ValueError(
            "Reproject to a metric CRS (e.g. EPSG:3857 / UTM) "
            "before applying metre-based tolerances."
        )

Step 3 — Coordinate equality with an absolute bound

from shapely.geometry import Point

def assert_coordinate_equality(actual: Point, expected: Point, tol_m: float) -> None:
    """Pass when planar distance is within tol_m (CRS must be metric)."""
    dist = actual.distance(expected)               # Shapely 2.x: Cartesian distance
    assert dist <= tol_m, (
        f"Coordinate drift {dist:.6f} m exceeds tolerance {tol_m} m"
    )

Step 4 — Topological equality with snap tolerance

equals_exact compares vertices within a snap distance; set_precision first collapses sub-grid slivers so orientation and micro-gaps do not cause false negatives.

from shapely import set_precision
from shapely.geometry import Polygon

def assert_topology_equality(actual: Polygon, expected: Polygon, snap_m: float) -> None:
    """GEOS-backed equality after snapping both geometries to a shared grid."""
    a = set_precision(actual.buffer(0), grid_size=snap_m)    # buffer(0) repairs self-intersections
    e = set_precision(expected.buffer(0), grid_size=snap_m)
    assert a.equals_exact(e, tolerance=snap_m), (
        "Topological mismatch exceeds snap tolerance"
    )

Step 5 — Area-ratio equality for scale-varying polygons

def assert_area_equality(actual: Polygon, expected: Polygon, ratio: float) -> None:
    """Unit-free: symmetric-difference area as a fraction of the larger feature."""
    delta = actual.symmetric_difference(expected).area
    denom = max(actual.area, expected.area) or 1.0
    rel = delta / denom
    assert rel <= ratio, f"Area delta {rel:.4%} exceeds {ratio:.2%}"

Step 6 — GeoPandas frames and the PostGIS side

For whole-frame comparisons use geopandas.testing.assert_geodataframe_equal with check_less_precise=True (or pass check_geom_type / check_crs as needed) — it delegates to GEOS equals_exact with a loosened precision. When the expected fixture comes from a database, normalize it server-side first with ST_ReducePrecision so both sides share a grid; see the PostGIS ST_ReducePrecision reference.

-- Align a PostGIS fixture to the same grid your Python snap tolerance uses.
SELECT ST_ReducePrecision(geom, 1e-6) AS geom_norm
FROM   fixtures.parcels
WHERE  parcel_id = :pid;

Calibrating these thresholds is exactly the kind of network-independent logic that benefits from deterministic fixtures — generate them with the patterns in Best Practices for Mocking PostGIS Connections so upstream provider drift never reaches your tolerance tests.

Choosing the number, not inheriting it

A threshold copied from another project is a threshold nobody can defend. Derive it instead from the three quantities that actually bound it: the measurement precision of the source data, the numerical noise the pipeline adds, and the smallest difference a consumer cares about. The usable range sits between the second and the third, and if that range is empty the pipeline needs fixing rather than a looser test.

The defensible band for a tolerance threshold A horizontal logarithmic axis of distance in millimetres. A left region, below the pipeline noise floor, is marked as producing false failures because the test is tighter than the arithmetic. A right region, above the smallest difference the consumer can detect, is marked as letting real defects through. The band between them is labelled defensible. A marker at three millimetres shows the measured noise floor, a marker at fifty millimetres shows the consumer threshold, and a chosen tolerance of ten millimetres sits inside the band. Below, a second short axis shows the pathological case where the noise floor sits to the right of the consumer threshold so that the band is empty, annotated with the instruction to fix the pipeline rather than widen the test. 1 mm 3 mm 10 mm 50 mm false failures defensible band defects pass pipeline noise floor smallest difference a consumer detects chosen: 10 mm Pathological case — the two bounds have crossed: defects pass — above the consumer threshold false failures — below the noise floor No gap between them: no tolerance is defensible. Reduce the noise — pin the transform, snap earlier, stop the float32 downcast — rather than widening the test.

Measure the noise floor rather than guessing it: run the pipeline twice over an unchanged input and record the distribution of per-vertex differences. Whatever the 99th percentile of that distribution is, your threshold has to sit above it or the suite will fail on inputs that never changed. The consumer threshold comes from the other end — the map scale at which the data is rendered, the survey standard it must meet, or the grid size of the tiles it feeds.

Verification pattern

A self-contained pytest module proves the thresholds behave: a sub-tolerance perturbation must pass, and a supra-tolerance one must fail.

import pytest
from shapely.geometry import Point
from shapely.affinity import translate

TOL_M = 0.005

def test_drift_within_tolerance_passes():
    p = Point(500000, 4649776)                  # UTM 32N, metric
    assert_coordinate_equality(translate(p, xoff=0.002), p, TOL_M)

def test_drift_beyond_tolerance_fails():
    p = Point(500000, 4649776)
    with pytest.raises(AssertionError):
        assert_coordinate_equality(translate(p, xoff=0.05), p, TOL_M)

Run it as a fast gate:

pytest -q test_spatial_tolerances.py

Following the layering in Understanding the GIS Test Pyramid, tighten the bound for unit tests (0.001 m), relax it for integration runs (0.01 m), and switch to relative area ratios for end-to-end pipeline checks where upstream transform noise is unavoidable.

Failure modes and edge cases

  1. Geographic CRS with a metre threshold. Applying 0.005 to EPSG:4326 compares ~550 m of slack. assert_metric_crs from Step 2 must guard every absolute-distance assertion, or reproject first.
  2. Anti-meridian wrap. A geometry spanning ±180° longitude produces a near-global bounding box and absurd distances. Densify and reproject to a CRS centred on the feature before measuring, or split at the dateline.
  3. Polar and high-latitude distortion. In EPSG:3857 a 1 m tolerance near 80° latitude represents far less ground distance than at the equator because of scale factor. Use a local UTM or polar stereographic CRS for assertions above ~60°.
  4. Empty and None geometries. Point().distance(other) returns inf and symmetric_difference on an empty geometry silently returns the other geometry — assert non-emptiness before measuring, or empties pass area checks by accident.
  5. Mixed Z/M coordinates. equals_exact compares only X/Y; two geometries equal in plan view but differing in elevation pass a 2D tolerance. Assert dimensionality (has_z) explicitly when Z is load-bearing.

When tolerance scoping touches anonymized or boundary-defining data, pair these rules with Scoping Rules for Map Data Validation so an over-tolerant assertion never silently passes a cadastral or jurisdictional violation.

Conclusion

Calibrated, CRS-aware tolerance thresholds turn spatial assertions from a flaky liability into a deterministic gate: choose the bound that matches the assertion family, store it as reviewable config, and prove both directions in pytest. For the full taxonomy of predicates these thresholds protect, return to Spatial Assertion Types Explained.