Shapely vs PostGIS for In-Pipeline Topology Checks

8 min read

A topology rule — no self-intersections, no gaps, no overlaps — can be enforced in-process with Shapely or pushed to PostGIS, and the right choice depends on where the data lives, how large it is, and whether the same rule must also gate ingestion. This comparison sits beneath choosing spatial testing tools and works through the decision on the axes that decide it in practice: memory footprint, whether the check must also run as a database constraint, cross-engine agreement between GEOS builds, and raw throughput. Both call into GEOS for the actual predicates, so the question is rarely correctness of a single check — it is where the check belongs in the pipeline.

The root difference: where the data already is

Shapely operates on geometries already in Python memory, so an in-process check is free of a database round-trip but bounded by RAM — loading a million-row layer to validate it can exhaust the runner. PostGIS operates where the data is stored, so a server-side check scales past memory and can double as an ingestion constraint that stops bad geometry from ever landing, at the cost of needing a database in the test environment. Because both delegate to GEOS, a rule expressed in each should agree — but only if the two GEOS builds match, which is exactly the pinning problem the containerized runtimes work addresses.

The data-movement cost that decides it

The comparison is often argued on library ergonomics, and it is almost always settled by data movement. A topology rule evaluated where the data already sits costs a query. The same rule evaluated elsewhere costs a serialisation, a transfer, a deserialisation, the query, and then the same three steps back if anything is written — and at layer scale those dominate everything the predicate itself does.

Evaluating a rule where the data is, versus moving the data to it Two paths from the same PostGIS table to the same verdict. The upper path issues one indexed query and returns a small result set, with a single step. The lower path serialises every row to well-known binary, transfers it across the connection, deserialises it into Shapely geometry objects, runs the identical predicate, and must reverse the serialisation, transfer and deserialisation to persist any result. Each of the extra steps is marked as scaling with the number of features, while the predicate evaluation itself is the same work in both paths. PostGIS table indexed, in place In-database one indexed query predicate runs beside the data small result set In-process serialise WKB per row transfer over the wire deserialise to Shapely same predicate same work three of these four steps scale with feature count — the predicate does not Reverse all three again if anything must be written back. That round trip, not the library, is what the choice is really about.

The corollary is that the argument flips entirely when the data is not already in the database. A frame produced two transformation steps earlier, held in memory, gains nothing from a round trip into PostGIS for a validity check — and loading it to run a query is the same cost in the other direction. Where the data already sits is not a detail of the comparison; it is the comparison.

Comparison reference

Axis Shapely (in-process) PostGIS (server-side)
Data location Already in GeoDataFrames Already in the database
Memory bound Limited by runner RAM Scales past memory
Doubles as ingestion gate No Yes (constraint / trigger)
Round-trip cost None Query per check
Engine GEOS via Shapely GEOS via PostGIS
Cross-engine risk — Divergence if GEOS builds differ
Best fixture size Fits in memory Large / production-scale

Where Shapely wins

When the test already holds the data as a GeoDataFrame and the fixture fits in memory, in-process Shapely is simplest and fastest — no database to stand up, no round-trip, and the failing geometry is right there to serialize into an artifact. It is the natural fit for the fast pre-merge tier.

# Shapely / GeoPandas: in-memory topology validation
import geopandas as gpd
from shapely import is_valid

gdf = gpd.read_file("tests/fixtures/parcels.gpkg")
invalid = gdf[~gdf.geometry.map(is_valid)]
assert invalid.empty, f"{len(invalid)} invalid geometries"

# no-overlap check via a self-spatial-join on the in-memory frame
pairs = gpd.sjoin(gdf, gdf, predicate="overlaps")
assert pairs.empty, "overlapping parcels detected"

Where PostGIS wins

When the data lives in PostGIS, is too large to load, or the identical rule must gate ingestion so invalid geometry cannot persist, the server-side check wins. Pushing the predicate to the database avoids moving millions of rows into Python and lets the same SQL run both as a test and as a production constraint — the enforcing no-gaps/no-overlaps with PostGIS pattern shows the ingestion-gate form.

-- PostGIS: validity + overlap gate that also scales past memory
SELECT id FROM parcels WHERE NOT ST_IsValid(geom);

SELECT a.id, b.id
FROM parcels a
JOIN parcels b ON a.id < b.id AND ST_Overlaps(a.geom, b.geom);

Drive it from the same suite with psycopg2 so database and in-process checks share one test run:

import psycopg2
with psycopg2.connect(dsn) as conn, conn.cursor() as cur:
    cur.execute("SELECT count(*) FROM parcels WHERE NOT ST_IsValid(geom);")
    (bad,) = cur.fetchone()
    assert bad == 0, f"{bad} invalid geometries in DB"

Keeping both consistent

The one real hazard in mixing the two is cross-engine disagreement: PostGIS and your local Shapely wheel can link different GEOS builds and return different results at the boundary of a predicate. Pin both to the same GEOS version and add a cross-engine test that runs a known edge case through each and asserts they agree, so a divergence surfaces in CI rather than as a “passes locally, fails in the database” report.

Where the two engines can legitimately disagree

Both call into GEOS, so the instinct is that they must agree — and for the overwhelming majority of inputs they do. The exceptions are narrow, predictable, and worth knowing, because a team that does not know them treats a genuine difference as a flaky test.

Boundary-touching cases. Predicates that hinge on whether shared boundaries count are the most common source of a legitimate difference, particularly when one side has snapped coordinates and the other has not. Two polygons that touch exactly agree everywhere; two that touch within a few ULPs may not.

Precision models. PostGIS can carry a precision setting on a geometry column, and a value inserted through it is snapped on write. The same geometry held in memory is not. Comparing the in-memory original against the stored copy then compares two slightly different geometries, and the difference is the column doing what it was configured to do.

Engine versions. The database’s GEOS and the Python process’s GEOS are two separate builds and can differ by a minor version. Nothing guarantees identical behaviour at the edges across that gap, which is why both belong in the structured log.

Empty and null handling. SQL distinguishes an empty geometry from a null row; Python code frequently collapses both into a falsy value. A rule that treats “no geometry” as a case will diverge across the two unless the distinction is preserved deliberately.

One fixture, two engines, one comparator A shared fixture branches to two implementations of the same topology rule: an in-process Shapely version and a PostGIS version. Both verdicts feed a comparator. Agreement marks the rule as genuinely portable across engines. Disagreement is routed to an investigation panel listing the four known legitimate causes: boundary-touching predicates, a precision model configured on the database column, differing GEOS builds between the database and the Python process, and empty geometry versus null row handling. A closing note states that a disagreement is a defect in the rule and never noise to be retried away. one fixture seeded, versioned Shapely rule in-process PostGIS rule server-side comparator verdicts must match agree rule is portable disagree — investigate 1 · boundary-touching case 2 · column precision model 3 · differing GEOS builds 4 · empty versus null A disagreement is a defect in the rule — never noise to be retried away. The four causes above are the entire list worth checking first.

The consistency test is cheap enough to justify wherever a rule genuinely exists in both places, and it has a second benefit beyond catching divergence: it documents that the duplication was deliberate. A rule implemented twice without one is indistinguishable from a rule someone forgot to delete.

Two questions, four answers A two-by-two grid crossing where the data lives against what the rule spans. Data already in PostGIS with a rule spanning multiple features gives a database check, because the index makes it tractable. Data in PostGIS with a per-feature rule gives either location, decided on cost. An in-memory frame with a multi-feature rule gives an in-process spatial join up to fixture scale, with a database becoming correct beyond it. An in-memory frame with a per-feature rule gives an in-process check without qualification. RULE SPANS MANY FEATURES RULE IS PER-FEATURE Data in PostGIS Data in a frame database the index makes it tractable either — decide on cost no round trip either way at fixture scale in-process join up to fixture scale; database beyond it in-process no qualification needed

Failure modes and edge cases

  1. Loading a huge layer into Shapely. Reading a production-scale table into a GeoDataFrame to check validity OOMs the runner; push the check to PostGIS.
  2. GEOS build mismatch. Different GEOS behind PostGIS and Shapely diverge on boundary-touching predicates; pin both and cross-check.
  3. In-process check that should gate ingestion. Validating only in Python leaves the database ingestion path unguarded, so bad geometry still lands from other writers.
  4. sjoin self-overlap O(n²) blowup. A self-spatial-join without an index degrades badly on large in-memory frames; ensure the frame is spatially indexed or move the join to a GiST-backed query — see R-tree vs GiST.
  5. Snap-to-grid slivers. Both engines can emit sub-centimetre overlap fragments after union/difference; guard overlap assertions with an area floor so artifacts do not read as defects.

Conclusion

Shapely and PostGIS both delegate topology to GEOS, so the decision is about placement, not correctness: in-process Shapely when the data is in memory and the fixture is small, PostGIS when the data lives in the database, is too large to load, or the rule must also gate ingestion. Pin GEOS on both sides and cross-check the boundary cases, and the two stay consistent. For the wider tool-selection framework, return to choosing spatial testing tools.

The decision is rarely permanent. A rule that starts in-process because the data was in a frame moves naturally into the database the first time the pipeline persists earlier — and the move is small if the threshold was already in configuration and the assertion was written against a property rather than against a specific engine’s output format.

Where a pipeline genuinely spans both — reading from a database, transforming in memory, writing back — put each rule at the point where its data is already materialised, and resist the urge to standardise on one side for consistency’s sake. Consistency of placement rules is what matters; consistency of tooling costs round trips.

Record which engine a rule ran against in the structured log, so a later disagreement can be attributed without re-running either side.

Neither engine is the default; the data’s location is, and that changes as a pipeline evolves.