Choosing Spatial Testing Tools

13 min read

More than one tool can enforce almost any spatial rule, and choosing badly is how teams end up with slow gates, duplicated checks, or validation that lives in the wrong layer of the stack. This decision framework sits beneath CI/CD spatial quality gates and gives a structured way to pick between the realistic options at the moment you write a gate: a schema-and-domain contract as a pytest assertion or a Great Expectations suite, a topology rule evaluated in-process with Shapely or pushed to PostGIS, and a spatial join in a test backed by an in-memory R-tree or a database GiST index. The right answer is rarely absolute — it depends on where the data already lives, how large the fixtures are, and whether the same rule must also run in production — so this framework is about matching the tool to those constraints rather than declaring a winner.

Decision tree for selecting a spatial validation tool by rule and data location From a root question of what the rule constrains, three branches. The contract branch picks pytest for a few rules or Great Expectations for a documented suite. The relationship branch picks in-process Shapely for in-memory data or PostGIS for database data. The join-performance branch picks an R-tree for in-memory joins or a GiST index for database joins. What does it constrain? fields / domain geometry relationship join performance how many rules? where is the data? where is the join? pytest Great Exp. Shapely PostGIS R-tree GiST few a suite in memory in DB in memory in DB

The three decisions below share one principle: put the check where the data already is, and prefer the tool whose failure output a reviewer can act on. Each decision has a full head-to-head guide — pytest-geo vs Great Expectations, Shapely vs PostGIS for topology, and R-tree vs GiST.

Decision Reference

Decision Choose the first when… Choose the second when…
pytest + Shapely vs Great Expectations A handful of imperative geometry assertions; developer-owned A documented, data-docs-published contract over many columns
In-process Shapely vs PostGIS topology Data is already in GeoDataFrames; fixtures fit in memory Data lives in PostGIS; the same rule must gate ingestion
R-tree vs GiST index The join is in-process against a fixture The join is a database query you are also validating

Contract Rules: pytest with Shapely, or Great Expectations

When the rule is “these fields exist, with these dtypes, in this domain, and this CRS,” both a pytest assertion and a Great Expectations for GIS suite can enforce it. Prefer plain pytest when there are a few rules that live naturally alongside the code and are read by developers. Prefer Great Expectations when the contract spans many columns, must be published as human-readable data docs for non-developers, and benefits from a declarative suite that a data owner can review without reading Python.

# pytest: direct and imperative — best for a few developer-owned rules
def test_parcel_contract(gdf):
    assert {"parcel_id", "land_use"}.issubset(gdf.columns)
    assert gdf.crs.to_epsg() == 3857
    assert gdf["land_use"].isin({"residential", "commercial"}).all()

The trade-off in depth — expectations, data docs, and where each shines — is the subject of the dedicated pytest-geo vs Great Expectations comparison.

Topology Rules: In-Process Shapely, or PostGIS

A topology rule such as “no polygon self-intersects” or “no gaps between adjacent parcels” can run in-process with Shapely or server-side with PostGIS. The deciding factor is where the data lives and whether the rule must also gate database ingestion. In-process Shapely is the right call when your test already holds a GeoDataFrame and the fixture fits in memory. PostGIS is the right call when the data is already in the database, the fixture is too large to load, or the identical rule must run as an ingestion constraint so bad geometry never lands — a duplication the Shapely vs PostGIS guide unpacks with benchmarks.

# In-process: no database needed, fixture in memory
from shapely import is_valid
assert gdf.geometry.map(is_valid).all()

-- Server-side: same rule, but gates ingestion and scales past memory
SELECT id FROM parcels WHERE NOT ST_IsValid(geom);

Spatial Joins: R-tree, or GiST

When a test performs a spatial join — “every point falls inside exactly one zone” — the index behind it drives both speed and, subtly, result ordering. An in-memory R-tree (via Shapely’s STRtree or GeoPandas sjoin) is ideal when the join runs against an in-process fixture. A database GiST index is the right choice when the join is a query you are also validating for production, because testing against the same index type the database uses catches ordering and cardinality differences an in-memory index would hide. The performance and determinism trade-offs are measured in R-tree vs GiST index performance.

Four Questions That Settle Most Tool Choices

Tool debates in spatial QA are long because they are usually conducted at the wrong altitude — comparing libraries in the abstract rather than answering what the specific rule needs. Four questions settle nearly all of them, and they are worth asking in this order because an early answer often makes the later questions moot.

Where does the data already live? If it is in PostGIS and the check is a set-level property, the answer is PostGIS and the discussion is over: moving several million rows into Python to compute what an indexed query answers in place is a decision that has to be justified, not defaulted into. If the data is an in-memory frame produced two steps earlier, the reverse holds.

Is the rule about one feature or about the relationship between features? Per-feature properties — validity, type, attribute domain — run anywhere and should run wherever it is cheapest, which is usually in-process. Set-level properties — overlaps, gaps, connectivity, join cardinality — need an index to be tractable at all, and that pushes them towards the database.

Who reads the failure? A rule whose failures are triaged by engineers wants a rich assertion message and belongs in pytest. A rule whose failures are sent to a data producer wants a structured, rendered report and belongs in an expectation suite. This question decides more than the previous two and is the one most often skipped.

How often does the threshold change? A rule with a stable, absolute threshold is fine as code. A rule whose threshold is under negotiation — being tightened quarterly as a producer improves — wants to live in configuration that changes without a code review cycle, which favours a declarative tool.

Four questions, asked in order Four question rows, each with two answers and the tool each answer implies. Question one asks where the data already lives: already in the database implies a server-side check, already in memory implies an in-process check. Question two asks whether the rule is about one feature or about relationships between features: a per-feature rule can run anywhere and should run where it is cheapest, a set-level rule needs a spatial index and pushes towards the database. Question three asks who reads the failure: an engineer implies a pytest assertion with a detailed message, a data producer implies a rendered expectation report. Question four asks how often the threshold changes: a stable threshold can live in code, a negotiated one belongs in configuration. A footer notes that answering in this order frequently makes the remaining questions unnecessary. QUESTION ANSWER IMPLIES 1 · Where does the data live? already in PostGIS already in a frame server-side check in-process check 2 · One feature, or many? per-feature property set-level property run it wherever it is cheapest needs an index → database 3 · Who reads the failure? an engineer a data producer pytest, rich assertion message expectation suite, rendered report 4 · How often does the threshold move? stable under negotiation fine as code configuration, reviewed separately Ask them in order: a clear answer to question one usually makes questions two and three academic.

When Two Tools Must Coexist

The realistic end state is not one tool but two or three, each owning a layer, and the risk then shifts from choosing to keeping them consistent. Three failure modes recur, and all are avoidable with a little deliberate plumbing.

Divergent thresholds. The same rule enforced in pytest and in an expectation suite with two different tolerance values will eventually disagree, and the disagreement surfaces as a check that passes in one place and fails in another. The remedy is one versioned configuration file that both read; neither tool should ever contain a literal threshold.

Divergent semantics. A topology rule expressed in Shapely and again in PostGIS can differ at boundary-touching cases even at matching GEOS versions, because the code paths are not identical. Where a rule genuinely must exist in both, add a small consistency test that runs the same fixture through both implementations and asserts they agree — and treat a disagreement as a bug in the rule, not as noise.

Divergent ownership. A check that exists in two places tends to be maintained in one. Naming a single owner per rule, regardless of how many tools implement it, is what stops the second copy from silently drifting into decoration.

The overhead of coexistence is real, which is itself an argument for keeping the number of tools small. Two is usually right: one imperative tool for code-owned rules, one declarative tool for producer-facing data contracts. A third earns its place only when it does something neither of the first two can.

Common Failure Modes and Gotchas

  1. Duplicating a rule in two tools. Enforcing the same contract in both pytest and Great Expectations doubles maintenance and lets the two drift; pick the layer that owns the rule.
  2. Testing against the wrong index. Validating an in-memory R-tree join when production uses GiST can pass while the database query returns rows in a different order or cardinality.
  3. In-process checks on out-of-memory fixtures. Loading a million-row layer into Shapely to check validity OOMs the runner; push it to PostGIS.
  4. Choosing by familiarity, not fit. Reaching for pytest because the team knows it, when a non-developer data owner needs to review the contract, buries the rule where its audience cannot read it.
  5. Ignoring the production layer. A rule that must also gate ingestion belongs in the database; enforcing it only in-process leaves the ingestion path unguarded.

The Cost of Choosing Late

A tool choice deferred is a tool choice made by accretion, and the resulting arrangement is always worse than any of the options considered honestly. The pattern is recognisable: rules land wherever the engineer who wrote them was comfortable, thresholds appear as literals in three files, and by the time anyone asks “where do our spatial rules live” the answer is a list rather than a place.

An accreted rule estate against a decided one Two arrangements compared. On the left, rules are scattered: several pytest modules, an expectation suite, a set of ad-hoc SQL scripts and a database constraint, with threshold values duplicated as literals in three separate places and no owner recorded for any rule. On the right, the same rules are placed deliberately: code-owned rules in pytest, producer-facing contracts in a single expectation suite, and invariants as database constraints, with every one of them reading thresholds from one versioned configuration file and each rule carrying a named owner. Accreted — chosen by whoever wrote it tests/test_geom.py tests/test_crs.py suite_v3.json scripts/checks.sql an EXCLUDE constraint nobody documented tolerance 0.01 appears as a literal in three of these no rule records an owner “Where do our spatial rules live?” — a list, not a place Decided — placed on purpose pytest — code-owned rules, engineer-facing failures one expectation suite — producer-facing contracts constraints — invariants that must be impossible thresholds.yaml — read by all three, versioned one named owner per rule The difference is not tooling. It is whether anyone decided.

Reversing accretion is expensive, which is the practical argument for spending an hour on the four questions before the third rule is written rather than after the thirtieth. The migration itself is mechanical — extract literals into configuration, move rules to their proper layer, delete duplicates — but it touches every check simultaneously, which makes it exactly the kind of work that never quite reaches the top of a backlog.

The lightweight preventative is a single page in the repository recording where each category of rule belongs and why. It takes twenty minutes to write, it answers the question for every future contributor, and it turns the tool choice from a recurring debate into a decision that was already made.

Frequently Asked Questions

Is it worth writing an abstraction so a rule can run in either place?

Almost never. An abstraction over Shapely and PostGIS has to expose the intersection of both, which excludes precisely the capabilities that made you want each one — the rich diagnostic string on one side, the indexed set-level query on the other. What it buys is the ability to switch, which teams rarely exercise. The cheaper arrangement is to write the rule where it belongs and, when it genuinely must exist twice, add the consistency test rather than the abstraction.

How do we choose when the rule is new and nobody knows how it will be used?

Start in-process, in pytest, with the threshold in configuration. It is the fastest to write, the easiest to change, and the cheapest to move later — an in-process rule migrating into a suite or into SQL is an afternoon, whereas a rule that started as a database constraint is entangled with migrations. Optimising for reversibility is the right instinct while the requirement is still soft.

Does the tool choice change if the pipeline is not in Python?

The categories survive; the products change. Every stack has an imperative test framework, most have a declarative data-quality tool, and any database has constraints. The four questions — where the data lives, whether the rule is per-feature or set-level, who reads the failure, and how often the threshold moves — are language-independent, and they produce the same three-way split.

What about doing everything in the database?

It is a coherent position when the pipeline is genuinely SQL-first, and it has real advantages: one execution engine, one place to look, no serialisation. The limits show up in two places. Diagnostics are poorer — a constraint violation identifies a row, not a reason — and unit-level logic that never touches storage becomes awkward to test at all. Teams that go this route usually end up with a small pytest suite anyway, for the code that transforms rather than stores.

How many rules is too many for one place?

The number matters less than whether anyone can still answer what the estate requires. A useful check is to ask a colleague who did not write them to state the dataset’s contract from the artefacts alone. If they can do it from a suite file, the arrangement is working. If they have to read several hundred assertions to find out, the estate has outgrown its current shape regardless of how many rules that turned out to be.

Should the same tool be used across every team in an organisation?

For the producer-facing contract layer, yes — a shared vocabulary is most of the value, and a report that looks different for every dataset is not a report. For code-owned rules, no; that layer is internal to each team’s suite and standardising it buys little while costing autonomy. Standardise the interface between teams and leave the interior alone.

One more decision worth making early

Beyond which tool runs a rule, decide where its threshold is reviewed. A tolerance in a code file is reviewed by whoever reviews code; the same tolerance in a configuration file can be reviewed by the team that owns the data contract. That difference determines whether a loosened threshold gets noticed, and it is easier to set up on day one than to retrofit across an estate.

The pattern that holds up is a single configuration file, owned jointly, referenced by every tool, with a comment beside each value recording why it is what it is. The comment matters more than it looks: a threshold whose justification is written down survives a personnel change, and one whose justification lived in someone’s head is loosened the first time it becomes inconvenient.

A note on evaluating a new tool

When a new option appears, resist evaluating it on a demo. Evaluate it on the rule your estate currently handles worst — the one with a negotiated threshold, a producer who needs a report, and a geometric predicate that no built-in vocabulary expresses. Every tool handles the easy rules well, and the hard rule is what determines whether adopting it reduces the number of places your rules live or adds one more.

The other question worth asking early is what the tool produces when it fails. A tool whose failure output is a stack trace belongs in the engineer-facing layer whatever else it can do; one whose output is a structured document belongs in the producer-facing layer. That single property predicts where it will end up more reliably than any feature comparison.

Conclusion

Choosing a spatial testing tool comes down to three questions — how many rules and who reads them, where the data lives, and where the join runs — each of which points cleanly at one option. Match the tool to those constraints and gates stay fast, rules live in exactly one place, and tests exercise the same engines production uses. For the gate architecture these tools plug into, return to CI/CD spatial quality gates.

Whatever is chosen, write the choice down where the next contributor will meet it: a short note in the repository saying which layer owns which kind of rule, and why. Twenty minutes of writing removes the same debate from every subsequent quarter, and it converts an accumulated set of individual preferences into a decision the team can revisit deliberately rather than rediscover by accident.