Validating Attribute Schemas with Great Expectations

9 min read

The attribute layer of a spatial dataset — the fields, dtypes, value domains and CRS that every downstream consumer assumes — is where silent corruption most often hides, and a declarative Great Expectations suite is a natural way to gate it. This guide sits beneath attribute and metadata checks and shows how to express a geospatial attribute contract as a suite: required columns, typed and range-bounded values, a constrained land-use domain, a declared CRS and a non-null geometry column — then run it as a checkpoint that gates the pipeline and publishes browsable data docs. The reason to reach for Great Expectations here rather than pytest is audience: the attribute contract is often the interface between a data producer and a non-developer consumer, and a data-docs report is something both can read.

Why the attribute layer needs its own gate

A geometry can be perfectly valid while its attributes are wrong: a parcel with a null parcel_id, a land-use code outside the agreed set, an elevation of 90,000 metres, or a layer whose declared CRS silently reverted to a default. None of those are caught by geometry validation patterns — they are contract violations, not shape defects. Expressing them as a suite makes the contract explicit, versioned and reviewable, so a producer cannot quietly change a domain without the gate turning red.

What a suite gives you that assertions do not

Everything Great Expectations checks could be written as a pytest assertion, so the honest question is what the extra machinery buys. Three things, and none of them is the checking itself.

The first is a declarative artefact. A suite is JSON, so it can be diffed, reviewed, generated, and read by someone who does not write Python. When a producer asks “what exactly do you require of this column”, a suite answers without anyone reading test code. The second is structured results. A failed expectation reports the column, the rule, the observed value, the count of unexpected rows, and a sample of them — as data, not as an assertion message — which is what makes an automated report or a dashboard possible. The third is partial success. A pytest assertion is binary; an expectation returns the unexpected fraction, so a rule can be configured to tolerate a small percentage while still reporting the trend. That single property is what lets a rule enter as an observation and be tightened later.

Assertion result versus expectation result on the same rule Two panels. The assertion panel shows a single boolean outcome with an error string, giving only pass or fail, no count of offending rows, no sample, and no way to tolerate a fraction. The expectation panel shows a structured result object carrying the expectation type, the column name, the number of unexpected rows, the unexpected fraction as a percentage, and a list of sample offending values; from that it supports a configured tolerance fraction, a trend line across runs, and a human-readable report. A footer notes that the structured form is what allows a new rule to be introduced against data that is already imperfect, by observing before enforcing. Plain assertion False "surface_type contained unexpected values" − no count of offending rows − no sample to look at − no way to tolerate a fraction − binary — cannot be observed first Expectation result expectation: values_to_be_in_set column: surface_type unexpected: 41 rows · 0.8% · ["grvl", …] + tolerate a configured fraction + trend the fraction across runs + a report a non-programmer can read The difference matters most when introducing a rule against data that is already imperfect: the structured form lets you observe the 0.8 per cent, agree a target with the producer, and enforce once it is met — instead of choosing between a red build and no rule at all.

Expectation reference

Contract rule Expectation Failure it catches
Field must exist expect_column_to_exist Renamed or dropped column
No missing ids expect_column_values_to_not_be_null Null primary key
Value domain expect_column_values_to_be_in_set Unknown category code
Numeric range expect_column_values_to_be_between Out-of-range measurement
Type expect_column_values_to_be_of_type String where numeric expected
Non-null geometry expect_column_values_to_not_be_null Missing geometry cell

Step-by-step implementation

The suite targets Great Expectations 0.18+ and a GeoDataFrame flattened to a WKT column, and gates on the checkpoint result.

Step 1 — Represent geometry as a serializable column

Great Expectations works on tabular batches, so serialize the geometry to WKT so both attributes and a geometry presence check live in one frame.

import geopandas as gpd

gdf = gpd.read_file("data/parcels.gpkg")
df = gdf.assign(wkt=gdf.geometry.to_wkt()).drop(columns="geometry")
df["srid"] = gdf.crs.to_epsg()           # carry CRS as a column for a domain check

Step 2 — Build the attribute suite

import great_expectations as gx

ctx = gx.get_context()
batch = ctx.data_sources.pandas_default.read_dataframe(df)

batch.expect_column_to_exist("parcel_id")
batch.expect_column_values_to_not_be_null("parcel_id")
batch.expect_column_values_to_be_in_set("land_use",
                                        ["residential", "commercial", "agricultural"])
batch.expect_column_values_to_be_between("elevation_m", -430, 8849)
batch.expect_column_values_to_be_in_set("srid", [3857])     # CRS contract
batch.expect_column_values_to_not_be_null("wkt")

Step 3 — Save the suite and wire a checkpoint

suite = batch.save_expectation_suite(expectation_suite_name="parcels_attributes")
checkpoint = ctx.add_or_update_checkpoint(
    name="parcels_attributes",
    validations=[{"batch_request": batch.batch_request,
                  "expectation_suite_name": "parcels_attributes"}],
)

Step 4 — Gate on the checkpoint

import sys
result = ctx.run_checkpoint(checkpoint_name="parcels_attributes")
sys.exit(0 if result.success else 1)     # non-zero blocks the merge

For the genuinely spatial rules — validity, bounds, topology — extend this suite with the custom expectations from writing custom geometry expectations, so one suite covers both attributes and geometry.

Where geometry fits in a tabular tool

Great Expectations reasons about columns, and a geometry column is not a type it understands. That mismatch is the whole design problem of using it on spatial data, and there are exactly three workable answers.

Project the geometry into scalar columns. Derive area, vertex_count, bounds_minx and friends as ordinary numeric columns, then express the rules on those. This is the highest-leverage option: it needs no custom code, the expectations are the built-in numeric ones, and the derived columns are exactly the quantities a data steward wants to see in a report anyway. Its limit is that it can only express properties that survive being reduced to a number — it cannot say anything about shape.

Serialise the geometry to WKT and use string expectations. Useful for a narrow set of rules: geometry type prefix, non-null, not EMPTY, length below a cap. It is cheap and requires nothing custom, but WKT string comparisons are a trap for anything numeric, because two identical geometries can serialise differently.

Write a custom expectation. The correct answer for genuine geometric predicates — validity, containment within an envelope, minimum vertex spacing. It costs real effort to write and maintain, so reserve it for rules that the first two options cannot express and that matter enough to justify a maintained class.

Rule you want to express Cheapest workable approach
Geometry column is never null WKT string, not-null expectation
All features are polygons Derived geom_type column, value set
Area within a plausible range Derived area column, numeric range
Vertex count below a cap Derived vertex_count column, numeric range
Feature lies inside the served extent Derived bounds columns, four numeric ranges
Geometry is OGC-valid Custom expectation — nothing else expresses it
No two features overlap Not a column rule at all — belongs in a topology check
Projecting a geometry column into columns a tabular tool can check A geometry column on the left fans out into four derived scalar columns: geometry type, area, vertex count, and bounds as four numbers. Each derived column is paired with the built-in expectation that covers it — a value-set expectation for geometry type and numeric range expectations for the rest. Below, a separate group holds the two rules that cannot be reduced to a column: OGC validity, which needs a custom expectation, and pairwise non-overlap, which is a set property and belongs in a topology check rather than in a column suite. geometry not a tabular type geom_type area vertex_count bounds_min/max values_to_be_in_set values_to_be_between values_to_be_between values_to_be_between × 4 Cannot be projected: OGC validity → custom expectation · pairwise non-overlap → not a column property, use a topology check Derive the columns once, in the same step that loads the frame, so the suite and the report both see the quantities a steward actually asks about.

The derived-column route has a side benefit worth naming: the same columns make an excellent drift signal. Tracking the distribution of area and vertex_count between runs catches upstream changes — a simplification tolerance altered, a different extract boundary — long before any individual value breaches its range. That is a monitoring capability you get free, and it is usually more valuable in the first year than any single expectation in the suite.

Verification pattern

Prove the gate blocks by feeding it a batch with a deliberate contract violation and asserting the checkpoint reports failure.

bad = df.copy()
bad.loc[0, "land_use"] = "industrial"        # not in the agreed domain
result = ctx.run_checkpoint(checkpoint_name="parcels_attributes",
                            batch_request={"dataframe": bad})
assert result.success is False               # the gate must reject it

Keeping the suite from rotting

A suite is code with none of code’s safety nets: no type checker reads it, no linter notices when a column disappears, and an expectation naming a column that no longer exists usually reports success rather than failure. Three habits keep it honest.

Assert the column set itself, first, before any column-level rule. A table_columns_to_match_set expectation turns a renamed or dropped column into an immediate, comprehensible failure instead of a suite that silently stops checking half of what it claims to. Without it, deleting a column makes a suite greener, which is the worst possible incentive.

Generate what can be generated, hand-write what carries judgement. Type, nullability, and column presence can be derived from a schema definition, and regenerating them on every schema change keeps them accurate for free. Value domains, ranges, and tolerated fractions encode decisions and must be written and reviewed by a person. Mixing the two in one file makes the generated parts unreviewable and the judged parts liable to be overwritten.

Version the suite with the pipeline, and record which version ran. A validation result that does not name the suite version cannot be compared against last week’s, which removes the trend signal that was one of the main reasons to use expectations at all.

A dropped column makes an unguarded suite greener Two sequences compared. In the unguarded suite, the schema loses a column, the expectations that referenced it evaluate against nothing, the failure count falls, and the run reports success even though coverage has decreased. In the guarded suite, a column-set expectation runs first, detects that the observed column set no longer matches the declared one, and fails the run with a message naming the missing column before any column-level rule executes. A closing line states the principle: deleting data must never make a suite greener. No column-set expectation column dropped its rules evaluate against nothing failure count falls run reports success coverage silently reduced With a column-set expectation first column dropped columns_to_match_set runs before anything else run fails, naming the missing column before a single column-level rule executes The principle: deleting data must never make a suite greener. Any check whose coverage depends on the data it is checking needs a guard that asserts the shape before the content.

Failure modes and edge cases

  1. CRS compared as a string. "EPSG:3857" vs "epsg:3857" gives a false failure; normalize to the integer authority code, as above with srid.
  2. Geometry dropped, not serialized. Removing the geometry column without a WKT surrogate means the suite cannot even check geometry presence.
  3. mostly hiding violations. A domain expectation with mostly=0.98 tolerates 2% unknown codes; use 1.0 for a hard contract.
  4. Type expectation on nullable columns. expect_column_values_to_be_of_type can fail on pandas nullable dtypes; assert the dtype explicitly or coerce first.
  5. Suite drift from the producer. If the producer adds a valid new land-use code, the frozen domain rejects it; version the suite and update it deliberately, not by loosening mostly.

Conclusion

A Great Expectations attribute suite turns the contract every consumer assumes — fields, types, domains, CRS, geometry presence — into an explicit, versioned gate that publishes a report both producer and consumer can read. Combined with custom geometry expectations, one checkpoint covers the whole spatial contract. For the broader attribute-checking context, return to attribute and metadata checks.