Validating Attribute Schemas with Great Expectations

3 min read

The attribute layer of a spatial dataset — the fields, dtypes, value domains and CRS that every downstream consumer assumes — is where silent corruption most often hides, and a declarative Great Expectations suite is a natural way to gate it. This guide sits beneath attribute and metadata checks and shows how to express a geospatial attribute contract as a suite: required columns, typed and range-bounded values, a constrained land-use domain, a declared CRS and a non-null geometry column — then run it as a checkpoint that gates the pipeline and publishes browsable data docs. The reason to reach for Great Expectations here rather than pytest is audience: the attribute contract is often the interface between a data producer and a non-developer consumer, and a data-docs report is something both can read.

Why the attribute layer needs its own gate

A geometry can be perfectly valid while its attributes are wrong: a parcel with a null parcel_id, a land-use code outside the agreed set, an elevation of 90,000 metres, or a layer whose declared CRS silently reverted to a default. None of those are caught by geometry validation patterns — they are contract violations, not shape defects. Expressing them as a suite makes the contract explicit, versioned and reviewable, so a producer cannot quietly change a domain without the gate turning red.

Expectation reference

Contract rule Expectation Failure it catches
Field must exist expect_column_to_exist Renamed or dropped column
No missing ids expect_column_values_to_not_be_null Null primary key
Value domain expect_column_values_to_be_in_set Unknown category code
Numeric range expect_column_values_to_be_between Out-of-range measurement
Type expect_column_values_to_be_of_type String where numeric expected
Non-null geometry expect_column_values_to_not_be_null Missing geometry cell

Step-by-step implementation

The suite targets Great Expectations 0.18+ and a GeoDataFrame flattened to a WKT column, and gates on the checkpoint result.

Step 1 — Represent geometry as a serializable column

Great Expectations works on tabular batches, so serialize the geometry to WKT so both attributes and a geometry presence check live in one frame.

import geopandas as gpd

gdf = gpd.read_file("data/parcels.gpkg")
df = gdf.assign(wkt=gdf.geometry.to_wkt()).drop(columns="geometry")
df["srid"] = gdf.crs.to_epsg()           # carry CRS as a column for a domain check

Step 2 — Build the attribute suite

import great_expectations as gx

ctx = gx.get_context()
batch = ctx.data_sources.pandas_default.read_dataframe(df)

batch.expect_column_to_exist("parcel_id")
batch.expect_column_values_to_not_be_null("parcel_id")
batch.expect_column_values_to_be_in_set("land_use",
                                        ["residential", "commercial", "agricultural"])
batch.expect_column_values_to_be_between("elevation_m", -430, 8849)
batch.expect_column_values_to_be_in_set("srid", [3857])     # CRS contract
batch.expect_column_values_to_not_be_null("wkt")

Step 3 — Save the suite and wire a checkpoint

suite = batch.save_expectation_suite(expectation_suite_name="parcels_attributes")
checkpoint = ctx.add_or_update_checkpoint(
    name="parcels_attributes",
    validations=[{"batch_request": batch.batch_request,
                  "expectation_suite_name": "parcels_attributes"}],
)

Step 4 — Gate on the checkpoint

import sys
result = ctx.run_checkpoint(checkpoint_name="parcels_attributes")
sys.exit(0 if result.success else 1)     # non-zero blocks the merge

For the genuinely spatial rules — validity, bounds, topology — extend this suite with the custom expectations from writing custom geometry expectations, so one suite covers both attributes and geometry.

Verification pattern

Prove the gate blocks by feeding it a batch with a deliberate contract violation and asserting the checkpoint reports failure.

bad = df.copy()
bad.loc[0, "land_use"] = "industrial"        # not in the agreed domain
result = ctx.run_checkpoint(checkpoint_name="parcels_attributes",
                            batch_request={"dataframe": bad})
assert result.success is False               # the gate must reject it

Failure modes and edge cases

  1. CRS compared as a string. "EPSG:3857" vs "epsg:3857" gives a false failure; normalize to the integer authority code, as above with srid.
  2. Geometry dropped, not serialized. Removing the geometry column without a WKT surrogate means the suite cannot even check geometry presence.
  3. mostly hiding violations. A domain expectation with mostly=0.98 tolerates 2% unknown codes; use 1.0 for a hard contract.
  4. Type expectation on nullable columns. expect_column_values_to_be_of_type can fail on pandas nullable dtypes; assert the dtype explicitly or coerce first.
  5. Suite drift from the producer. If the producer adds a valid new land-use code, the frozen domain rejects it; version the suite and update it deliberately, not by loosening mostly.

Conclusion

A Great Expectations attribute suite turns the contract every consumer assumes — fields, types, domains, CRS, geometry presence — into an explicit, versioned gate that publishes a report both producer and consumer can read. Combined with custom geometry expectations, one checkpoint covers the whole spatial contract. For the broader attribute-checking context, return to attribute and metadata checks.