Hashing Spatial Fixtures for Content-Addressed Storage

9 min read

A content hash turns a fixture from a filename into an identity. This guide sits beneath fixture versioning and provenance and shows how to compute one for spatial data with Shapely 2.x and GeoPandas 0.14+ that is stable across formats, drivers, machines and library versions — which is the whole requirement, and the reason hashing the file itself does not work.

The property to aim for is precise: two fixtures should hash the same if and only if they would produce identical test outcomes. Anything that changes the bytes without changing the outcome must be normalised away; anything that changes the outcome must be inside the digest.

Root cause: why hashing the file fails

Serialising the same GeoDataFrame twice can produce different bytes. Drivers embed timestamps and version strings, compression is not always deterministic, feature order can depend on an index, and coordinate precision follows a creation option that may not be pinned. Hash the file and the identifier changes when none of those affect a single assertion.

The reverse failure is worse. Two genuinely different fixtures can share a file hash if the difference lives somewhere the format does not record — a CRS that one tool wrote and another omitted, an attribute dtype that a driver widened on write. The identifier then claims two datasets are the same when a test would distinguish them.

A file hash is unstable in one direction and blind in the other Two failure modes of hashing the serialised file. In the first, the same data written twice yields different bytes because drivers embed timestamps and version strings, compression is not always deterministic, feature order can vary and coordinate precision follows a creation option, so the identifier changes while nothing a test observes has changed. In the second, two genuinely different fixtures share a file hash because the difference lives somewhere the format does not record, such as an omitted coordinate reference system or an attribute dtype that the driver widened on write, so the identifier claims they are the same when a test would distinguish them. A content hash computed over normalised geometry, an authority code and a deterministic attribute digest avoids both. Unstable: same data, different hash · driver embeds a timestamp or version string · compression is not byte-deterministic · feature order follows an index · coordinate precision is a creation option the identifier moves; no assertion would notice Blind: different data, same hash · a CRS one tool wrote and another omitted · an attribute dtype the driver widened · a precision setting applied on write · a mask or index the format cannot carry the identifier claims sameness a test would deny A content hash over normalised geometry + authority code + a deterministic attribute digest has neither problem. It is stable across formats and drivers, and it changes exactly when something a test can observe changes. The target property: two fixtures hash the same if and only if they would produce identical test outcomes.

Normalisation reference

Component Normalisation Why
Geometry shapely.normalize, then WKB Fixes ring winding and coordinate order
Precision set_precision at the declared grid Two snappings are different data
Feature order Sort by a stable key, then hash in order Row order is not part of the data
CRS crs.to_epsg(), an integer WKT varies by PROJ release
Column order Sort column names Reordered columns are the same data
Dtypes Include the dtype string per column A widened dtype is different data
Null representation Canonicalise to one sentinel None and NaN are the same absence

Step-by-step implementation

The digest targets Shapely 2.x, GeoPandas 0.14+ and hashlib.

Step 1 — Normalise the geometry before serialising

normalize puts rings and coordinate sequences into a canonical order, so two representations of the same shape produce identical bytes.

import hashlib
import geopandas as gpd
import shapely

def geometry_digest(gdf: gpd.GeoDataFrame, grid_size: float) -> bytes:
    geoms = shapely.set_precision(gdf.geometry.values, grid_size=grid_size)
    geoms = shapely.normalize(geoms)                 # canonical winding + vertex order
    h = hashlib.sha256()
    for wkb in shapely.to_wkb(geoms, flavor="iso", include_srid=False):
        h.update(wkb)
    return h.digest()

Excluding the SRID from the WKB is deliberate: the CRS enters the digest once, as an authority code, rather than repeated per geometry in a form that varies by writer.

Step 2 — Digest the attributes deterministically

Column order and row order must both be normalised, and the dtype must be included so a widened column changes the identity.

import pandas as pd

def attribute_digest(df: pd.DataFrame, key: str) -> bytes:
    frame = df.drop(columns=["geometry"], errors="ignore").sort_values(key)
    h = hashlib.sha256()
    for col in sorted(frame.columns):
        h.update(col.encode())
        h.update(str(frame[col].dtype).encode())
        # A canonical string form: None and NaN both become the same token.
        values = frame[col].map(lambda v: "\x00NULL" if pd.isna(v) else repr(v))
        h.update("\x1f".join(values).encode())
    return h.digest()

Step 3 — Combine into one fixture identity

Order the components and length-prefix them so no combination of contents can produce a collision by concatenation.

def fixture_id(gdf: gpd.GeoDataFrame, *, key: str, grid_size: float) -> str:
    epsg = gdf.crs.to_epsg() if gdf.crs else None
    if epsg is None:
        raise AssertionError("fixture has no resolvable EPSG code — identity would be ambiguous")
    parts = [
        b"geom:" + geometry_digest(gdf.sort_values(key), grid_size),
        b"attr:" + attribute_digest(gdf, key),
        b"crs:" + str(epsg).encode(),
        b"grid:" + repr(grid_size).encode(),
    ]
    h = hashlib.sha256()
    for p in parts:
        h.update(len(p).to_bytes(4, "big"))          # length-prefix: no concatenation collisions
        h.update(p)
    return "sha256:" + h.hexdigest()

Step 4 — Prove the properties you claim

The digest makes two claims, and both are testable. Round-tripping through a different format must not change it; changing anything a test observes must.

def test_hash_is_stable_across_formats(tmp_path, parcels):
    a = fixture_id(parcels, key="parcel_id", grid_size=0.001)
    parcels.to_file(tmp_path / "p.gpkg", driver="GPKG")
    parcels.to_file(tmp_path / "p.fgb", driver="FlatGeobuf")
    for name in ("p.gpkg", "p.fgb"):
        back = gpd.read_file(tmp_path / name)
        assert fixture_id(back, key="parcel_id", grid_size=0.001) == a, name

def test_hash_changes_when_the_data_does(parcels):
    a = fixture_id(parcels, key="parcel_id", grid_size=0.001)
    moved = parcels.copy()
    moved.loc[moved.index[0], "geometry"] = moved.geometry.iloc[0].buffer(0.01)
    assert fixture_id(moved, key="parcel_id", grid_size=0.001) != a
Stability and sensitivity, and how each is proven Two properties with their proving tests. Stability requires that the same data written through different drivers and read back yields the identical identifier; it is proven by round-tripping a fixture through two formats and comparing. Sensitivity requires that any change a test could observe yields a different identifier; it is proven by perturbing in turn a geometry, an attribute dtype, the coordinate reference system and the declared precision, and asserting the identifier changed each time. A closing note observes that a hash with only stability is effectively a constant and a hash with only sensitivity is just a file hash, so both must be tested. Stability same data through different drivers → identical identifier PROVEN BY a round trip through two formats without it, the hash is a file hash Sensitivity any observable change → a different identifier PROVEN BY perturbing geometry, dtype, CRS, precision without it, the hash is a constant Both must be tested. A digest that passes only the first is a constant dressed as an identifier; one that passes only the second is what you already had. The sensitivity test is the one that gets skipped, and it is the one that catches a normalisation applied too aggressively.

Verify the fix

Run the two property tests together and confirm they fail for the right reasons when broken:

pytest -q tests/test_fixture_hash.py -v

Remove the normalize call and the stability test should fail while sensitivity still passes. Hash only the geometry and the sensitivity test should fail on the dtype perturbation. Those two failure modes are the ones a hand-written digest most often has.

Using the identity as a storage key

Once the digest is stable, it doubles as a storage key, and that is what makes content-addressed storage work. A fixture is written to a path derived from its own hash, so identical content is stored once regardless of how many suites reference it, and a fetch can verify what it received rather than trusting where it came from.

Two operational properties follow. Deduplication is automatic — two teams generating the same coverage from the same seed store one copy. And verification is local: recompute the digest from the fetched bytes and compare against the key that requested them, so a corrupted transfer or a mismatched cache entry fails immediately instead of producing a test run against unknown data.

The one thing to avoid is deriving the key from a path that happens to include a hash. The reference must be the hash, so that moving, renaming or re-uploading the artefact cannot break the identity — which is exactly the failure a filename-based scheme reintroduces.

Fetch by hash, verify on arrival A sequence starting from a test that requests a fixture by its content hash. The client consults a local cache keyed on that hash; on a miss it fetches the object from the store. The bytes received are re-digested and the result compared against the hash that was requested. A match proves the content is exactly what the test asked for, and the object is cached under that key. A mismatch fails immediately with a message naming both hashes. A closing note contrasts this with a path-based fetch, where a wrong or corrupted object is indistinguishable from a correct one until something downstream behaves oddly. test requests by hash, not by path local cache keyed on the hash store fetch on a miss re-digest compare to the request match → provably right cache it under that key mismatch → fail now both hashes in the message In a path-based fetch, a wrong or corrupted object is indistinguishable from a correct one until something downstream behaves oddly.

Deciding what the digest should be sensitive to

The normalisation choices above are not universal — they encode a judgement about which differences matter, and that judgement belongs to the pipeline rather than to the hashing function. Three decisions come up on every project and each has a defensible answer in both directions.

Should vertex count matter? Two polygons describing the same area with different numbers of collinear vertices are geometrically identical and computationally different: one costs more to process and may simplify differently. If the suite tests simplification or performance, vertex count must be inside the digest and normalize alone is not enough. If it tests only shape, removing collinear vertices before hashing makes the identifier more stable.

Should attribute order within a row matter? Never — a row is a mapping, not a sequence, and sorting column names before digesting is the right normalisation everywhere.

Should the feature count matter when the content is otherwise identical? Always. Two fixtures where one contains a duplicated feature are different inputs to any uniqueness or aggregation test, and a digest that deduplicates before hashing would hide exactly that.

The general rule is to normalise away what the format varies and preserve what the data varies. Format variation — winding, coordinate order, column order, driver quirks — is noise. Data variation — a moved vertex, a widened dtype, an extra row, a different precision — is signal, and any of it that the digest cannot see is a class of change the identity system will silently miss.

Failure modes and edge cases

  1. Normalising too aggressively. Snapping to a coarser grid inside the digest makes two genuinely different fixtures hash the same. Normalise to the declared precision and include that precision in the digest.
  2. Row order left unspecified. Hashing features in whatever order they arrive makes the identifier depend on a read order that formats do not guarantee. Sort by a stable key, and assert that key is unique.
  3. Floating-point repr in the attribute digest. repr of a float differs between platforms in the last digit for some values. Round to a declared precision, or format with an explicit specification.
  4. Empty and null geometry. POLYGON EMPTY and a null geometry are different states that some serialisations conflate. Canonicalise deliberately and document which you chose.
  5. Hashing a lazily-read frame. A frame read with a bounding-box filter contains a subset, so its digest identifies the subset rather than the file. Record the read parameters alongside, or read fully before hashing.
  6. Changing the digest algorithm silently. Every stored fixture’s key becomes unreachable. Version the scheme in the identifier’s prefix so old and new can coexist during a migration.

Conclusion

A fixture hash is a claim that two datasets are interchangeable for testing purposes, and the claim is only as good as the normalisation behind it. Canonicalising geometry, expressing the CRS as a code, digesting attributes with their dtypes in a stable order, length-prefixing the components, and testing both stability and sensitivity produces an identifier that survives formats, drivers and machines — the foundation everything else in fixture versioning and provenance rests on.