Fixture Versioning and Provenance

15 min read

Fixture versioning and provenance is the discipline of knowing, for any test run, exactly which spatial state it ran against and how that state was produced. It sits beneath Test Data Generation & Mocking Strategies as the bookkeeping layer that makes everything above it reproducible: a generator is only useful if its output can be identified, a regression test only proves something if the data it ran against is known, and a fixture that grows past what a repository can hold needs somewhere to live that does not lose its identity.

The question this layer answers is deceptively simple. When a test passed six months ago and fails today, did the data change? Without provenance that question requires an afternoon of archaeology. With it, it is a comparison of two hashes.

Content Addressing: What a Spatial Hash Must Cover

A fixture’s identity is the content that affects a test’s outcome, and for spatial data that is more than the geometry. Four components must all be inside the hash, and leaving any one out produces an identifier that says two different fixtures are the same.

The geometry, canonically serialised. WKB rather than WKT, because textual serialisation varies by precision setting and library version while the binary form does not. Coordinate order and ring winding must be normalised first, or two identical shapes hash differently.

The CRS, as an authority code rather than as a WKT string. The same coordinate reference system serialises to different WKT under different PROJ versions, so hashing the string makes the identifier depend on the environment rather than on the data.

The attribute table, with a stable column order and explicit dtypes. A frame whose columns arrive in a different order is the same data and must hash the same; a frame where an integer column became a float is different data and must not.

The declared precision or grid size, because two fixtures that differ only in snapping are genuinely different inputs to any topology check.

Inside and outside a spatial fixture hash Two lists. Inside the hash: geometry serialised canonically as well-known binary with ring winding and coordinate order normalised first; the coordinate reference system expressed as an authority code; the attribute table with a stable column order and explicit dtypes recorded; and the declared precision or grid size, since two fixtures differing only in snapping are genuinely different inputs. Outside the hash: the file path, the file modification time, the driver that wrote the file, and any coordinate reference system serialised as a well-known-text string, because each of those varies with the environment rather than with the data and would make the identifier unstable across machines. Inside the hash geometry as canonical WKBCRS as an authority codeattribute table, stable order + dtypesdeclared precision / grid size winding and coordinate order normalised firstnot the WKT — that varies by PROJ versionreordered columns are the same datatwo snappings are different inputs Outside the hash the file paththe modification timethe writing driverCRS as a WKT string the same data can live anywherechanges on every copyGeoPackage and FlatGeobuf carry the same dataserialises differently per PROJ release Everything on the right varies with the environment rather than with the data. Including any of it makes the identifier unstable across machines.

The right-hand column is the half teams get wrong. Hashing the serialised file is easy and produces an identifier that changes when the driver changes, the compression setting changes, or the file is simply rewritten — so it identifies a file rather than a fixture, and two identical datasets written by different tools appear to be different data.

The Provenance Record

A hash says which data ran; a provenance record says where that data came from. Both are needed, and the second is what makes a fixture regenerable rather than merely identifiable.

Field Example Answers
fixture_id sha256:9f2a… Which spatial state ran
generator fixtures.parcels:build_coverage What produced it
generator_version git revision of the generator Which implementation
config_hash sha256:7a2e… Which parameters
seed 20260811 Which draw from the generator
engine_versions GEOS 3.12.1, PROJ 9.4.1 Which libraries shaped it
created_at ISO timestamp When, for retention
defect_class topology:bowtie Why it exists — and when it can be retired

The last row is the one that is almost never recorded and the one that makes deletion possible. A fixture whose purpose is written down can be retired when the defect it provokes becomes impossible; one whose purpose lives in somebody’s memory is kept forever, because nobody can prove it is safe to remove.

Where Fixtures Should Live

A spatial fixture outgrows a repository sooner than a textual one, and the storage decision has consequences for both reproducibility and everyday developer experience. Four arrangements are common and they trade differently.

Generated on demand is the best default. Nothing is stored; a seeded generator plus a versioned configuration produces the fixture at test time, and the provenance record is the generator’s inputs. It costs generation time on every run and it makes the fixture’s identity provable rather than asserted.

Committed to the repository suits small fixtures that must be reviewed as artefacts — a hand-crafted pathological geometry, a regression case extracted from an incident. Past a few megabytes it becomes hostile, because every clone pays for it forever and binary diffs are unreviewable.

Large-file storage attached to the repository keeps the fixture logically versioned with the code while storing the bytes elsewhere. It preserves the workflow at the cost of an extra tool and a fetch step, and it is the usual answer for fixtures in the tens or hundreds of megabytes.

Content-addressed object storage decouples the fixture from the repository entirely: tests reference a hash, a small client fetches and caches by that hash, and identical content is stored once regardless of how many suites use it. It scales furthest and requires the most plumbing.

Four places a fixture can live Four storage arrangements compared across four properties. Generation on demand from a seeded generator has no practical size ceiling, adds nothing to a clone, offers a reviewable generator rather than a reviewable artefact, and gives the strongest reproducibility because identity is provable from the recorded inputs. Committing to the repository suits small artefacts that must be reviewed directly, but every clone pays for the bytes permanently and binary differences are unreviewable. Large-file storage attached to the repository preserves the logical versioning while moving the bytes elsewhere, at the cost of an additional tool and a fetch step. Content-addressed object storage scales furthest, stores identical content only once however many suites reference it, and requires the most plumbing to set up. ARRANGEMENT SUITS COSTS Generated on demand seed + versioned config almost everything — the default identity provable from inputs generation time per run Committed to the repo bytes in version control small hand-crafted pathologies incident regressions every clone pays, forever Large-file storage pointer in the repo, bytes outside tens to hundreds of megabytes keeps the review workflow an extra tool and a fetch step Content-addressed store tests reference a hash gigabytes, shared across suites identical content stored once the most plumbing

The four are not exclusive, and a healthy arrangement usually uses three of them: generation for the bulk of the suite, a handful of committed pathologies that must be reviewable, and a store for the production-shaped extracts that only the scheduled tier reads.

What a Failing Run Should Be Able to Tell You

Provenance earns its keep at exactly one moment: when a run fails and somebody has to decide whether the code changed, the data changed, or the environment changed. A run that carries its fixture identity answers that in seconds; one that does not begins with an hour of establishing what it ran against.

Three comparisons resolve almost every case, and all three are comparisons of recorded fields rather than investigations.

Code revision differs, fixture hash identical. The data is the same and the code moved, so the change under review caused the failure. This is the case a merge gate is designed for, and it is the only one where blocking the author is correct.

Code revision identical, fixture hash differs. The code is unchanged and the data moved. Either a generator input changed — which is visible as a different config hash — or a stored fixture was replaced. Neither is the author’s fault, and the fix is upstream of the suite.

Both identical, engine versions differ. Neither the code nor the data moved, so the runtime did. This is the case that produces the most wasted investigation when it is not recorded, because every instinct points at the data and the data is provably unchanged.

Attributing a failure from three recorded fields Three comparisons between a failing run and the last passing one. When the code revision differs but the fixture hash is identical, the data was unchanged and the change under review caused the failure, so the author can act and blocking the merge is correct. When the code revision is identical but the fixture hash differs, the code was unchanged and the data moved, either through a changed generator configuration or a replaced stored fixture, so the cause lies upstream of the suite and the author cannot fix it. When both the code revision and the fixture hash are identical but the recorded engine versions differ, neither the code nor the data moved and the runtime changed, so the owner of the container image should act. A footer records that the question of who can act is what decides whether a failure should block a merge. WHAT DIFFERS FROM THE LAST PASS CAUSE WHO CAN ACT code revision differs fixture hash identical engine versions identical the change under review the author block the merge fixture hash differs code revision identical check the config hash next the data moved whoever owns the input do not block the merge engine versions differ code and fixture identical GEOS, PROJ, GDAL, grid package the runtime moved the image owner report as infrastructure Who can act is what decides whether a failure should block a merge — and it is answerable from three recorded fields rather than from an investigation.

The practical requirement is small: the run must emit those fields, and they must be retained long enough to compare against. A JSON summary written as a CI artefact on every run, successful or not, is sufficient — and the “successful or not” clause matters, because comparing a failure against the last passing run is impossible if passing runs record nothing.

Versioning the Generator, Not Just Its Output

A generated fixture has two sources of change and only one is usually versioned. The configuration is obvious and is normally in the repository; the generator’s own implementation is less obvious and moves more often.

The distinction matters because the two produce different kinds of change. A configuration change is intentional and reviewable — someone widened a bounding box, added a defect class, changed a seed — and the resulting fixture change is expected. An implementation change may be entirely incidental: a refactor that alters the order in which vertices are appended, a library upgrade that changes how a random draw is consumed, a bug fix that corrects something nobody knew was wrong. Each of those alters the output from identical inputs, and none of them is visible in the configuration.

Recording the generator’s revision alongside the config hash separates the two, and it changes what a mass fixture-hash change means. If the config hash moved, the change was intended. If only the generator revision moved, something incidental altered the output, and that is worth a moment’s attention before it is accepted — occasionally it is a bug fix and occasionally it is a bug.

A related discipline is to keep the generator’s own dependencies pinned alongside the pipeline’s. A change in how a pseudo-random generator draws from a distribution, or in how a geometry constructor orders coordinates, produces different bytes from the same seed. Without the pin, a mass hash change appears with no corresponding change in either the configuration or the generator, which is the most confusing version of this situation and the least diagnosable.

Retirement Is Part of Versioning

The stage teams skip is deletion, and skipping it has a compounding cost. A suite that only ever adds fixtures gets slower every quarter while its coverage stays flat, and the fixtures nobody dares remove are exactly the ones whose purpose was never written down.

Two triggers make retirement decidable. A fixture whose defect class is now prevented by construction — the invalid geometry that a database constraint makes impossible, the duplicate identifier that a unique index rejects — is provably redundant, and the constraint is better evidence than the test. A fixture whose test has not failed in two years against a rule that has not changed is a candidate for review rather than automatic deletion; it may be a rare-but-real case, or it may be dead weight, and the provenance record’s defect_class field is what lets someone decide in a minute rather than an hour.

Reviewing that list once a year takes about an hour and is the only mechanism most suites have for getting smaller.

Caching Fixtures Safely

A generated fixture that takes seconds to build is worth caching, and a cache is where fixture identity earns its keep operationally. The rule is short: key the cache on the fixture’s identity, never on anything else. A cache keyed on the hash of the generator source plus its configuration plus the seed can never serve stale content, because different inputs produce a different key. A cache keyed on a branch name, a date, or a filename can and eventually will.

The failure mode of a badly-keyed fixture cache is worse than a slow suite. A stale fixture served from cache means the suite is testing yesterday’s data while reporting on today’s commit, and every assertion passes because the data genuinely satisfies them — it is simply the wrong data. Nothing in the run indicates it, because the fixture identity, if it is recorded at all, is recorded from the cached artefact rather than from what should have been built.

Two safeguards make this robust. Verify after restore: recompute the fixture’s hash from the restored bytes and compare it against the key that fetched it, so a corrupted or mismatched entry fails immediately rather than silently. And include a manual generation counter in the key, so a human can invalidate everything without editing a generator, which is occasionally necessary when a cache holds something wrong for a reason nobody can identify.

The same discipline extends to fixtures fetched from a content-addressed store. The hash is the reference, so verification is a natural part of the fetch rather than an extra step — recompute on arrival and compare. That property is a large part of why content addressing scales better than a path-based arrangement: correctness is checkable locally rather than assumed from where the bytes came from.

Frequently Asked Questions

Should the hash cover the file or the data?

The data. Hashing the file means the identifier changes when the driver, the compression setting, or the write order changes, none of which alter what the test sees. Hashing canonically-serialised geometry plus CRS plus attributes gives an identifier that is stable across formats, which is what makes it possible to say two fixtures in different formats are the same fixture.

How do you version a fixture that is deliberately random?

By versioning its generator and its seed rather than its output. A fixture drawn from a seeded generator is fully determined by the generator revision, the configuration hash and the seed, so those three fields are its version — and they are far smaller than the data. Recording them in the failure message is what turns a random exploration into a reproducible case.

Does every fixture need a provenance record?

Every fixture that a test depends on, yes; ephemeral data constructed inline inside a single test, no. The dividing line is whether anyone could later need to know what the test ran against. A four-vertex polygon built in the test body is self-documenting; a coverage of ten thousand parcels is not, however it was produced.

What happens when a generator changes and every hash moves?

That is the system working, and it should be a reviewable event rather than a surprise. A generator change that alters output is a change to what the suite tests, so it belongs in its own commit, with the mass hash change visible in the diff and the test run against the new fixtures as the review. Bundling it with a code change is what makes it look alarming.

How long should fixtures be retained?

Long enough to reproduce any run whose result is still relied upon, which for most teams means the current release plus a small number of previous ones. Content-addressed storage makes this cheap because identical content is stored once, and the retention decision becomes about which references to keep rather than about bytes.

Is a fixture derived from production data ever acceptable?

Only after it has stopped being production data. A redacted or synthesised derivative is a fixture; a copy with the names removed is still personal data with a shorter field list. The provenance record should state which of the two it is, because the retention and access obligations differ completely and nobody can tell by looking at the file.

Provenance as a Security Property

There is a second reason to record where a fixture came from, and it has nothing to do with reproducibility. A fixture derived from real data inherits the obligations of the data it came from — retention limits, access restrictions, deletion requests — and those obligations follow the derivative wherever it goes. Without a provenance record, nobody can tell by looking at a file whether it carries them.

The record therefore needs one more field beyond the reproducibility set: the derivation of the fixture, stating whether it was synthesised from nothing, generated from a schema, or derived from a real extract, and in the last case what treatment was applied. A file marked as synthesised can be shared freely, cached anywhere and kept indefinitely. A file marked as derived from a production extract with coordinate truncation applied cannot, and the difference is invisible in the bytes.

Two consequences follow for how fixtures are handled. Anything derived from real data should be excluded from general caching unless the cache honours the same access controls as the source, because a cache is a copy and copies are where obligations get lost. And a deletion request against the source has to be answerable for the derivatives, which is only possible if the derivation is recorded — otherwise the honest answer is that nobody knows how many copies exist or where.

This is the strongest practical argument for generating rather than deriving. A synthesised fixture has no provenance obligations at all, needs no special handling, and can be regenerated from its seed forever. The extra effort of writing a generator, measured against the ongoing cost of tracking a derivative through every cache and clone it reaches, is usually recovered within the first quarter.

Conclusion

A fixture without an identity is a test without a subject. Content-addressing over geometry, CRS, attributes and precision — excluding everything environmental — gives a stable identifier; a provenance record naming the generator, its configuration, its seed, the engine versions and the defect class makes it regenerable and, eventually, retirable. Together they turn the question “did the data change?” from an investigation into a comparison, which is the whole contribution this layer makes to test data generation and mocking strategies.