Recording Fixture Provenance Metadata in CI

9 min read

A provenance record is what lets a failing run answer the only question that matters at 9 a.m.: did the code change, did the data change, or did the environment change? This guide sits beneath fixture versioning and provenance and shows how to emit that record from a pytest suite, what it must contain to be useful, and the one detail teams get wrong that makes the whole thing worthless — writing it only when the run fails.

The record is small, cheap and unglamorous. Its value is entirely in comparison, which is why it has to exist for the passing runs too.

Root cause: attribution needs a baseline

A failing run in isolation says nothing about cause. The code, the data and the runtime all changed at some point between the last success and now, and without a record of what each was, the investigation starts by reconstructing them — from a build log if one survives, from a container tag if it was pinned, from a fixture directory that may have been regenerated since.

With a record on both runs the comparison is mechanical: three fields differ or they do not, and which ones differ names the cause. That is the entire mechanism, and its only prerequisite is that the successful run wrote something down.

A record on failure alone has nothing to compare against Two timelines of runs. In the first, only failing runs emit a provenance record, so the sequence of passing runs leaves no trace and the eventual failure has a record with nothing to compare it to; the investigation must reconstruct the previous state from build logs, container tags and a fixture directory that may since have been regenerated. In the second, every run emits a record regardless of outcome, so the failing run is compared field by field against the last passing one and the differing field names the cause immediately. A note records that the cost of writing on success is a few kilobytes per run. Record on failure only passpasspasspassFAIL no record no record record nothing to compare against — reconstruct from build logs and container tags Record on every run recordrecordrecordrecordrecord compare these two — the differing field names the cause The cost of recording on success is a few kilobytes per run, and it is the only thing that makes the failing record interpretable.

Record reference

Field Source Answers
code_revision CI environment Which commit ran
fixture_ids Computed per fixture Which spatial state ran
generator_revision Git revision of the generator module Which implementation produced it
config_hash Hash of the generation config Which parameters
seed The generator’s seed Which draw
geos_version / proj_version Imported libraries, at runtime Which engines
proj_data_version The grid package Which datum shifts
image_digest CI environment Which runtime
outcome pytest exit status Whether this is a baseline or a failure

Step-by-step implementation

The recorder targets pytest 7+ and writes a JSON artefact regardless of outcome.

Step 1 — Collect the environment once per session

Reading versions from the libraries that were actually imported matters — a shell command may resolve to a different installation than the one the tests use.

# conftest.py
import json, os, platform
from pathlib import Path

def _engine_versions() -> dict:
    import shapely, pyproj, rasterio
    return {
        "geos_version": shapely.geos_version_string,
        "proj_version": pyproj.proj_version_str,
        "gdal_version": rasterio.__gdal_version__,
        "python": platform.python_version(),
    }

Step 2 — Register fixture identities as they are used

A session-scoped collector lets any fixture announce itself, so the record covers exactly what the run touched rather than everything that exists.

import pytest

@pytest.fixture(scope="session")
def provenance() -> dict:
    return {"fixtures": {}, "engines": _engine_versions(),
            "code_revision": os.environ.get("GIT_SHA", "unknown"),
            "image_digest": os.environ.get("IMAGE_DIGEST", "unknown")}

@pytest.fixture
def parcels(provenance, tmp_path):
    gdf = build_parcels(seed=20260811)
    provenance["fixtures"]["parcels"] = {
        "fixture_id": fixture_id(gdf, key="parcel_id", grid_size=0.001),
        "generator": "fixtures.parcels:build_parcels",
        "seed": 20260811,
        "config_hash": config_hash(),
    }
    return gdf

Step 3 — Write the record whatever happened

The sessionfinish hook runs on success and on failure, which is the property that makes the baseline exist.

def pytest_sessionfinish(session, exitstatus):
    prov = session.config._store.get("provenance", None) or {}
    prov["outcome"] = "pass" if exitstatus == 0 else f"fail:{exitstatus}"
    out = Path(os.environ.get("PROVENANCE_PATH", "provenance.json"))
    out.write_text(json.dumps(prov, indent=2, sort_keys=True))

Step 4 — Publish it as an artefact, always

A record written into a container that is then discarded has recorded nothing. The publish step must run regardless of the job’s outcome.

      - name: Publish provenance
        if: always()                  # the failing run is the one that needs it
        uses: actions/upload-artifact@v4
        with:
          name: provenance-$
          path: provenance.json
          retention-days: 90
Assembling and publishing the record A four-stage sequence across a test session. At session start the engine versions, the code revision and the container image digest are collected once. During the run, each fixture registers its own identity, generator name, seed and configuration hash as it is constructed, so the record covers exactly what the run touched. At session finish the outcome is appended and the whole record is written to disk. A publish step configured to run under all conditions uploads it as a retained artefact. Two failure conditions are marked: writing the record only when the run fails, which removes the baseline, and publishing only when the job succeeds, which discards precisely the record that was needed. session start engines, revision, image each fixture registers id, generator, seed, config session finish outcome appended, written published, always retained as an artefact Break 1: write only on failure the baseline never exists, so the record is uninterpretable Break 2: publish only on success the run that needed it is exactly the one discarded Both are one-word conditions in a workflow file, and both silently remove the entire value of the record.

Verify the fix

Confirm the record survives a failing run, which is the case that matters:

pytest -q tests/ ; ls -l provenance.json && jq '.outcome, .engines' provenance.json

The file must exist with an outcome of fail:1 after a deliberate failure. If it exists only after a success, the write is in the wrong hook; if it does not exist at all in CI, the publish step is conditioned on success.

Reading two records against each other

The comparison is worth automating, because the manual version is a diff of two JSON files and reviewers reliably skip it. A short script that takes the failing run’s record and the last passing one, and reports which of the three groups differ, turns the whole mechanism into one line of output in the job summary.

The three groups are the ones described in the parent page: code, data and environment. Reporting them by group rather than field by field is what makes the output actionable — a reader does not need to know that proj_data_version moved, only that the environment changed and the image owner should look. Fields within a group can be listed underneath for whoever does look.

One refinement is worth adding once the basic comparison works: report which fixtures changed rather than that some did. A run touching a dozen fixtures where one moved is a much narrower investigation than “the data changed”, and the record already contains everything needed to say so.

A grouped comparison, generated automatically A summary produced by comparing a failing run's provenance record against the last passing one. It groups differing fields into three categories. The code group shows no change, with the revision identical. The data group shows a change: one fixture identity moved, its generator revision is unchanged, and its configuration hash differs. The environment group shows no change, with the engine versions and image digest identical. The conclusion printed beneath is that a configuration change altered a single fixture, which narrows the investigation to one parameter instead of the whole suite. provenance diff — run 4812 (fail) vs run 4809 (pass) code — unchanged code_revision identical data — CHANGED fixtures.parcels.fixture_id sha256:9f2a… → sha256:c41b… fixtures.parcels.generator_revision identical fixtures.parcels.config_hash sha256:7a2e… → sha256:b903… environment — unchanged GEOS, PROJ, GDAL, grid package, image digest all identical → a configuration change altered one fixture — investigate that parameter, not the suite.

Making the record useful to a human, not just a script

A JSON artefact is the right storage form and the wrong reading form. Two additions make it usable by whoever opens the failed job first, and both are cheap.

A one-line job summary. Most CI systems allow a job to write a short markdown summary that appears at the top of the run page. A single line stating which of the three groups changed — code, data, environment — puts the answer where nobody has to download anything. When nothing changed but the outcome, that line says so, which is itself informative because it points at nondeterminism.

Fixture identities in the failure message. When an assertion fails, the message should carry the identity of the fixture it ran against. This is the same principle as recording a generator’s seed: a failure that names its input is reproducible from the message alone, while one that names only a rule requires the reader to work out what data was involved.

Neither replaces the artefact. The artefact is what a comparison script reads and what remains available in three months; the summary and the message are what a person reads in the first minute, and the first minute is where most of the wasted time in a failing build is spent.

There is a third addition worth making once the first two exist: publish the record for the default branch’s scheduled run as well as for merge requests. That run is the natural baseline — it has no change under review, so any difference in it is drift rather than intent — and comparing a merge-request failure against it separates “this branch broke something” from “this was already broken” without any further investigation.

Failure modes and edge cases

  1. Writing the record in a teardown that does not run on error. A fixture finaliser can be skipped when a session aborts. Use the session-finish hook, which runs on collection errors too.
  2. Recording versions from a shell command. gdalinfo --version may resolve to a different installation than the one Python imported. Read from the imported modules.
  3. Publishing conditioned on success. The default in most CI systems is to skip subsequent steps after a failure, which discards the record precisely when it is needed. Condition the publish to run always.
  4. A record with no retention. An artefact that expires in a day cannot be compared against a failure that appears a week later. Retain long enough to cover a slow investigation.
  5. Recording every fixture that exists. The record should cover what the run used, not the fixture directory’s contents, or it grows without bound and stops naming anything specific.
  6. Timestamps as the only ordering. Two runs on different runners can have clock skew. Order by the CI run number, and keep the timestamp for humans rather than for comparison.

What not to put in the record

A provenance record grows by accretion unless something stops it, and a bloated record is read by nobody. Three categories are worth excluding deliberately.

Anything derivable. The container image digest identifies the image; listing every package inside it duplicates information the digest already fixes and makes the record unreadable. Record the digest and let anyone who needs the contents resolve it.

Anything that changes on every run regardless. A timestamp, a run identifier, a hostname — these belong in the CI system’s own metadata, and including them in a comparison means every comparison reports differences. Keep the record to fields whose stability is itself informative.

Anything sensitive. A fixture path can disclose a customer name; a query recorded verbatim can contain a coordinate. The record is an artefact that may be retained for months and shared more widely than the run itself, so it should carry identifiers rather than contents. This is the same discipline that keeps raw coordinates out of log lines, applied to a longer-lived artefact.

The test for whether a field belongs is simple: would its difference between two runs tell somebody what to do next? If yes, it belongs. If it would merely be noticed and dismissed, it makes the useful fields harder to find.

Conclusion

A provenance record is a few kilobytes of JSON whose entire value is that it exists on both sides of a comparison. Collecting engine versions from imported modules, letting each fixture register its own identity, writing at session finish regardless of outcome and publishing unconditionally turns “did the data change?” from an investigation into one line of a job summary — completing the bookkeeping that fixture versioning and provenance exists to provide.