Managing Large Spatial Fixtures with Git LFS

9 min read

A spatial fixture outgrows a repository sooner than a textual one, and the transition is unpleasant because git handles it silently until it does not. This guide sits beneath fixture versioning and provenance and covers the practical mechanics of large-file storage for spatial data: which files genuinely belong there, why a committed GeoPackage costs more than its size suggests, how to migrate history without rewriting every developer’s clone, and how to make CI fetch only what a job needs.

The decision to reach for large-file storage should come after the decision to generate rather than store. Everything here applies to the residue — the hand-crafted pathology, the incident regression, the production-shaped extract — that genuinely cannot be produced from a seed.

Root cause: why a binary fixture is expensive in git

Git stores every version of every file forever, and it stores them by content. For a text file that is cheap: successive versions share most of their content and pack efficiently. For a spatial binary it is not, and three properties make it worse than the file size implies.

Small edits produce entirely new objects. Rewriting one attribute in a GeoPackage changes internal page layout, so the new version shares almost nothing with the old. Ten revisions of a 40 MB fixture is 400 MB in the repository, permanently, for every clone.

Compression does not help. Most spatial formats are already compressed internally, so git’s packing achieves little. A 40 MB GeoPackage packs to roughly 40 MB.

History cannot be pruned selectively. Once the object is in history, removing it requires rewriting history — which invalidates every existing clone and every reference to a commit hash. The cost of a mistake here is not the disk space; it is the migration.

Repository growth: committed binary versus generator A chart of cumulative repository size against the number of fixture revisions. The committed binary line rises in steep steps, adding close to the full forty megabytes at each revision because a small edit changes the internal page layout and the format is already compressed so packing achieves little; after ten revisions it reaches roughly four hundred megabytes, all of which every clone pays for permanently. The generator line stays almost flat, since each revision is a few kilobytes of source and the fixture is reconstructed on demand from a seed. A note records that removing objects from history requires rewriting it, which invalidates every existing clone and every reference to a commit hash. repo size fixture revisions committed binary — ~40 MB per revision generator script — a few kB per revision 1 4 7 10 ~400 MB Every clone pays the full history, forever. Removing an object requires rewriting history, which invalidates every existing clone and commit reference. The cost of the mistake is not disk space — it is the migration.

Decision reference

Fixture Store as Because
Seeded generator output Nothing — generate it Identity is provable from inputs
Hand-crafted pathology, < 100 kB Committed directly Must be reviewable as an artefact
Incident regression, < 1 MB Committed directly Its value is that it is exactly this data
Production-shaped extract Large-file storage Too big to commit, too specific to generate
Multi-gigabyte reference layer Content-addressed store Shared across suites; dedupes
Anything derived from real data Neither, without review Carries the source’s obligations

Step-by-step implementation

The commands target git-lfs 3.x.

Step 1 — Track by directory, not by extension

Tracking *.gpkg globally captures every GeoPackage in the repository including the small reviewable ones, which is usually not what is wanted. Track the directory that holds large fixtures instead.

git lfs install
git lfs track "tests/fixtures/large/**"
git add .gitattributes

The attributes file must be committed before the fixtures, or the first fixture lands in ordinary git history and the migration problem starts immediately.

Step 2 — Verify the pointer, not the file

A tracked file is replaced in git by a small pointer. Confirming that is what proves the tracking pattern matched.

git add tests/fixtures/large/parcels.gpkg
git cat-file -p :tests/fixtures/large/parcels.gpkg | head -3
# version https://git-lfs.github.com/spec/v1
# oid sha256:9f2a...
# size 41943040

If the output is binary rather than three lines of text, the pattern did not match and the file is going into ordinary history.

Step 3 — Fetch selectively in CI

The default clone fetches every tracked object, which for a repository with many fixtures is most of the cost of the job. Fetch only what the job reads.

- uses: actions/checkout@v4
  with:
    lfs: false                      # do not pull everything
- run: |
    git lfs pull --include="tests/fixtures/large/parcels.gpkg"

Step 4 — Assert the fixture is real before using it

A missing large-file object materialises as a pointer file, and a spatial reader given a pointer produces a confusing parse error rather than a clear one.

from pathlib import Path
import pytest

def assert_materialised(path: Path):
    head = path.read_bytes()[:64]
    if head.startswith(b"version https://git-lfs"):
        pytest.exit(
            f"{path} is an unfetched LFS pointer, not data — run 'git lfs pull'",
            returncode=3,
        )
Pointers in the repository, objects in the store Two arrangements compared. Without large-file tracking, the repository itself holds the complete binary at every revision, so a clone transfers all of them regardless of which the job needs. With tracking, the repository holds only a small text pointer per revision while the actual objects live in a separate store; a clone transfers just the pointers unless a fetch is explicitly requested, and a selective fetch pulls only the objects a particular job will read. A warning records that a missing object materialises as a pointer file on disk, which a spatial reader reports as a parse error rather than as a missing fixture, so an explicit check for the pointer signature is worth having. Untracked repository holds every revision, in full a clone transfers all of them whether the job needs them or not Tracked repository holds a pointer per revision objects live in a separate store a clone transfers pointers only clone cost = the whole history grows with every fixture revision, forever clone cost = kilobytes; fetch what the job reads git lfs pull --include=… A missing object appears on disk as a pointer file — and a spatial reader reports that as a parse error, not a missing fixture. Check the first bytes for the pointer signature and fail with a message that says what to run.

Verify the fix

Confirm the repository is genuinely smaller and the fixtures still load:

git count-objects -vH | grep size-pack
pytest -q tests/ -k fixture

A pack size that has not changed after tracking means the fixtures were committed before .gitattributes, and the objects are already in history. That is the case that requires a migration rather than a configuration change.

Migrating history that already contains fixtures

When large objects are already in history, tracking them going forward does not shrink anything — the old objects remain. Two paths exist and they differ in disruption rather than in outcome.

Rewrite history. Every commit containing a large object is rewritten so the object becomes a pointer. The repository shrinks immediately and completely, and every existing clone becomes invalid: every developer must re-clone, every open branch must be rebased, and every recorded commit hash — in tickets, in deployment records, in release notes — no longer resolves. This is correct when the repository is small enough that a coordinated re-clone is practical.

Leave history alone and track from now. New revisions become pointers, old objects stay. Nothing breaks, nobody re-clones, and the repository keeps its existing size without growing further. This is usually the right choice for an established repository, because the disruption of the alternative is larger than the disk it recovers.

The decision rests on one question: is the repository still growing at a rate that matters? If yes, tracking from now solves the future and the past is a fixed cost. If the past is itself the problem — a repository nobody can clone in reasonable time — a rewrite is the only remedy, and it should be scheduled like a migration rather than done on a Friday.

Rewrite, or track from now Two migration options compared. Rewriting history converts every past commit containing a large object so that the object becomes a pointer; the repository shrinks immediately and completely, but every existing clone is invalidated, every open branch must be rebased, and every commit hash recorded in tickets, deployment records or release notes stops resolving. Tracking from now leaves history untouched so nothing breaks and no developer must re-clone, but the objects already committed remain and the repository keeps its current size while stopping further growth. A closing line gives the deciding question: whether the past is itself the problem, or merely a fixed cost that has already been paid. Rewrite history + repository shrinks completely, at once + no residual objects anywhere − every clone becomes invalid − every open branch must be rebased − recorded commit hashes stop resolving Track from now + nothing breaks, nobody re-clones + growth stops immediately − existing objects stay in history − the current size is permanent − a clone still pays for the past The deciding question: is the past itself the problem, or a fixed cost already paid? For most established repositories the second option is correct, because the disruption of a rewrite exceeds the disk it recovers.

Keeping the tracked set small

The most effective thing that can be done about large fixtures is to have fewer of them, and the review question that achieves it is short: could this be generated? Applied to each candidate before it is added, it removes most of what would otherwise accumulate.

Three categories genuinely cannot be generated and should be tracked without argument. A hand-crafted pathology whose exact vertex arrangement is the point — a geometry that provoked a specific engine bug — cannot be reproduced from a description. An incident regression is valuable precisely because it is the data that broke production, and synthesising something similar loses the property that made it worth keeping. A production-shaped extract used for performance work has statistical properties that a simple generator will not reproduce faithfully.

Everything else is usually generatable with a day’s work on a generator that then serves the whole suite. The comparison worth making is not “generator versus one fixture” but “generator versus every fixture of that shape the team will ever need”, and on that basis the generator almost always wins.

A periodic review helps too. Listing the tracked objects by size once a quarter, and asking of the largest few whether they are still referenced by any test, typically finds one or two that outlived their purpose — and while removing them does not shrink history, it does stop them being fetched.

Failure modes and edge cases

  1. Tracking configured after the first commit. The fixture is already in ordinary history and tracking it now changes nothing. Commit .gitattributes first, always, and verify with git cat-file that the staged content is a pointer.
  2. A clone without the extension installed. Developers who have not installed the tooling receive pointer files and see parse errors from every spatial reader. The materialisation check turns that into a message naming the command to run.
  3. CI pulling everything. The default checkout fetches all objects, so a repository with fifty fixtures pays for fifty in every job. Disable the automatic pull and fetch by pattern.
  4. A fixture edited in place. Every save creates a new object in the store, so an iteratively-developed fixture accumulates revisions nobody needs. Develop it outside the repository and commit once.
  5. Storage quota exhausted mid-migration. A rewrite pushes every historical object at once, which can exceed a quota and leave the repository in a partially-migrated state. Check the projected size before starting.
  6. Assuming tracked means versioned. Large-file storage versions the pointer, and the object store may have its own retention. A fixture whose object has been pruned is unrecoverable even though its pointer is still in history.

Conclusion

Large-file storage is the right tool for the residue of fixtures that genuinely cannot be generated, and the wrong tool for everything else. Tracking by directory before the first commit, verifying that pointers rather than binaries reach history, fetching selectively in CI, and checking materialisation before a reader touches the file keeps repository workflow intact — while the underlying decision, made in fixture versioning and provenance, remains to generate wherever generation is possible.