Caching GDAL/PROJ Wheels in GitHub Actions

9 min read

The fast pre-merge tier of a spatial gate only stays fast if installing the geometry stack is fast, and GDAL, PROJ and their bindings are among the heaviest wheels in the Python ecosystem. This guide sits beneath GitHub Actions spatial testing and shows how to cache those wheels so a sub-minute gate is not dominated by a cold install, while keeping the cache deterministic enough that it never masks a dependency change. The tension is real: a cache that is too sticky serves stale binaries and undermines the version pinning the containerized GIS test runtimes work depends on, while no cache at all can add 30–60 seconds to every push.

Why the install dominates a fast gate

An uncached pip install geopandas on a runner resolves and downloads Shapely, pyproj, fiona and pyogrio — each carrying bundled or linked GEOS, PROJ and GDAL binaries — then compiles nothing but still writes tens of megabytes. On a warm public runner the download and unpack routinely take longer than the spatial assertions themselves. Because the pre-merge tier’s whole value is feedback in under a minute, the install is the single biggest lever, and caching the resolved wheels turns a repeated 40-second cost into a few seconds of restore.

What the key must contain, and what it must not

A cache is only as safe as its key, and a spatial dependency cache has more inputs than most. Every property that could change the installed bytes has to appear in the key, or a restore will serve an environment that does not match the lockfile — which is worse than no cache, because it looks like a working pin.

Cache key inputs: required and forbidden Two lists. The required inputs are a hash of the lockfile, so any dependency change invalidates the entry; the runner operating system and CPU architecture, so a wheel built for one platform is never restored on another; the Python version with its ABI tag, since compiled wheels are ABI-specific; and a manual bust counter that lets a human invalidate everything without editing the lockfile. The forbidden inputs are the branch name, which causes every branch to miss and refill the cache; a date stamp, which expires entries that were still valid; and the commit SHA, which guarantees a miss on every single run. A closing note identifies a missing architecture component as the reason an ARM runner can restore an x86 wheel set and fail at import. Must be in the key hash of the lockfilerunner OS + architecturePython version + ABI tagmanual bust counter any dependency change invalidatesa wheel is platform-specificcompiled wheels are ABI-specificlets a human invalidate everything Must not be in the key branch namea date stampthe commit SHA every branch misses and refillsexpires entries that were still validguarantees a miss on every run A key missing the architecture is how an ARM runner restores an x86 wheel set and fails at import — with a message about a missing symbol, not about a cache.

The manual bust counter is worth including even though it is never needed in theory. In practice a cache occasionally holds something wrong — a partially-written entry, a wheel built from a source that has since been yanked — and without a way to invalidate everything, the only remedies are editing the lockfile pointlessly or waiting for expiry. One number in the key turns that into a one-line change.

Cache key reference

Key ingredient Why it belongs in the key Effect if omitted
Lockfile hash (requirements.lock) Cache must invalidate when a dependency changes Stale wheels served after an upgrade
Python version Wheels are ABI-specific per minor version Wrong-ABI wheel restored, import errors
Runner OS manylinux vs macOS wheels differ Platform-mismatched cache hit
Cache “generation” salt Manual bust for a poisoned cache No way to force a clean rebuild

Step-by-step implementation

The pattern below targets actions/cache@v4 and a hash-pinned lockfile, producing a cache that hits when dependencies are unchanged and rebuilds cleanly when they change.

Step 1 — Produce a hash-pinned lockfile

Resolve dependencies once and commit a lockfile with hashes, so every install is byte-identical and the cache key has something stable to hash.

pip install pip-tools
pip-compile --generate-hashes -o requirements.lock pyproject.toml

Step 2 — Key the cache on the lockfile

      - name: Cache spatial wheels
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ runner.os }}-py3.12-${{ hashFiles('requirements.lock') }}-g1
          restore-keys: |
            pip-${{ runner.os }}-py3.12-

The trailing -g1 is a manual generation salt: bump it to -g2 to force a clean rebuild if a cache is ever poisoned. The restore-keys fallback lets an unchanged-OS run reuse most of a previous cache even when the lockfile changed, so an upgrade downloads only the deltas.

Step 3 — Install with hashes, from cache

      - name: Install spatial stack
        run: pip install --require-hashes -r requirements.lock

--require-hashes ties the install to the exact artifacts the lockfile pins, so a cache hit cannot smuggle in a different wheel than the one you resolved.

Step 4 — Prebuild a wheelhouse for private or source builds

If you build GDAL/PROJ bindings from source (for an exact version not on PyPI), build them once into a wheelhouse artifact and cache that directory, so downstream jobs install from local wheels instead of recompiling.

      - name: Restore wheelhouse
        uses: actions/cache@v4
        with:
          path: wheelhouse
          key: wheelhouse-${{ runner.os }}-gdal3.9.2-proj9.4.1
      - run: pip install --no-index --find-links=wheelhouse geopandas

Caching the wheel, not the environment

There are two things a job can restore, and confusing them produces most cache-related surprises. Restoring the downloaded artefacts — the wheel files themselves — leaves the installation step to run normally, so the environment is always built the same way and only the network is skipped. Restoring the installed environment skips the installation too, which is faster and considerably more fragile: an installed tree carries absolute paths, compiled extensions bound to a specific interpreter, and anything a post-install step wrote.

Cache the downloads, or cache the installed tree Two restore strategies compared across a three-step pipeline of fetch, install and run. In the first, the wheel download cache is restored so the fetch step is skipped while the install step still executes, producing an environment built identically every time. In the second, the installed site-packages tree is restored so both fetch and install are skipped, which is faster but carries absolute paths and compiled extensions bound to a particular interpreter build, meaning a runner image change or a Python patch bump can yield an environment that imports successfully and behaves differently. A recommendation marks the first as the default and the second as acceptable only when the cache key includes the exact interpreter build. Cache the downloads — default fetch skipped — restored install runs every time run environment built the same way on every run Cache the installed tree — faster, riskier fetch skipped install skipped too run absolute paths · ABI-bound extensions travel with it Cache downloads by default. Cache an installed tree only when the key includes the exact interpreter build — and expect to bust it more often. The failure mode of the second is an environment that imports cleanly and behaves differently, which is the hardest kind to notice.

There is a third option worth knowing for the awkward cases: build a wheelhouse once, as its own scheduled job, and publish it as an artefact or to an internal index. This suits private packages and anything that has to be compiled from source, because it moves the expensive build out of the gate entirely and turns every subsequent install into a fast, offline operation against known bytes. It costs a job and a little plumbing, and it is the only arrangement that makes a source-built spatial dependency compatible with a five-minute gate.

Build once on a schedule, install everywhere from the wheelhouse A scheduled job compiles the source-built spatial dependency once and publishes the resulting wheels into an internal index or artefact store. Three subsequent gate runs are shown, each installing from that wheelhouse offline with no compiler present and no fetch from a public index. Beneath, the alternative is drawn: without a wheelhouse, every gate run performs the compile itself, which is annotated as incompatible with a five-minute pre-merge budget. scheduled build job compiles once, weekly wheelhouse internal index or artefact gate run — installs offline, no compiler gate run — installs offline, no compiler gate run — installs offline, no compiler Without a wheelhouse gate run — compiles from source gate run — compiles from source gate run — compiles from source Every run pays the compile, which does not fit inside a five-minute budget — so the gate gets moved to nightly and stops gating.

Verification pattern

Confirm the cache both hits and stays correct: check the Actions log for a Cache restored line, and assert the installed engine versions match what the lockfile intends, so a stale cache surfaces immediately rather than as a mysterious tolerance failure downstream.

python -c "import shapely, pyproj; print(shapely.geos_version, pyproj.proj_version_str)"
# Expect the exact GEOS/PROJ the lockfile pins, e.g. (3, 12, 1) 9.4.1

Failure modes and edge cases

  1. Cache key without the lockfile hash. Keying on the branch or a static string serves the same wheels forever; a dependency bump is silently ignored until the salt changes.
  2. Cross-OS restore-key bleed. A restore-keys prefix that omits runner.os can restore Linux wheels onto a macOS job; keep the OS in both the key and the fallback.
  3. Python minor-version mismatch. Upgrading the runner from 3.11 to 3.12 without the version in the key restores wheels built for the wrong ABI, producing import-time failures.
  4. Poisoned cache after a bad build. A wheelhouse cached from a broken source build sticks until you bump the generation salt — always keep a manual bust mechanism.
  5. pip cache vs installed site-packages. Caching ~/.cache/pip avoids re-download but still re-installs; for the very fastest gate, prefer a container image that ships the stack pre-installed, per the containerized runtimes approach.

Measuring whether the cache is helping

A cache is easy to configure and easy to leave misconfigured, because a broken cache looks exactly like a working one apart from the duration. Record two numbers from every run: the restore result, hit or miss, and the elapsed time of the install step. A cache with a low hit rate is usually keyed on something that changes too often; one with a high hit rate and no time saving is restoring something the install step does not use.

Both failures are common and both are invisible without the measurement. The second is the more wasteful, since it costs the restore time on every run and returns nothing — and it typically comes from caching a directory the package manager stopped using after an upgrade.

A last operational note: restore keys are a fallback, not a guarantee. A partial-match restore serves an older entry whose contents may not satisfy the current lockfile, and the install step then quietly reconciles the difference. That is usually fine and occasionally not, so treat a restore-key hit as a signal worth recording rather than as an equivalent outcome to an exact hit.

Conclusion

Caching GDAL/PROJ wheels keeps the pre-merge tier fast without sacrificing determinism: key the cache on a hash-pinned lockfile plus the OS and Python version, install with --require-hashes, keep a manual generation salt for poisoning, and verify the restored engine versions on every run. Done this way, the cache saves seconds on every push and never serves a wheel your lockfile did not intend. For the surrounding workflow, return to GitHub Actions spatial testing.

Measure the hit rate and the saved seconds before deciding the cache is finished; both are cheap to record and neither is visible otherwise.

The overall aim is narrow: keep the install off the critical path without letting it become a source of ambiguity about what was installed. A cache that is keyed correctly, measured, and bustable meets that; anything looser trades a few seconds for a class of failure that is very hard to attribute.