Setting Runtime Budgets for Spatial Test Suites

9 min read

A runtime budget is a decision about what a lane may cost, made before the lane grows rather than after somebody complains. This guide sits beneath performance benchmarking spatial suites and covers the practical mechanics: choosing the number from how people behave rather than from what the suite currently takes, allocating it across the stages a spatial gate actually has, deciding which checks run against a sample, and enforcing it through a trend alert instead of a per-test assertion.

The budget’s purpose is not efficiency. It is to keep the pre-merge lane inside the window in which an engineer waits for the result, because a gate that is read after merging has stopped being a gate.

Root cause: the budget is set by behaviour, not by engineering

There are three durations that matter, and none of them is a property of the code. Below roughly five minutes an engineer waits and acts on the result. Between five and fifteen they context-switch and come back. Past fifteen they start the next task, and whatever the gate says arrives as an interruption rather than as feedback.

That means the budget is chosen first and the suite fitted to it, not the other way round. A team that measures the current runtime and declares it the budget has described the status quo; a team that picks four minutes and then decides what fits has made an engineering decision with consequences.

Three bands, and where a gate stops working A time axis divided into three behavioural bands. The first, below about five minutes, is labelled as the band in which an engineer waits for the result and acts on it immediately, so the gate functions as feedback. The second, between five and fifteen minutes, is labelled as the band in which the engineer context-switches to something else and returns, so the result arrives late but is still mentally connected to the change that caused it. The third, past fifteen minutes, is labelled as the band in which another task has already begun and the gate's result arrives as an interruption rather than as feedback. A note states that the budget is chosen from these bands and the suite fitted to it, rather than the current runtime being adopted as the budget. waits for it the gate is feedback context-switches, returns late, but still connected has started something else arrives as an interruption ~5 min ~15 min Pick the band first and fit the suite to it. Measuring the current runtime and calling it the budget describes the status quo and commits to nothing. The number is a product decision about engineer attention, not a property of the code. Everything below is about what fits inside it once the number is chosen.

Allocation reference

A four-minute pre-merge budget for a spatial gate divides roughly as follows. The numbers are a starting point; the shape is the durable part, because it puts the cheap high-yield checks on everything and represents the expensive ones by a sample.

Stage Share Runs against Deferred to nightly
Container start ~30 s — Image rebuild
Dependency restore ~20 s Cached, keyed on a lockfile Fresh resolution
Schema and CRS contract ~15 s Every feature —
Geometry validity ~60 s Every feature —
Topology rules ~90 s One coherent spatial subset Full-coverage dissolve
Cross-format parity ~45 s Primary output format only Every output format
Service-level checks 0 Nothing Tiles, routing, joins at volume

Two features of that table are worth stating explicitly. The cheapest checks run against everything, because their cost is linear and their yield is high. The expensive ones are represented by a coherent sample rather than omitted — coherent because a random sample of features destroys adjacency and makes topology checks meaningless.

Step-by-step implementation

Step 1 — Measure the current shape before changing anything

The allocation only makes sense against a measurement, and the measurement is one line per stage.

# conftest.py — record per-stage duration without asserting on it
import time, json, os
from collections import defaultdict

STAGE_TIME = defaultdict(float)

def record_stage(name: str):
    class _Timer:
        def __enter__(self): self.t = time.perf_counter(); return self
        def __exit__(self, *exc): STAGE_TIME[name] += time.perf_counter() - self.t
    return _Timer()

def pytest_sessionfinish(session, exitstatus):
    Path(os.environ.get("STAGE_REPORT", "stages.json")).write_text(
        json.dumps(dict(STAGE_TIME), indent=2, sort_keys=True))

Step 2 — Enforce the lane, not the test

A per-test timeout catches a hang; the lane budget catches accumulation. Both are useful and they are different mechanisms.

      - name: Spatial gate
        timeout-minutes: 6            # a hard stop, above the 4-minute budget
        run: pytest -q -m "not slow" --timeout=60

The per-test timeout guards against one test hanging forever; the job timeout guards against the lane as a whole. Setting the job timeout somewhat above the budget leaves room for ordinary variance while still bounding the worst case.

Step 3 — Alert on the trend, not on the limit

The limit fires when the lane is already unusable. The trend fires while there is time to act.

def test_lane_duration_trend(recent_durations):
    """recent_durations: last 20 runs of the pre-merge lane, in seconds."""
    baseline = sorted(recent_durations[:10])[5]     # median of the older half
    recent = sorted(recent_durations[10:])[5]       # median of the newer half
    growth = (recent - baseline) / baseline
    assert growth <= 0.25, (
        f"pre-merge lane median grew {growth:.0%} over the window "
        f"({baseline:.0f}s → {recent:.0f}s) — budget is 240s"
    )

Step 4 — Make the sample coherent, and say so

A topology check over a random subset is meaningless because adjacency is destroyed. Sample by tile or administrative area, and record what was sampled.

@pytest.fixture(scope="session")
def coverage_subset(full_coverage, request):
    """One coherent area, not a random sample — adjacency must survive."""
    area = os.environ.get("SAMPLE_AREA", "district-07")
    subset = full_coverage[full_coverage["district"] == area]
    request.config.stash["sampled_area"] = area
    request.config.stash["sampled_features"] = len(subset)
    return subset
Random sampling destroys adjacency; coherent sampling preserves it Three panels of a parcel coverage. The full coverage shows parcels tiling an area with every boundary shared between neighbours. The randomly sampled version retains scattered individual parcels that no longer touch one another, so gap and overlap checks find nothing to check and report success vacuously. The coherently sampled version retains one contiguous district in which every shared boundary is preserved, so the same checks measure exactly what they would measure over the full coverage, only over less of it. A closing note records that the identity of the sampled area must itself be recorded, or two runs sampling different areas are not comparable. Full coverage every boundary is shared Random sample nothing is adjacent to anything gap and overlap checks pass vacuously Coherent sample one contiguous district the checks measure what they should Record which area was sampled. Two runs over different districts are not comparable, and a sample that silently rotates makes the series meaningless. A vacuously passing topology check is worse than a slow one — it reports coverage it does not have.

Verify the fix

Run the lane and confirm the stage breakdown adds up to what the budget assumed:

pytest -q -m "not slow" && jq 'to_entries | sort_by(-.value) | .[:5]' stages.json

The top five stages should be the ones the allocation table anticipated. A stage that dominates and is not in the table is the finding — usually a fixture that is rebuilt per test, which is the cost described under performance benchmarking spatial suites.

When the budget cannot be met

Sometimes the suite genuinely does not fit, and there are only four honest responses. Choosing among them explicitly is much better than the default, which is to let the lane grow until somebody stops using it.

Move a check to the scheduled tier. The cheapest response and the one with a real cost: the check no longer gates, so a regression it would have caught reaches the default branch and is found the next morning. Acceptable for expensive checks whose failures are rare.

Reduce the sample. Halve the coherent subset, and record that the coverage is now smaller. This preserves the check’s presence in the gate while reducing its sensitivity, which is a genuine trade and should be recorded as one.

Make the check faster. Usually a fixture scope or an index, and usually the right answer when the measurement has not been taken yet. It is also the response most often assumed to be impossible without measuring.

Raise the budget. Legitimate when the previous number was arbitrary and the behavioural band has room — moving from three minutes to four costs little. Illegitimate as a repeated response, because each increment is small and the accumulation is what puts the lane past fifteen minutes.

The one response that is not on the list is to leave the lane growing and say nothing. That is what happens by default, and it converts a gate into a report over the course of a year without anybody deciding to.

Four responses to a suite that does not fit Four explicit responses when a suite exceeds its budget, each with its cost. Moving a check to the scheduled tier is the cheapest to implement and costs gate coverage, since a regression it would have caught now reaches the default branch and is found the following morning. Reducing the coherent sample keeps the check in the gate and costs sensitivity, which is a genuine trade that should be recorded. Making the check faster costs engineering time and nothing else, and is usually the right answer when no measurement has yet been taken. Raising the budget costs engineer attention and is legitimate as a one-off when the previous number was arbitrary, but corrosive as a repeated response. A fifth row records doing nothing as the default that silently converts a gate into a report over the course of a year. RESPONSE COSTS move it to the scheduled tier gate coverage — the regression reaches the branch reduce the coherent sample sensitivity — record it as a trade make the check faster engineering time — usually right if unmeasured raise the budget attention — fine once, corrosive as a habit do nothing — the default, and the one that converts a gate into a report over a year without anyone deciding to

Reviewing the budget on a cadence

A budget set once and never revisited becomes either an obstacle or a fiction. A short quarterly review keeps it honest and takes about twenty minutes.

Three questions cover it. Is the lane still inside the band? The measurement answers this directly, and if the lane has drifted past the chosen band the review is the moment to choose one of the four responses rather than to notice a year later. Has the allocation shifted? A stage that was ten per cent of the budget and is now forty has changed without anyone deciding, which is worth understanding whether or not the total moved. Is the sample still coherent and still representative? A district chosen two years ago may no longer contain the feature types the pipeline now handles, in which case the topology check is running against something unrepresentative and reporting confidence it has not earned.

The review is also the natural place to record deliberate increases. A budget raised from three minutes to four because the suite now checks something it previously did not is a good trade, and writing that down stops a future reader from treating the step as an unexplained regression to be recovered.

Failure modes and edge cases

  1. Per-test time assertions. They fail on a busy runner and remove the correctness signal along with the timing one. Use a per-test timeout as a hang guard and a lane budget for accumulation.
  2. A budget on the wrong lane. Constraining the nightly tier limits something nobody waits for while the pre-merge lane grows unwatched.
  3. A random sample for a topology check. Adjacency is destroyed, the check passes vacuously, and the gate reports coverage it does not have.
  4. A rotating sample. If the sampled area changes between runs, the durations and the results are both incomparable. Fix it, and change it deliberately.
  5. Ignoring the fixed costs. Container start and dependency restore are often half the budget and are the easiest to reduce, yet optimisation effort usually goes to the checks.
  6. A job timeout equal to the budget. Ordinary variance then fails the job. Set the hard stop above the budget and alert on the trend below it.

Conclusion

A runtime budget is a decision about engineer attention expressed in seconds. Choosing the number from the behavioural bands, allocating it so cheap checks run against everything and expensive ones against a coherent sample, guarding hangs with a per-test timeout and accumulation with a trend alert keeps a spatial gate inside the window where it functions — which is the operational half of performance benchmarking spatial suites.