Performance Benchmarking Spatial Suites

15 min read

Performance benchmarking for spatial suites is the practice of measuring what validation costs, keeping that measurement stable enough to act on, and turning it into a budget rather than a periodic surprise. It sits beneath CI/CD spatial quality gates because the gate’s usefulness depends on its speed: a correct suite that takes twenty minutes stops being a gate and becomes a report people read after merging.

The central difficulty is that wall-clock time on a shared runner is noisy, and the naive response — putting a timing assertion inside a functional test — produces a flaky suite while measuring almost nothing. Everything here is about measuring quantities that are stable, and gating on the ones that are.

Benchmarks Are Not Tests

A correctness test asserts a property of the output. A benchmark measures a property of the execution. Mixing them produces something that is bad at both jobs, and the failure is specific: a timing assertion inside a functional test makes the test fail on a busy runner, which teaches everyone to re-run it, which removes the correctness signal too.

Keeping them separate means three things in practice. Benchmarks live in their own selection and never run in the fast lane. They run on a runner whose variance is known, ideally a dedicated one, because comparing measurements across heterogeneous machines is comparing noise. And their failures do not block a merge; they raise a trend alert, because a slowdown is almost never attributable to a single change.

A timing assertion cannot sit anywhere useful Two duration distributions compared. On a shared runner the distribution of a spatial test's execution time is broad with a long right tail caused by unrelated work on the machine; a timing threshold tight enough to detect a genuine regression falls inside that distribution and fails on ordinary runs, while a threshold loose enough to avoid false failures sits beyond any regression worth catching. On a dedicated runner with known variance the distribution is narrow, and a threshold can be placed between normal operation and a meaningful regression. The conclusion is that the problem is the runner's variance rather than the idea of measuring. Shared runner — wide distribution tight threshold fails on ordinary runs loose threshold misses every regression worth catching Known-variance runner — narrow a threshold that means something normal operation is tightly bounded a regression is outside it The problem is the runner’s variance, not the idea of measuring. Move the measurement somewhere its distribution is narrow enough to reason about. Then the threshold has somewhere to sit.

What to Measure Instead of Wall Clock

Where a dedicated runner is unavailable — which is most of the time — the answer is to measure quantities that do not depend on the machine. Three are available in spatial work and all three are more diagnostic than duration.

Operation counts. How many geometry predicates were evaluated, how many candidate pairs the index returned, how many rows the database examined. These are exact, machine-independent, and they change only when the algorithm or the data does — which is precisely the regression a benchmark exists to catch.

Peak memory. More stable than time on a shared machine and often the actual constraint. A validation stage whose peak has doubled is processing more than intended, whether or not it got slower.

Complexity, measured over sizes. Run the operation over several input sizes and fit the growth. A change from near-linear to quadratic is unmistakable in that fit and invisible in a single timing, and it is the most valuable regression this whole layer catches.

Quantity Machine-dependent Catches Cost
Wall-clock duration Yes, heavily Everything, unreliably Free
Predicate call count No Algorithmic change An instrumented wrapper
Index candidates returned No A lost or unused index One counter
Rows examined No A plan change Query statistics
Peak resident memory Slightly Growth in working set One sampler
Growth exponent over sizes No A complexity change Several runs

Complexity Is the Regression Worth Catching

A pipeline whose validation is 15 per cent slower is a nuisance. A pipeline whose validation has gone from O(nlog⁡n)O(n \log n) to O(n2)O(n^2) is an outage waiting for the dataset to grow, and the two are indistinguishable at today’s size.

Measuring the exponent is straightforward: run the operation at several input sizes, fit a line to the log of size against the log of cost, and the slope is the exponent. Asserting on it — that the slope stays below, say, 1.3 — catches the class before it matters.

log⁡t=klog⁡n+c⟹t∝nk,assert k≤kmax⁡\log t = k \log n + c \quad\Longrightarrow\quad t \propto n^k, \quad \text{assert } k \le k_{\max}

The most common cause of an exponent change in spatial work is an index that stopped being used: a predicate rewritten in a way the planner cannot reduce to a bounding-box test, or an in-memory join that lost its tree. Both look like a modest slowdown at fixture scale and both are quadratic.

Two curves that agree at fixture scale A chart of cost against input size showing two curves. The near-linear curve rises gently. The quadratic curve rises steeply. At the small input size used by a test fixture the two curves are close together and a single timing measurement cannot distinguish them. At production input size they differ by orders of magnitude. A second panel shows the same two datasets plotted on logarithmic axes, where each becomes a straight line whose slope is its exponent; the slopes are clearly different and the separation is immediate. A note records that the slope is machine-independent, so it can be asserted on a shared runner where a duration cannot. Linear axes — indistinguishable at fixture scale fixture size the two curves are ~2% apart here n² n log n Log axes — the slope is the exponent k ≈ 1.1 k ≈ 2.0 separated immediately, from the same measurements The slope is machine-independent, so it can be asserted on a shared runner where a duration cannot. The commonest cause of a slope change in spatial work is an index that stopped being used.

Budgets Per Lane, Not Per Test

A runtime budget belongs to a lane rather than to an individual test, because what matters is whether the gate stays inside the window in which people wait for it. Per-test budgets produce a large number of small, noisy assertions and no answer to the question anybody is asking.

The lane budgets that work follow the behavioural thresholds described under CI/CD spatial quality gates: the pre-merge lane targets minutes, the scheduled lane targets whatever the schedule permits, and the benchmark lane has no budget at all because its purpose is measurement rather than gating.

Tracking the lane duration as a metric and alerting on the trend catches the accumulation that per-test assertions miss entirely — twenty tests each growing five per cent produce a lane that has doubled while no individual test looks unusual.

Where the Cost Actually Is

Before optimising anything it is worth knowing what a spatial suite spends its time on, because the intuitive answer is usually wrong. Suites feel compute-bound — geometry is expensive, predicates are complicated — and in practice the time is somewhere else.

Four costs dominate, in roughly this order for a typical suite. Fixture construction and I/O is usually the largest, particularly where a fixture that should be session-scoped is being rebuilt per test. Index construction is second, for the same reason: a tree built once per test rather than once per session multiplies with the test count and again with the worker count. Process and connection setup is third — a database connection established per test, an interpreter importing a heavy stack, a container starting. Only fourth do the predicates themselves appear, and they are frequently a small single-digit percentage of the total.

Assumed cost versus measured cost Two horizontal breakdowns of a spatial suite's runtime. The assumed breakdown, which reflects intuition, gives geometry predicates the overwhelming majority of the time with fixture handling, index construction and setup as minor slices. The measured breakdown reverses this: fixture construction and input-output is the largest slice, index construction is the second largest, process and connection setup is third, and the geometry predicates are a small final slice. A closing note records that optimisation effort directed at the predicates therefore addresses the smallest component, while the largest is usually removed by a single fixture scope annotation. What teams assume geometry predicates fixture · index · setup treated as minor What measurement shows fixture build + I/O index construction setup preds the predicates are frequently a single-digit percentage of the total Effort aimed at the predicates addresses the smallest component. The largest is usually one fixture scope annotation away from disappearing. Measure before optimising — the ordering above is a hypothesis, and it is cheap to check.

The practical consequence is an ordering for optimisation work. Check fixture scopes first, because the fix is an annotation and the win is often most of the suite’s runtime. Check index construction second, for the same reason. Look at connection and process setup third. Only then consider the predicates — and by that point the measurement will usually show that they were never the problem.

Making a Measurement Comparable

A benchmark number is only useful against another benchmark number, and two measurements are comparable only if several things held constant. Recording them is what turns a series of runs into a trend rather than a collection of unrelated figures.

The input. A benchmark over a generated fixture must record the generator’s seed and configuration hash, or a change in the data is indistinguishable from a change in the code. This is the same provenance discipline that applies to correctness fixtures, and it matters more here because performance is more sensitive to data shape than correctness is.

The environment. Engine versions, image digest, worker count, and the runner class if the platform exposes one. A benchmark comparing a run on one machine class against another is measuring the machines.

The concurrency. A measurement taken with eight workers and one taken serially are different measurements of different things. Fixing the worker count for the benchmark lane — usually to one — removes an entire source of variance and makes the numbers mean something.

The warm-up state. A first iteration that includes an import, a connection or a cache fill is measuring setup. Most harnesses discard warm-up rounds by default; the important part is knowing whether yours does, because a benchmark that includes a cold import in its mean is dominated by it.

Recording these four alongside every measurement costs a few lines and converts a benchmark from a number that moves for unknown reasons into one whose movement is attributable — which is the same property that makes a correctness failure diagnosable, applied to a different quantity.

Four increments, ordered by return per unit of effort Four steps in building a benchmarking layer, plotted with effort increasing to the right and the return from each shown as a bar. Recording the lane duration needs no instrumentation at all and returns the most, since it is the quantity the behavioural thresholds are expressed in and it catches accumulation that per-test assertions cannot see. Adding operation counts needs light instrumentation and returns a machine-independent signal that can be asserted rather than merely watched. Adding a complexity fit costs several runs per operation and catches the regression class whose cost grows with the data. Adding per-operation timing requires a runner with stable variance and returns least, because the preceding layers have already caught most of what matters. STEP EFFORT RETURN 1 · lane duration one number per run, no instrumentation none catches accumulation nothing else sees 2 · operation counts wrap two or three dominant calls light machine-independent, assertable 3 · complexity fit several sizes per operation moderate catches the scaling class 4 · per-operation timing needs a stable runner high marginal by then

Frequently Asked Questions

Should a benchmark ever fail a build?

Only on a complexity assertion, and even then reluctantly. An exponent that has moved from linear to quadratic is a defect regardless of the machine, so failing on it is defensible. A duration that has grown is a trend, and trends belong in alerts rather than gates because the attribution to a single change is usually wrong.

How many input sizes are needed to fit an exponent?

Four or five spanning at least an order of magnitude. Fewer than four gives a fit that noise dominates; more than about six costs runtime for little extra confidence. The span matters more than the count — three sizes within a factor of two say almost nothing.

What about the cost of the benchmark itself?

Keep it out of the gate entirely and run it on a schedule. A complexity fit over five sizes is inherently several times the cost of one run, which is fine nightly and unacceptable pre-merge. This is the clearest case in the whole discipline for the scheduled tier.

Is profiling worth automating?

Rarely as a gate; often as an artefact. Capturing a profile on the scheduled benchmark run and publishing it means that when the trend alert fires, the evidence already exists rather than needing to be reproduced. Profiling every run is expensive and produces data nobody reads.

How do we benchmark database-side work?

Measure rows examined and the plan rather than elapsed time. A query plan is exact and machine-independent, and a plan change is what a database performance regression actually is — the timing is downstream of it. Asserting that a specific query still uses its spatial index is a stronger and more stable check than any duration.

When Performance Work Is Worth Doing

Not every slowdown deserves attention, and a benchmarking layer that produces work indiscriminately is worse than none. Three questions decide whether a measured regression is worth acting on.

Does it cross a behavioural threshold? A pre-merge lane growing from ninety seconds to two minutes matters far less than one growing from four minutes to six, because the second crosses the point at which people stop waiting. Absolute change is a poor guide; position relative to the threshold is a good one.

Is it a complexity change or a constant factor? A constant-factor regression is bounded — it costs what it costs and will not get worse on its own. A complexity change gets worse as the data grows, which means the cost of deferring it rises over time. That asymmetry is usually decisive.

Who is paying? A slow scheduled job costs machine time; a slow pre-merge lane costs engineer time multiplied by the number of merges. The second is almost always the larger figure, and it is the one that never appears on any infrastructure bill.

The corollary is that some slowdowns should be accepted deliberately and recorded rather than fixed. A validation stage that got 20 per cent slower because it now checks something it previously did not is a good trade, and noting that in the benchmark’s history stops a future reader from treating it as a regression to be recovered.

That record — a short note attached to the point where the series stepped — is worth as much as the measurement itself. Without it, every step in a benchmark series eventually looks like an unexplained degradation, and somebody spends a day rediscovering that it was intentional.

Common Failure Modes and Gotchas

  1. Timing assertions inside functional tests. They fail on a busy runner, teach everyone to re-run, and destroy the correctness signal along with the timing one. Keep the two selections separate.
  2. Benchmarking against a fixture that changes. A measurement over regenerated data is comparing two things at once. Pin the seed and the configuration, and record both alongside the number.
  3. Comparing across runner classes. A platform that mixes machine types silently makes half the series incomparable. Record the class, and break the series when it changes.
  4. Measuring with warm-up included. A mean that includes a cold import or a first connection is dominated by setup and moves whenever the environment does. Confirm the harness discards warm-up rounds.
  5. A complexity fit over too narrow a range. Three sizes within a factor of two produce a slope that noise dominates. Span at least an order of magnitude.
  6. Optimising the predicate first. It is usually the smallest component. Measure the breakdown before choosing where to work, or the effort produces a change nobody can detect.
  7. A budget on the wrong lane. Budgeting the scheduled tier constrains something nobody waits for while the pre-merge lane grows unwatched. Put the budget where the behavioural threshold is.
  8. No baseline retention. A benchmark series with thirty days of history cannot answer a question about a quarterly trend, which is the timescale on which spatial workloads actually grow.

Building the Benchmark Suite Incrementally

A full benchmarking layer is a substantial investment, and most of its value arrives from the first small piece. The order below front-loads the return.

Start with the lane duration. Record how long each CI lane takes, and plot it. This is one number per run, requires no instrumentation at all, and catches the accumulation that individual assertions never see. Teams that add nothing else still get most of the practical benefit from this, because the lane duration is the quantity the behavioural thresholds are expressed in.

Add operation counts next. Instrument the two or three calls that dominate — the predicate evaluations, the index queries, the database round trips — and record the counts. They are machine-independent, so they can be asserted rather than merely watched, and a count that changes without a corresponding code change is a strong and unambiguous signal.

Add the complexity fit for the operations that scale. Not everything needs it; the candidates are the set-level operations whose cost depends on the number of features rather than on a fixed schema. Those are the ones where an index can be silently lost, which is the regression class worth the extra runs.

Add per-operation timing last, and only with a stable runner. By this point the earlier layers have caught most of what matters, and the marginal value of a duration measurement is lower than its variance problem suggests it should be.

Two things are worth avoiding at every stage. Do not benchmark what nobody would change — measuring a library call you have no intention of replacing produces a number with no action attached. And do not let the benchmark suite grow faster than the attention available to read it: five measurements somebody looks at weekly are worth more than fifty nobody opens.

There is a natural home for all of this alongside the quality metrics described under spatial test observability and metrics. Runtime and correctness trends answer different questions and are read by the same people at the same moment, and keeping them on one page means a slowdown that coincides with a quality change is noticed as a single event rather than as two unrelated ones.

Conclusion

Spatial validation gets slower gradually and breaks suddenly, and the two have different causes. Separating benchmarks from correctness tests, measuring machine-independent quantities where a dedicated runner is unavailable, fitting the growth exponent to catch complexity changes before scale exposes them, and budgeting by lane rather than by test keeps the gate inside the window where it functions — which is what makes everything else in CI/CD spatial quality gates worth having.