Spatial Test Observability and Metrics

15 min read

Spatial test observability is the practice of treating a validation suite as a source of measurements rather than only of verdicts. It sits beneath CI/CD spatial quality gates and answers a question a pass or fail cannot: is this getting worse? A gate that reports green tells you the data satisfied a threshold today. A metric series tells you the measured drift has tripled over six weeks and will breach the threshold next month — which is information you can act on before anything is red.

The distinction matters more in spatial work than in most domains, because the interesting quantities are continuous. Geometry drift, coordinate residual, invalid-geometry rate, topology violation count: every one of these is a number that a gate collapses to a boolean, and the number is where the early warning lives.

The Metric Set Worth Exporting

A suite can emit hundreds of numbers, and most of them are noise. Five families carry nearly all the signal, and each answers a different operational question.

Rule outcomes, as counters by rule name and severity. Not “the suite passed” but “rule topology.no_overlap evaluated 41,203 features and found 12 violations”. The count is the metric; the pass or fail is a threshold applied to it.

Measured drift, as a distribution rather than a scalar. Coordinate residual, area delta, Hausdorff distance — recorded as mean, 95th percentile and maximum, because the shape of the distribution distinguishes a systematic offset from magnitude-scaled noise.

Population sizes, as counters. Features evaluated, cells valid, rows loaded. A rule whose population changed is a rule whose result is not comparable, and this is the metric that makes that visible.

Runtime, per stage. Not for performance work but for early warning: a validation stage whose duration has doubled is usually processing more data than anyone intended.

Environment fingerprints, as labels rather than as values. Engine versions, image digest, grid package. These are what make every other metric attributable when it moves.

Five metric families and what each answers Five rows pairing a metric family with the operational question it answers. Rule outcome counters, labelled by rule name and severity, answer how many violations exist rather than merely whether any do. Drift distributions, carrying mean, ninety-fifth percentile and maximum, answer whether a deviation is systematic across all features or scales with coordinate magnitude. Population counters, recording features evaluated and cells valid, answer whether two results are comparable at all. Per-stage runtimes answer whether the workload has grown beyond what anyone intended. Environment fingerprints, carried as labels rather than values, answer whether an observed change is attributable to the runtime rather than to the data. METRIC FAMILY SHAPE ANSWERS Rule outcomes by rule name and severity counter how many, not merely whether any Measured drift residual, area delta, Hausdorff mean · p95 · max systematic, or magnitude-scaled? Population sizes features, cells, rows counter are two results even comparable? Stage runtime per stage, not per test histogram has the workload grown? Environment fingerprints labels, not values is the change attributable to the runtime?

The last row is a design decision worth stating explicitly. Engine versions belong as labels on the other metrics rather than as metrics of their own, because their purpose is to slice: “drift by PROJ version” answers a question that “PROJ version over time” cannot.

Drift as a Service-Level Objective

The most useful thing this layer produces is a framing rather than a number. Once drift is a measured series with a budget, spatial accuracy becomes an objective with an error budget in exactly the sense reliability engineering uses — and that changes the conversation from “is the data good” to “how much of our accuracy budget have we spent this quarter”.

The construction is straightforward. Choose the quantity that matters to consumers — usually round-trip residual or area delta — and state an objective: the 95th percentile of coordinate drift stays below one centimetre over a rolling 30 days. The error budget is the fraction of that window during which the objective may be violated. Every run contributes a measurement; the budget is consumed when measurements exceed the target.

budget consumed=∣{r:p95(r)>τ}∣∣{r}∣over the window\text{budget consumed} = \frac{\lvert \{ r : p_{95}(r) > \tau \} \rvert}{\lvert \{ r \} \rvert} \quad \text{over the window}

Two consequences follow that a pass-fail gate cannot produce. A partially-spent budget is a warning with time to act — it says the objective is at risk before anything is red. And an exhausted budget is a decision point, not merely a failure: either the pipeline is fixed or the objective was wrong, and both are legitimate outcomes that a boolean gate never surfaces.

An error budget consumed before any gate turns red A time series of ninety-fifth percentile coordinate drift, one point per run, plotted across a rolling thirty-day window. A horizontal line marks the objective. Early runs sit comfortably below it and consume none of the error budget. Over the window the series trends upward; several later runs cross the objective and each consumes part of the budget, shown as a vertical bar at the right filling progressively. By the end of the window the budget bar is close to full even though no single run has breached the harder threshold that would fail a gate outright. The diagram's point is that the budget produces a warning with time to act, whereas a pass-fail gate produces nothing until the failure. p95 drift rolling 30 days objective — p95 ≤ 1 cm these runs consume budget budget spent error budget ~50% used No single run has failed a gate. The budget has spent half of itself, the trend is monotonic, and there is a month of warning. A pass-fail gate produces nothing at all until the last point on this chart.

Alert on Change, Not Only on Threshold

A threshold alert fires when a value crosses a line, which for a slowly degrading quantity means it fires long after the degradation began. A change alert fires when the value moves relative to its own recent history, which is usually days or weeks earlier.

Both are needed and they answer different questions. The threshold says this is now unacceptable; the change detector says something happened. For spatial metrics the second is generally the more valuable, because most of the interesting events — an upstream producer altering a simplification tolerance, a grid package changing, a new data source joining a feed — produce a step change well inside the acceptable range.

Signal Fires when Catches Misses
Threshold on the value The measurement crosses the budget Anything unacceptable, eventually Every degradation until it is too late
Step change vs baseline The value jumps relative to recent runs Upstream changes, engine changes Slow monotonic drift
Trend over a window The slope is consistently positive Slow monotonic drift Sudden one-off events
Population change The evaluated count moves Truncated loads, filter changes Anything not size-related
Absence of a metric A rule stopped reporting at all Silently disabled checks Nothing — this is the cheapest alert there is

The last row is the one nobody configures and the one that catches the worst failure. A rule that stops emitting because it was skipped, renamed, or collected zero tests looks identical to a rule that passed. Alerting on the absence of an expected metric is a few lines and it is the only defence against a check that quietly stopped running.

Structured Logs That Survive the Runner

Metrics answer “how much”; logs answer “which one”. Both are needed and the log schema is where most of the practical value sits, because a failure that can be reproduced from a log line does not need the runner to still exist.

A useful spatial log line carries the rule, the feature identifier, the measured value, the threshold, and the environment fingerprint. It carries them as fields rather than as prose, so a query can aggregate them. And it carries no coordinates, because a coordinate in a log is location data with the retention properties of a log — which is exactly the problem discussed under security boundaries in spatial QA.

The relationship between the three artefacts is worth stating plainly. The metric is the aggregate a dashboard watches. The log line is the per-event record a query filters. The quarantine artefact is the geometry an engineer opens. Each is unusable in the other’s role: a dashboard cannot show a polygon, and a GeoPackage cannot show a trend.

Where the Numbers Come From

Instrumentation is easier than it looks because the suite already computes everything. Every assertion has a measured value and a threshold; the only change is to record the measured value rather than discarding it once the comparison is made.

The pattern is a small reporter that assertions call instead of comparing inline. It takes the rule name, the measured value, the threshold and the population, records them, and then performs the comparison. The assertion still fails exactly as before; what changes is that a passing assertion now also leaves a number behind — and passing measurements are the baseline that makes failing ones interpretable.

Three properties make the reporter worth building rather than sprinkling metric calls through the suite. It gives one place where the metric naming convention lives, so names stay consistent as the suite grows. It makes the population mandatory, because the function signature requires it, which is the field teams otherwise forget. And it allows the emission target to change — a JSON artefact today, a metrics system next quarter — without touching a single assertion.

One reporter turns every assertion into a measurement Two arrangements compared. Without a reporter, each assertion compares its measured value against a threshold inline and then discards the value, so only the pass or fail verdict survives and passing runs leave no trace at all. With a reporter, each assertion instead calls a shared function that records the rule name, the measured value, the threshold and the population, and then performs the identical comparison; the verdict behaviour is unchanged while every run, including every passing one, contributes a measurement. The reporter is additionally the single place where the metric naming convention lives and the single place where the emission target can be changed from a JSON artefact to a metrics system without editing any assertion. Inline comparison assertion measures, compares value discarded only the verdict survives passing runs leave nothing behind so a failing run has no baseline to be read against Through a reporter assertion calls the reporter reporter records rule, value, threshold, population same verdict — plus a measurement on every run one naming convention; the emission target can change freely The suite already computes every number this layer needs. The only change is to stop throwing them away at the moment of comparison. Nothing about the assertions’ behaviour changes — which is what makes the migration safe to do incrementally.

Cardinality Is the Constraint

The one way this layer goes wrong at scale is cardinality: too many distinct label combinations, which makes a metrics system expensive and a dashboard unreadable. Spatial data invites the mistake, because the obvious labels are exactly the high-cardinality ones.

Never label by feature identifier. A metric labelled with a parcel id produces one series per feature, which for a national dataset is millions. The feature belongs in the log line, where filtering is cheap and retention is bounded; the metric carries only the count.

Be careful with tile or partition labels. A metric per tile is attractive for locating a problem and quickly becomes tens of thousands of series. Aggregate to a coarser level — an administrative area, a zoom level — and let the log line carry the specific tile.

Rule name and severity are safe. Both are small, bounded sets that change only when someone edits the suite, which is exactly the property a label needs.

The general rule: a label’s cardinality should be bounded by how many things a human would want to compare, not by how many things exist. Nobody compares a million parcels on a dashboard; everybody compares a dozen rules.

Frequently Asked Questions

How much of this is worth doing for a small pipeline?

The counters and the absence alert, on day one; they cost almost nothing and the absence alert alone justifies the exercise. Drift distributions become worthwhile once there is a tolerance anybody argues about. The full error-budget framing pays off when there is a consumer to have the conversation with, and not before — it is a coordination tool as much as a technical one.

Where should the metrics go?

Wherever the team already looks. A spatial metric in a dashboard nobody opens is not observability. If the organisation runs a metrics system, emit there; if it does not, a JSON artefact per run plus a small script that plots the last thirty is genuinely sufficient and far better than nothing.

Should test runtime be a gate or a metric?

A metric, with an alert on the trend. Timing assertions inside functional tests make them flaky on shared runners, because machine-to-machine variance of thirty per cent is normal. Tracking the trend catches the same regressions without the noise, and it catches them earlier because a doubling is visible long before any absolute limit.

How do you avoid a dashboard nobody reads?

Keep it to the five families and put the absence alert first. A page with two hundred spatial metrics is a page nobody can interpret; one with a violation count, a drift percentile, a population size, a stage runtime and a list of rules that did not report is readable in ten seconds. Add the sixth metric only when somebody asks a question the first five cannot answer.

Does this replace the gate?

No — it complements it, and the division is clean. The gate decides whether this change may merge, and it must be fast and binary. Observability decides whether the system is getting better or worse, and it is slow and continuous. Teams that try to make the gate do both end up with a slow gate and a poor signal.

Making the Signal Reach a Person

A metric that fires into a channel nobody reads has the same operational value as no metric at all, and spatial quality signals are unusually prone to this because they rarely map onto an existing on-call rotation. Nobody is paged for a coordinate residual, and nobody should be — but somebody has to look.

Three routing decisions make the difference between a signal that gets acted on and one that accumulates.

Route by who can act, not by severity. A drift alert caused by an upstream producer belongs in that team’s channel, with enough context to be actionable — the rule, the measurement, the trend, the sample. Routing it to the platform team because the platform team owns the dashboard guarantees a forwarding step that eventually stops happening.

Give each alert a documented response. An alert whose runbook is “investigate” will be dismissed, because dismissing it is cheaper than deciding what to do. An alert whose runbook says “compare the config hash against the last passing run; if it moved, the change was intentional” is actionable in a minute and gets handled.

Review the ones that fired and were dismissed. A monthly look at dismissed alerts is the cheapest tuning mechanism available. Every dismissal is either a threshold that is too tight, a signal routed to the wrong person, or a genuine finding somebody let go — and all three are worth knowing.

There is a related organisational point about the error budget framing. Its value is that it makes a conversation possible: a producer and a consumer can look at the same series and agree what the objective should be, rather than arguing about whether the data is “good”. That conversation is the actual product of this layer, and it only happens if the series is visible to both parties rather than living inside one team’s CI.

Where no such conversation is possible — a supplier who will not engage, a dataset delivered without negotiation — the budget still helps, but its purpose changes. It becomes a record of what the pipeline is absorbing, which is what turns “the data is sometimes bad” into a quantity that can be put in front of somebody with the authority to change the arrangement.

Common Failure Modes and Gotchas

  1. Metrics collected but never watched. A series nobody looks at is storage cost with no return. Put the five families on one page somebody opens weekly, and delete anything that has not informed a decision in six months.
  2. A gate built on a metric. Failing a merge because a drift metric moved makes the gate depend on a noisy continuous quantity and reintroduces every problem the threshold framing solved. Gates use thresholds; observability uses trends, and mixing them produces a slow, flaky gate.
  3. Labels with unbounded cardinality. A metric per feature or per tile grows without limit and eventually costs more than the pipeline it watches. Aggregate for the metric, and keep the specific identifier in the log.
  4. Coordinates in the log schema. A structured log with a geometry field is location data with a log’s retention and access properties. Log identifiers and measured distances, never positions.
  5. No absence alert. A rule that stops reporting looks exactly like a rule that passes. This is the single cheapest alert available and the one most often missing.
  6. Comparing across a population change. A drift percentile computed over 40,000 features and one computed over 4,000 are not comparable, and a dashboard will happily plot them adjacent. Carry the population as a field and refuse to compare when it moved materially.
  7. Retention shorter than the investigation. A series that keeps thirty days cannot answer a question about a quarterly trend, and a run artefact that expires in a day cannot be compared against a failure discovered the following week. Set both from how long questions actually take to arrive.

Conclusion

A spatial suite is already computing the numbers that would tell you where the pipeline is heading; most suites simply discard them at the moment they are collapsed into a verdict. Exporting rule counts, drift distributions, populations, runtimes and environment labels — then alerting on change and on absence rather than only on thresholds — converts a gate that reports today’s answer into a system that reports the trend, which is the contribution this layer makes to CI/CD spatial quality gates.