Tracking Geometry Drift as a Service-Level Objective

10 min read

Spatial accuracy is a continuous quantity that most pipelines report as a boolean. This guide sits beneath spatial test observability and metrics and shows how to express it instead as a service-level objective with an error budget: choosing an indicator that consumers actually feel, setting a target from what they need rather than from what the pipeline currently achieves, computing the burn, and using the result to have a conversation that a pass-fail gate makes impossible.

The framing is borrowed from reliability engineering and it transfers cleanly, because the underlying situation is the same. Perfect is unattainable, some degradation is acceptable, and the useful question is how much has been spent rather than whether anything has broken.

Root cause: a threshold discards the information you need

A gate compares a measurement against a limit and reports a boolean. That is correct behaviour for a gate — a merge decision has to be binary — and it destroys the two things needed to manage accuracy over time: the magnitude of the measurement and its history.

Consider a drift measurement that has moved from 2 mm to 8 mm over a quarter against a 10 mm limit. Every run in that quarter was green. Nothing in the gate’s output distinguishes the first week from the last, and the first indication anybody receives is a failing build in the following month, at which point the cause is a quarter old and the investigation starts cold.

The same quarter, seen two ways Two views of one quarter of runs. The upper strip shows the gate's output: a long unbroken run of green results followed by a single red one at the end, with no distinction between the first week and the last. The lower chart shows the underlying measurement, which rises steadily from about two millimetres to about eight against a limit of ten, crossing it only at the very end. A caption records that the gate produces its first signal a full quarter after the change began, by which time the cause is long cold, whereas the measurement series showed the rise from the first week. What the gate reported nine identical green results, then one red — no gradient anywhere in the output What the measurement did limit 10 mm 2 mm 8 mm The gate’s first signal arrives a quarter after the change began. The series showed it in week one.

Choosing the indicator

A service-level indicator is the measurement a consumer would recognise as their experience of the system, and picking one that nobody feels is the most common way this framing fails. Three candidates recur in spatial pipelines and they suit different consumers.

Round-trip coordinate residual suits any consumer that overlays your data against another source. It is measurable without any consumer involvement, which makes it the easiest to start with, and it is a good proxy for “does this line up”.

Area or length delta after processing suits consumers who aggregate — a reporting pipeline, a billing calculation, anything where a total must reconcile. It is the indicator that catches a simplification tolerance change, which residual will not.

Feature-level agreement against a reference suits consumers doing joins or lookups: the fraction of features whose relationship to a reference layer is unchanged. It is the most expensive to compute and the closest to what a downstream system actually depends on.

Indicator Consumer who feels it Catches Costs
Round-trip residual Anyone overlaying data Transform and datum problems One transform per fixture point
Area or length delta Anyone aggregating Simplification, precision changes One pass over the geometry
Feature agreement Anyone joining Relationship changes, boundary shifts A spatial join per run
Invalid-geometry rate Anyone rendering or tiling Structural degradation upstream Already computed by the suite

Setting the target from need, not from current performance

The instinct when setting an objective is to measure what the pipeline currently does and set the target just above it. That produces a number that is always met and therefore never informative — the objective becomes a description rather than a commitment.

The target should come from the consumer’s requirement: the map scale the data is rendered at, the survey standard it must satisfy, the tolerance the downstream join uses. That number is usually harder to obtain and always more useful, because it makes the objective capable of being missed — which is the entire point of having one.

Where the requirement and the current performance are far apart, the honest response is to state both. An objective of 1 cm against a current 8 cm is a commitment with a plan attached, and it is far more useful than an objective of 9 cm that nothing will ever violate.

SLO: p95(drift)≤τ over a rolling W days,budget=α⋅∣RW∣\text{SLO: } p_{95}\big(\text{drift}\big) \le \tau \text{ over a rolling } W \text{ days}, \quad \text{budget} = \alpha \cdot |R_W|

where τ\tau comes from the consumer, WW is the window, and α\alpha is the fraction of runs in that window permitted to exceed the target.

A target from performance, and a target from need Two charts of the same measurement series. In the first, the objective line is drawn just above the highest point the pipeline currently reaches, so every run sits below it and the objective is satisfied by construction; the annotation records that such an objective describes the status quo rather than committing to anything. In the second, the objective line is drawn where the consumer's requirement lies, well below the current series, so most runs exceed it and the gap between current performance and requirement is visible as a measurable quantity; the annotation records that this version can be missed, which is what makes it worth having. Target from current performance objective satisfied by construction — no run can violate it a description, not a commitment Target from consumer requirement objective — what the consumer needs most runs exceed it; the gap is a measurable quantity a commitment with a plan attached An objective that cannot be missed provides no information. Stating a target the pipeline does not yet meet is more useful than one it always will. The gap is not an embarrassment — it is the quantity the improvement work is measured against.

Computing and reading the burn

The burn is the fraction of the error budget consumed, and its rate of change is more informative than its level. Three readings recur and each implies a different action.

Flat burn near zero means the objective is comfortably met and possibly too loose. If it has been flat for two quarters, the target is worth revisiting — either tightening it towards what consumers actually need, or acknowledging that this indicator is not the one to watch.

Steady burn tracking the window means the pipeline is operating at its objective, spending budget at the rate the objective anticipated. This is the healthy state and it needs no action, which is worth saying explicitly because it looks like a problem to anyone reading the number without the rate.

Accelerating burn is the actionable signal. The budget will exhaust before the window rolls, and the time between noticing and exhausting is the window in which a fix is cheap. This is the entire operational value of the construction.

The natural cadence for reading it is weekly, alongside whatever review already happens. Alerting on the burn rate rather than the level gives the earliest useful warning: a rate that would exhaust the budget before the window ends is worth a message regardless of how much budget currently remains.

Three burn patterns, three different actions Three small charts of cumulative error budget consumed against time within a rolling window. The first is nearly flat and close to zero, indicating an objective that is comfortably met and possibly too loose, with the suggested action being to revisit the target. The second rises in a straight line that reaches the budget limit exactly as the window closes, indicating a pipeline operating at its objective; this is the healthy state and requires no action, which is worth stating because the rising line looks alarming to anyone reading the level without the rate. The third curves upward and crosses the budget limit well before the window closes, indicating an accelerating burn; the interval between the point where the acceleration becomes visible and the point where the budget exhausts is annotated as the window in which a fix is still cheap. Flat — target may be too loose two quarters near zero → revisit the target Steady — healthy reaches the limit as the window closes → no action — this is the objective working Accelerating — act now exhausts early the gap to the window edge is the cheap-fix interval → alert on the rate, not the level The middle chart is the one that gets misread. A rising line looks like a problem; a rising line that arrives exactly on time is the objective doing its job. Alerting on the level produces noise from the middle case; alerting on the rate produces a signal only from the third.

What the Framing Makes Possible

The operational value is real but secondary. The primary value is that a measured series with an agreed target enables a conversation that a pass-fail gate makes impossible.

With a gate, a producer and a consumer can only discuss whether the data is acceptable, and the discussion has no shared quantity. With an objective, both parties look at the same series, and the questions become tractable: is the target right, is the current performance improving, how much of the budget does a proposed change consume. Disagreements become about numbers rather than about adjectives.

That is also why the objective should be agreed rather than imposed. A target set unilaterally by the consuming team is a demand; the same number agreed after both parties have looked at the series is a commitment, and the difference shows up in whether anybody acts when the budget starts burning.

Where no negotiation is possible — a supplier who will not engage, a dataset that arrives without discussion — the objective still earns its place, but its purpose changes. It becomes a record of what the pipeline is absorbing, which is what turns “the data is sometimes bad” into a quantity that can be shown to somebody with the authority to change the arrangement.

Failure modes and edge cases

  1. An indicator nobody feels. A drift metric on a quantity no consumer depends on produces an objective that is technically met and operationally irrelevant. Choose the indicator by asking who would notice if it degraded.
  2. A target set from current performance. It cannot be missed, so it reports nothing. Set it from need and treat the gap as the improvement backlog.
  3. A window shorter than the change rate. A seven-day window on a quantity that drifts over months resets before any trend accumulates. Match the window to the timescale of the changes you want to catch.
  4. Comparing across a population change. A percentile computed over a different number of features is not comparable to the previous one, and a chart will happily place them adjacent. Carry the population and break the series when it moves materially.
  5. Burning budget on a known cause. If a batch is known to be bad, excluding it from the calculation is legitimate and must be recorded, or the series quietly stops meaning what it claims.
  6. Treating budget exhaustion as a failure. It is a decision point: fix the pipeline, or change the objective. Both are valid, and pretending only the first is turns the framing back into a gate.

Conclusion

An error budget converts spatial accuracy from a property that is either acceptable or not into a quantity with a rate of change. Choosing an indicator a consumer feels, setting the target from their requirement rather than from current performance, computing the burn and alerting on its rate gives a signal that arrives with time to act — and, more valuably, gives two teams a shared number to negotiate over, which is the contribution spatial test observability and metrics makes beyond the gate.