Emitting OpenTelemetry Spans from Spatial Tests

9 min read

A span is a timed, attributed record of one unit of work, and a spatial suite is full of units of work worth recording: a fixture build, a reprojection, a topology pass over a coverage, a database round trip. This guide sits beneath spatial test observability and metrics and covers instrumenting a pytest suite with OpenTelemetry: which spans are worth creating, what attributes to attach without exploding cardinality, and how to keep the instrumentation from becoming the slowest thing in a fast gate.

The value is specific. Metrics tell you a stage got slower; a trace tells you which part of it did, and for spatial work the answer is frequently surprising — a suite that appears to be spending its time on geometry is often spending it on fixture I/O or on a database connection that is re-established per test.

Root cause: a suite’s cost is not where people assume

Spatial validation feels compute-bound, so the instinct when a suite slows is to look at the predicates. In practice the time is usually somewhere else: reading a fixture from disk, building a spatial index that could have been session-scoped, establishing a connection, or serialising a frame to hand to another process.

Those costs are invisible in a test-level timing report, which attributes everything to whichever test happened to trigger the work. A span tree separates them, because the fixture build is its own span with its own duration regardless of which test caused it to run.

One four-second test, decomposed by span A single test whose reported duration is four seconds is broken into its constituent spans. The fixture read occupies a large share, the spatial index build occupies a larger one, the geometry predicate under test occupies only a small slice, and a database round trip accounts for the remainder. Above the decomposition, a test-level report is shown attributing the entire four seconds to the test name alone. The annotation records that an engineer working from the test-level report begins by optimising the predicate, which is the smallest of the four components, while the index build that dominates is invisible because it is a fixture rather than a test. What a test-level report shows test_no_overlaps — 4.0 s one number What the spans show fixture.read — 1.3 s index.build — 1.6 s predicate db.query 0.4 s 0.7 s Working from the top row, an engineer optimises the predicate — the smallest of the four components. The index build that dominates is invisible, because it is a fixture rather than a test. Session-scoping that one fixture removes more time than any change to the assertion could.

Span reference

Span Parent Attributes worth attaching
suite.session root engine versions, image digest, code revision
fixture.build session fixture name, generator, seed, feature count
fixture.read session or test format, feature count, byte size
index.build test or fixture index type, entry count
rule.evaluate test rule name, population, violations found
crs.transform rule source and target authority codes
db.query rule operation name, rows returned

Every attribute in that column is deliberately low-cardinality. A feature identifier or a geometry never appears, for the same reason it never appears in a metric label — traces are stored, queried and retained, and an identifier turns a bounded dataset into an unbounded one.

Step-by-step implementation

The instrumentation targets the OpenTelemetry Python SDK and pytest 7+.

Step 1 — Start a session span and put the environment on it

The root span carries what every child inherits, which is how a trace becomes attributable without repeating the fingerprint on every span.

# conftest.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
import pytest, os, shapely, pyproj

def pytest_configure(config):
    resource = Resource.create({
        "service.name": "spatial-suite",
        "geos.version": shapely.geos_version_string,
        "proj.version": pyproj.proj_version_str,
        "vcs.revision": os.environ.get("GIT_SHA", "unknown"),
    })
    trace.set_tracer_provider(TracerProvider(resource=resource))

Step 2 — Wrap the expensive fixtures, not the tests

Spans on tests duplicate what pytest already reports. Spans on the work are what add information.

tracer = trace.get_tracer("spatial-suite")

@pytest.fixture(scope="session")
def parcels():
    with tracer.start_as_current_span("fixture.build") as span:
        gdf = build_parcels(seed=20260811)
        span.set_attribute("fixture.name", "parcels")
        span.set_attribute("fixture.features", len(gdf))
        span.set_attribute("fixture.seed", 20260811)
        return gdf

Step 3 — Make each rule its own span

A rule span with the population and violation count on it is the single most useful record this instrumentation produces, because it is simultaneously a timing and a result.

from contextlib import contextmanager

@contextmanager
def rule_span(name: str, population: int):
    with tracer.start_as_current_span("rule.evaluate") as span:
        span.set_attribute("rule.name", name)
        span.set_attribute("rule.population", population)
        yield span

def test_no_overlaps(parcels):
    with rule_span("topology.no_overlap", len(parcels)) as span:
        violations = find_overlaps(parcels)
        span.set_attribute("rule.violations", len(violations))
        assert not violations, f"{len(violations)} overlapping parcels"

Setting the violation count before the assertion matters: an assertion that raises still leaves the span with its attributes, so a failing run is as informative as a passing one.

Step 4 — Decide sampling before it costs anything

Full tracing on every run of a fast gate is usually wasted — nobody looks at the trace of a green run. Sample so that the cost lands where the value is.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Trace every scheduled run; sample pre-merge runs lightly.
ratio = 1.0 if os.environ.get("CI_SCHEDULE") else 0.05
trace.set_tracer_provider(
    TracerProvider(resource=resource, sampler=ParentBased(TraceIdRatioBased(ratio)))
)
Sampling by what the trace will be used for Three run categories with their appropriate sampling ratios. Pre-merge runs are frequent and their traces are rarely inspected, so a low sampling ratio around five per cent keeps the gate fast while still producing occasional traces for reference. Scheduled runs are infrequent and are the ones examined when someone investigates a trend, so they are traced in full. Failed runs of any category are traced in full regardless of the configured ratio, because a failure is precisely the moment a trace is wanted. A closing note frames sampling as a cost decision and stresses that failing runs must be exempted from it. RUN CATEGORY SAMPLE BECAUSE Pre-merge many per day, rarely inspected ~5% the gate must stay fast; a sample is enough for reference Scheduled few per week, always inspected 100% this is the run somebody opens when investigating a trend Any failing run exempt from the ratio 100% a failure is exactly when the trace is wanted Sampling is a cost decision, and the one category that must never be sampled away is the one people actually look at.

Verify the fix

Run with a console exporter and confirm the span tree has the shape you expect:

OTEL_TRACES_EXPORTER=console pytest -q tests/ -k topology

The output should show fixture.build as a sibling of the tests rather than a child of one, and each rule.evaluate should carry a population and a violation count. A fixture span nested inside the first test that used it means the fixture is function-scoped when it should not be — which is a finding in itself.

Reading a trace to find the actual cost

The reason to instrument at all is the question a trace answers and a metric cannot: within one slow stage, which part is slow. Three patterns recur in spatial suites and each is recognisable from the span tree alone.

A fixture span repeated per test means a session-scoped resource is being rebuilt. This is the most common finding and usually the largest win, because the fix is a scope annotation rather than an algorithmic change.

A db.query span whose duration exceeds its row count by orders of magnitude means the query is not using an index — the planner could not reduce the predicate to a bounding-box test, which is the failure described in detail under topology rule enforcement.

A crs.transform span appearing inside a per-feature loop means the transformer is being constructed repeatedly rather than once. Construction dominates transformation for small inputs, so this shows up as a large number of short spans whose total exceeds everything else.

None of the three is visible in a test-level timing report, and all three are obvious in a span tree after ten seconds of looking. That asymmetry is the entire argument for the instrumentation.

Three recognisable span-tree shapes Three diagnostic patterns drawn as span arrangements. The first shows an identical fixture span repeated once beneath each test, identifying a resource that should be session-scoped being rebuilt for every test; the remedy is a scope annotation. The second shows a single database query span whose length is grossly disproportionate to the small number of rows it returned, identifying a query whose predicate the planner could not reduce to an index lookup. The third shows a long run of very short transform spans nested inside a loop, identifying a transformer being constructed per feature instead of once. A closing note records that all three are obvious in a trace within seconds and none is visible in a test-level timing report. Fixture repeated per test the same build, four times → scope it to the session Query long, rows few db.query — 2.9 s rows returned: 14 duration disproportionate to result → the index was not used Many short transforms crs.transform × 4 200 construction dominates transformation → build the transformer once All three are obvious in a trace within seconds. None of them appears in a test-level timing report, which attributes every cost to whichever test triggered it. That asymmetry is the whole argument for instrumenting the work rather than the tests.

Connecting the suite’s trace to the pipeline’s

A spatial test suite usually validates a pipeline that is itself instrumented, and the two traces are far more useful joined than separate. When the suite’s spans share a trace context with the pipeline run they validate, a single view shows the transformation and the assertion that checked it — which is what makes it possible to say that a rule failed because a specific upstream stage produced something unusual, rather than merely that it failed.

The mechanics are ordinary context propagation: the pipeline emits a trace identifier, the CI job passes it into the suite’s environment, and the session span is created as a child of it rather than as a root. Nothing spatial about it, and it changes what questions the trace can answer.

Where the pipeline is not instrumented, a weaker link is still worth having: put the pipeline’s run identifier on the session span as an attribute. It does not join the traces, but it makes them correlatable by query, which covers most of the value for a fraction of the work.

One caution about propagating context into a parallel suite. Under xdist each worker is a separate process, and a naively-shared context makes every worker’s spans children of the same parent, producing a trace that is technically correct and visually unreadable. Give each worker its own child span of the session and let its work nest beneath that — the worker identifier is a low-cardinality attribute and exactly the right thing to slice by when one worker behaves differently from the others.

Failure modes and edge cases

  1. Spans on every test. Duplicates what the runner already reports and multiplies the trace volume for no new information. Instrument the work, not the test function.
  2. High-cardinality attributes. A feature identifier or a WKT string on a span makes the trace store expensive and the data unqueryable. Attach counts and names; keep identifiers in logs.
  3. Instrumentation that dominates a fast gate. Exporting synchronously on every span turns a two-minute lane into a four-minute one. Batch the export, and sample pre-merge runs.
  4. Losing the trace when the process dies. An out-of-memory kill leaves spans unexported. A batching exporter with a short flush interval loses less, and the session span’s absence is itself a signal worth alerting on.
  5. Sampling away failures. A ratio sampler applied uniformly discards the traces that matter. Exempt failing runs explicitly.
  6. Attributes set after the assertion. An assertion that raises skips everything after it, so a failing span carries no result attributes. Set them before the comparison.

Conclusion

Instrumenting the work rather than the tests turns a suite from something that reports durations into something that explains them. A session span carrying the environment, fixture and rule spans carrying counts and populations, low-cardinality attributes throughout, and a sampling policy that spends the budget on scheduled and failing runs gives the diagnostic layer that spatial test observability and metrics needs beneath its aggregate view.