Testing Datum Shifts with pyproj Transformer

9 min read

A datum shift is the part of a coordinate transformation that changes the reference frame rather than the map projection, and it is where the largest silent errors in a spatial pipeline come from. This guide sits beneath coordinate reference system testing and shows how to construct a pyproj.Transformer whose behaviour is pinned rather than inferred, how to assert that PROJ selected the operation you intended, and how to detect the fallback that occurs when a datum grid file is unavailable. The specific defect this prevents is the one that produces no error at all: a transformation that succeeds, returns plausible coordinates, and places every feature one to two metres from where it belongs.

Why the default transformer is not deterministic

Transformer.from_crs(src, dst) asks PROJ for a path between two coordinate reference systems, and between many datum pairs there are several. A high-accuracy path uses a datum grid — a file of interpolated shifts derived from survey observations — and typically reports sub-decimetre accuracy. A low-accuracy path uses a seven-parameter Helmert transformation or, worse, no shift at all, and can be metres out at the edges of the area.

PROJ ranks the candidates and returns the best one it can actually use, which is the key qualification. If the grid file is absent from the environment, the high-accuracy operation is not available, the ranking silently promotes the next candidate, and the transformation proceeds. No exception is raised, because from PROJ’s point of view nothing went wrong: it was asked for a transformation and it supplied one.

Candidate operations and the silent promotion A request for a transformation between two datums yields a ranked list of three candidate operations. The first uses a datum grid file and reports accuracy better than a decimetre. The second uses a Helmert parameter set and reports roughly one metre. The third performs no shift at all and reports several metres. PROJ returns the highest-ranked candidate whose requirements the environment satisfies, so when the grid file is absent the second candidate is used instead of the first, with no error raised. A closing note records that nothing in the returned coordinates reveals which candidate was applied. from_crs(src, dst) no operation named 1 · grid-file operation accuracy < 0.1 m · needs the grid present 2 · Helmert parameters accuracy ≈ 1 m · always available 3 · null transformation accuracy ≈ several m · last resort grid absent → you get 2 no exception, no warning Nothing in the returned coordinates says which candidate ran. The difference between rows one and two is roughly a metre, everywhere, in the same direction.

The consequence is an environment-dependent pipeline: correct on a machine with the full PROJ data package, quietly one metre out on one without it. Because the offset is constant, no per-feature check notices, and the data looks entirely reasonable until someone overlays it on an authoritative reference.

Parameter reference

Construct Purpose Failure it prevents
Transformer.from_crs(a, b, always_xy=True) Fixes coordinate order to easting-northing Latitude/longitude inversion
TransformerGroup(a, b) Lists every candidate operation Blind reliance on PROJ’s ranking
.transformers[i].description Names the operation actually selected Silent substitution after an upgrade
.accuracy Reported accuracy in metres A coarse fallback passing as precise
.unavailable_operations Candidates PROJ could not use A missing grid file, named explicitly
Transformer.from_pipeline(...) Pins an exact operation string Any re-ranking at all
errcheck=True Raises rather than returning infinity Out-of-domain input passing silently

Step-by-step implementation

The pattern below targets pyproj 3.6+ against PROJ 9.x, and produces a transformer whose behaviour is asserted rather than assumed.

Step 1 — Enumerate the candidates before choosing

TransformerGroup exposes what from_crs hides. Run it once, in development, to see what your environment actually offers.

from pyproj.transformer import TransformerGroup

group = TransformerGroup("EPSG:4326", "EPSG:27700")
for t in group.transformers:
    print(f"{t.accuracy:>6} m  {t.description}")
for missing in group.unavailable_operations:
    print("UNAVAILABLE:", missing.name)     # names the grid file you are missing

The unavailable_operations list is the important half. An entry there means PROJ knows of a better operation and cannot use it, which is exactly the condition that produces a silent metre of error.

Step 2 — Construct the transformer explicitly

Once the intended operation is known, stop asking PROJ to choose. Always set always_xy so coordinate order is fixed regardless of the authority definition.

from pyproj import Transformer

# always_xy=True → (easting, northing) / (lon, lat) in both directions
TF = Transformer.from_crs("EPSG:4326", "EPSG:27700", always_xy=True, errcheck=True)

errcheck=True matters more than it looks: without it, a coordinate outside the transformation’s domain returns infinity rather than raising, and the infinity propagates as NaN through everything downstream.

Step 3 — Assert the operation and its accuracy

This is the assertion that turns a working transformer into a pinned one. It fails when the environment changes underneath the pipeline, which is precisely when you want to know.

import pytest
from pyproj.transformer import TransformerGroup

EXPECTED_ACCURACY_M = 0.1

def test_datum_operation_is_the_intended_one():
    group = TransformerGroup("EPSG:4326", "EPSG:27700")
    assert not group.unavailable_operations, (
        f"PROJ knows a better operation it cannot use: "
        f"{[o.name for o in group.unavailable_operations]}"
    )
    best = group.transformers[0]
    assert best.accuracy <= EXPECTED_ACCURACY_M, (
        f"selected operation reports {best.accuracy} m, expected <= {EXPECTED_ACCURACY_M}; "
        f"description: {best.description}"
    )

Step 4 — Verify against an externally-known coordinate

A round trip cannot detect an error its inverse shares, so at least one point must have a value known from outside the transformation. A published survey monument with coordinates in both systems is the canonical source.

# A monument with published values in both frames.
MONUMENT_WGS84 = (-1.542324, 53.797416)
MONUMENT_BNG   = (429157.19, 434005.51)   # published easting, northing
TOLERANCE_M    = 0.05

def test_monument_lands_where_it_is_published():
    east, north = TF.transform(*MONUMENT_WGS84)
    dx = east - MONUMENT_BNG[0]
    dy = north - MONUMENT_BNG[1]
    drift = (dx ** 2 + dy ** 2) ** 0.5
    assert drift <= TOLERANCE_M, f"monument is {drift:.3f} m from its published position"
Four assertions over one transformer Four stacked assertions applied to a single transformer. The availability assertion confirms that PROJ has no higher-ranked operation it was unable to use, which catches an absent datum grid file. The accuracy assertion confirms that the operation actually selected reports an accuracy within the configured budget, which catches a coarse fallback presenting as precise. The monument assertion compares a transformed coordinate against a value published independently, which is the only one able to catch errors shared by the forward and inverse transformations. The domain assertion confirms that a coordinate outside the valid area raises an exception rather than returning infinity. Each assertion catches a class that the ones above it cannot see. 1 · availability — no unavailable_operations catches: a datum grid file missing from this environment costs: one lookup 2 · accuracy — selected operation within budget catches: a coarse fallback presenting as precise costs: one lookup 3 · monument — matches a published coordinate catches: errors shared by forward and inverse — nothing else can costs: one transform 4 · domain — out-of-area input raises catches: infinity propagating as NaN downstream costs: one transform All four together are under a second, and each covers a class the others are blind to.

Verify the fix

Run the assertions as a start-up guard rather than as ordinary tests, so a mis-provisioned environment fails before the suite spends minutes producing misleading geometry results:

pytest -q tests/test_datum.py --maxfail=1

A single command with --maxfail=1 is right here because these assertions are dependent: if the grid file is missing, every subsequent numeric result is a consequence rather than a cause, and reporting all of them buries the one that matters.

Where to pin, and how hard

There are three strengths of pinning available, and they trade reproducibility against maintenance. Choosing deliberately is better than defaulting to the weakest, which is what from_crs alone amounts to.

Assert the outcome. Keep from_crs, and add the availability and accuracy assertions above. PROJ still chooses, but a change in what it chooses fails the suite. This is the right default for most pipelines: it survives a data-package upgrade that improves the operation, and it fails one that degrades it.

Name the operation. Look up the operation’s identifier and construct from it, so PROJ performs no ranking at all. Appropriate when a regulator or a downstream consumer requires a specific transformation by name, and the cost is that a genuinely better operation will not be adopted without a code change.

Pin the full pipeline string. Construct from an explicit PROJ pipeline definition, which fixes every step including the grid file used. Maximum reproducibility, maximum maintenance: the string is opaque to most readers and must be regenerated whenever anything about the transformation is intentionally changed.

Three strengths of pinning, and their trade-offs Three options in increasing order of strictness. Asserting the outcome keeps the automatic selection but adds availability and accuracy assertions, so an improved operation is adopted silently while a degraded one fails the suite; it is marked as the sensible default. Naming the operation constructs the transformer from a specific operation identifier so no ranking occurs, guaranteeing exactly which transformation runs but requiring a deliberate code change before a better one can be used. Pinning the full pipeline string fixes every step of the transformation including which grid file is applied, giving the strongest reproducibility guarantee at the cost of a definition most readers cannot interpret and which must be regenerated by hand whenever the transformation is intentionally changed. STRENGTH WHAT IT FIXES WHAT IT COSTS Assert the outcome from_crs + availability + accuracy a change in choice fails improvements adopted freely nothing — the sensible default Name the operation construct from an identifier no ranking happens at all exactly one transformation a better path needs a code change Pin the pipeline string every step, including the grid byte-level reproducibility nothing is inferred opaque to readers; regenerated by hand Start at the top row. Move down only when something external — a regulator, a contract, a reproducibility requirement — demands it.

The first row is right for the large majority of pipelines, and the reason is worth stating: pinning too hard has a real cost. A transformation named explicitly will not benefit when a national mapping agency publishes an improved grid, and nobody will notice for years, because the assertion that would have surfaced the improvement was replaced by a hard-coded choice. Asserting the outcome keeps the upgrade path open while still failing when the environment degrades.

Failure modes and edge cases

  1. Assuming a symmetric transformation. The forward and inverse can select different operations when one direction has a grid and the other does not. Assert the operation in both directions, not just the one your pipeline uses most.
  2. Points near the diagonal. A monument whose easting and northing are numerically close makes an axis-order swap undetectable. Choose a reference point whose two coordinates differ substantially.
  3. A transformation that is correct and irrelevant. Testing EPSG:4326 to EPSG:3857 exercises a projection with no datum change at all, so it passes regardless of grid availability. Test the pair your data actually crosses.
  4. Vertical components. A horizontal transformation says nothing about heights. If elevation is carried, the vertical datum needs its own operation assertion and its own budget, and the difference between an ellipsoidal and a geoid-based height can be tens of metres.
  5. Caching the transformer across CRSs. A module-level transformer built for one pair silently applies to another if a caller passes different data. Key any cache on the CRS pair, or construct per use.
  6. Grid files present but wrong version. Two data packages can both contain a grid of the same name with different contents. Assert the package version alongside the operation, or the check proves only that a file exists.

Conclusion

A Transformer constructed with from_crs and no further assertions is a request for whatever PROJ can manage in the current environment, and that is not a specification. Enumerating the candidates, pinning the operation, asserting its reported accuracy, and validating one externally-published coordinate converts a datum shift from an environment-dependent behaviour into a tested contract — which is the whole point of treating coordinate reference system testing as a layer of its own.