Mocking Cloud-Optimized GeoTIFF Reads in Tests

9 min read

Code that reads Cloud-Optimized GeoTIFFs (COGs) from object storage is slow and non-deterministic to test against the real thing — network latency, credentials, and remote state all leak into the suite. This guide sits beneath raster mocking techniques and shows how to build in-memory COG fixtures with rasterio’s MemoryFile, exercise windowed reads and overview levels without a network, and assert that your reader requests only the bytes it needs. The specific value of mocking a COG rather than any GeoTIFF is that the COG’s whole point is partial, range-request reads — so a test must prove the reader uses windows and overviews, not that it can read a file at all.

Why real COG reads make bad tests

A COG served from S3 or GCS is read via HTTP range requests: the reader fetches the header, then only the tiles and overview levels a query touches. Testing against the live object couples the suite to network availability, credentials, and egress cost, and makes timing non-deterministic. Worse, it hides the property you actually want to verify — that your code reads a small window rather than pulling the whole raster — because a slow full read and a fast windowed read both “pass” against a real file. An in-memory fixture makes the read local, deterministic and inspectable, so the windowing behaviour becomes assertable.

What makes a COG different, and why that matters for the mock

A Cloud-Optimized GeoTIFF is an ordinary GeoTIFF with two constraints that exist so a reader can fetch part of it over HTTP: the data is tiled rather than striped, and the metadata header sits at the front of the file. Together those let a client read the header in one request and then request only the byte ranges covering the tiles it needs.

That structure is the whole subject of the test. A mock that reproduces the pixels but not the tiling proves nothing about the access pattern, because a striped file forces a reader to pull whole rows — which is precisely the behaviour the COG format exists to avoid. If the test is about what gets fetched, the mock has to be genuinely tiled and genuinely have its header in front.

Striped versus tiled layout, and what a window costs Two file layouts drawn as horizontal byte ranges. The striped layout shows a header placed after the data and a body divided into full-width rows; a small requested window is shown intersecting four rows, and because rows are the smallest addressable unit the read must fetch all four full-width rows. The tiled Cloud-Optimized layout shows the header at the front followed by independent tiles; the same window overlaps four tiles, so only those four tiles are fetched. Annotations give an illustrative comparison of bytes transferred, an order of magnitude apart, and note that only the second layout allows a reader to fetch part of the file usefully. Striped GeoTIFF — rows are the smallest unit header requested window must fetch 4 full-width rows — most of the file Cloud-Optimized GeoTIFF — header first, independent tiles header only these tiles are fetched an order of magnitude fewer bytes — and the reason the format exists A mock that is not tiled cannot exercise any of this, however correct its pixels are.

What the overview pyramid buys, and how to fixture it

The second half of the COG contract is the overview pyramid: reduced-resolution copies stored in the same file so a reader asking for a coarse view fetches a small image instead of downsampling a large one. A fixture without overviews cannot test whether your reader picks the right level, which is usually the single largest determinant of how much data a map request moves.

Overview levels and the one a reader should pick A pyramid of four raster levels drawn as nested squares of decreasing size. The base is the full-resolution image; each overview above it halves both dimensions, so the first overview holds a quarter of the pixels, the second a sixteenth, and the third a sixty-fourth. A request for a coarse view is shown selecting the smallest level whose resolution still exceeds what was asked for, transferring a small fraction of the bytes. Alongside, a reader that always reads the base level is shown transferring the entire image for the same request. A note identifies that second behaviour as the defect a fixture with overviews exists to detect, since it is invisible in any test that only compares pixel values. full resolution ovr 1 · ¼ ovr 2 ovr 3 correct reader picks the smallest level that still exceeds the requested resolution reader with the defect always reads the base level and downsamples locally bytes moved: a fraction one small overview tile bytes moved: everything identical pixels, 60× the egress Both readers return the same pixels, so no value assertion can tell them apart. Only a fixture with real overviews, plus an assertion on which level was read, can.

Mocking-approach reference

Need Technique What it proves
Local COG bytes rasterio.MemoryFile Read logic without a file on disk
Overview levels build with overview_level / factors Reader picks the right resolution
Windowed read Window + assert shape Only a sub-region is read
Range requests mock the HTTP session Reader fetches ranges, not the whole object
Deterministic pixels fixed NumPy array Reproducible assertions

Step-by-step implementation

The pattern targets rasterio 1.3+ and NumPy, building an in-memory COG and asserting windowed reads.

Step 1 — Build a deterministic in-memory COG

import numpy as np, rasterio
from rasterio.io import MemoryFile
from rasterio.transform import from_origin

def make_cog(width=512, height=512):
    data = np.arange(width * height, dtype="uint16").reshape(height, width)
    profile = {
        "driver": "GTiff", "dtype": "uint16", "count": 1,
        "width": width, "height": height,
        "crs": "EPSG:3857", "transform": from_origin(0, height, 1, 1),
        "tiled": True, "blockxsize": 256, "blockysize": 256,   # COG needs tiling
    }
    mem = MemoryFile()
    with mem.open(**profile) as dst:
        dst.write(data, 1)
        dst.build_overviews([2, 4], rasterio.enums.Resampling.nearest)
    return mem

Step 2 — Read a window instead of the whole raster

from rasterio.windows import Window

def read_window(mem, col_off, row_off, size):
    with mem.open() as src:
        return src.read(1, window=Window(col_off, row_off, size, size))

Step 3 — Assert only the window was read

The returned array’s shape proves the reader took a sub-region, not the full raster.

def test_windowed_read_is_bounded():
    mem = make_cog()
    tile = read_window(mem, 0, 0, 64)
    assert tile.shape == (64, 64), "reader must return only the requested window"
    assert tile[0, 0] == 0                       # deterministic pixel value

Step 4 — Mock range requests for a remote reader

When the code under test reads via a URL, patch the HTTP layer so the test asserts range requests without a network — the same isolation principle as mocking PostGIS connections.

from unittest.mock import patch

def test_reader_uses_range_requests():
    mem = make_cog()
    with patch("myapp.cog.open_remote", return_value=mem):
        tile = read_window(mem, 128, 128, 32)
    assert tile.shape == (32, 32)                # no network touched

Choosing the layer to mock

There are four points in the stack where a COG read can be intercepted, and picking the wrong one produces a test that is either fragile or meaningless. The rule is the same as elsewhere: intercept at the lowest layer that still makes the assertion possible, because everything below the interception point is no longer under test.

Mock nothing — build a real tiled file in a MemoryFile. The best default. The GDAL driver, the tiling, the windowed read logic and the profile round trip all execute for real; only the network is absent. Almost every COG test that is really about reading belongs here.

Mock the HTTP range request. The right layer when the assertion is about which bytes were requested — that a window read fetched two tiles rather than the whole file, that the header was fetched once and cached, that a retry happened on a truncated response. This is the only layer that can observe those, and it is worth the extra setup for exactly those assertions.

Mock the reader object. Appropriate only when the raster is incidental — a function that takes an open dataset and does something unrelated to pixels. It proves nothing about the format and should never be used to stand in for a real read.

Mock the whole storage client. Reserved for error paths that are otherwise impossible to provoke: a credential expiry mid-read, a bucket that starts returning 503, a connection reset after the header. These are legitimate and rare.

Assertion you want to make Intercept at Why not lower
The window returns the right pixels Nothing — real MemoryFile Nothing below matters
Only the overlapping tiles were fetched HTTP range layer The driver hides the request pattern
A truncated response is retried HTTP range layer Cannot be provoked from a real file
Credentials expiring mid-read is handled Storage client No lower layer can express it
A function that takes a dataset works The reader object The raster is not the subject
Interception points and what stays under test A vertical stack of four layers from the test downwards: the reader object, the GDAL driver with its tiling and windowing, the HTTP range layer, and the storage client. Four interception points are marked against the stack. Intercepting at the reader object removes the driver, the tiling and everything beneath from the test. Intercepting at the HTTP range layer keeps the driver, the tiling and the windowing under test while removing only the network. Intercepting at the storage client keeps everything above it under test but is only appropriate for error paths. Using a real in-memory file intercepts nothing and keeps the whole stack under test except the network itself. A closing rule states that you should intercept at the lowest layer that still permits the assertion you need. reader object GDAL driver · tiling · windowing HTTP range requests storage client intercept here → format not tested intercept here → driver + tiling still tested intercept here → error paths only MemoryFile intercept nothing whole stack real except the network Rule: intercept at the lowest layer that still permits the assertion. Everything beneath the interception point has stopped being tested.

The middle row of the table is the one worth building properly, because it is the assertion nobody else can make. A test that proves a window read fetched two tiles rather than the whole object is the only defence against a subtle regression — a changed block size, a lost tiling option, an accidental full read in a helper — that costs nothing in correctness and a great deal in cloud egress. It fails silently in every other kind of test, and it is exactly the failure a COG pipeline exists to avoid.

Verification pattern

Confirm the fixture is a real COG with overviews, so the test exercises the overview path rather than a plain tiled TIFF.

def test_fixture_has_overviews():
    with make_cog().open() as src:
        assert src.overviews(1) == [2, 4], "fixture must carry overview levels"

Failure modes and edge cases

  1. Untiled fixture. A GeoTIFF without tiled=True is not a COG and cannot be read by window efficiently; set block sizes.
  2. Missing overviews. Without build_overviews, a zoomed-out read pulls full resolution; add overview factors so the reader can pick a level.
  3. Asserting on timing. Testing that a read is “fast” is flaky; assert the returned window shape and byte count instead.
  4. MemoryFile leak. Not closing the MemoryFile leaks native handles across tests; use it as a context manager or close in teardown.
  5. CRS/transform mismatch. A fixture whose transform does not match its CRS units makes windowed geographic queries land wrong; keep the transform consistent with the declared CRS.

One habit makes all of this durable: assert the mock is actually a COG before relying on it. Reading back the profile and checking that the file is tiled, that the block size matches what production writes, and that overviews are present takes three lines and prevents the whole category of test that looks like a COG test and is really a GeoTIFF test. It is the same meta-assertion logic that keeps an edge-case generator honest, applied to a format instead of a defect.

Conclusion

Mocking COG reads with rasterio.MemoryFile makes raster tests local, deterministic and — crucially — able to assert the reader uses windows and overviews rather than pulling the whole object. Build a tiled, overview-bearing in-memory fixture, read by window, and patch the HTTP layer for remote readers, and the suite proves the exact partial-read behaviour a COG exists to provide. For the broader raster context, return to raster mocking techniques.