You inherited a computer-vision codebase with no trustworthy tests. The tempting move is to save a set of images and assert that every future output matches them. I would not make pixel-perfect golden images the default: they can fail after harmless rendering changes while missing errors in the decisions users depend on. Start by testing the boundaries and relationships that should survive a change of camera, codec, or model.
Pixel-perfect snapshots protect the wrong contract
A golden image answers one narrow question: did this run produce the same pixels as an earlier run? That is useful when pixels themselves are the product, but many vision services return boxes, labels, measurements, or decisions. A test that compares annotated output images can pass while a caller receives the wrong box coordinates, because the annotation code and the response serializer need not use the same values. It can also fail when a font-rendering update moves a label by one pixel without changing the response.
In an untested repository, an existing output is not automatically a correct output. Approving hundreds of snapshots on day one turns unknown behavior into an authority, because subsequent reviewers tend to treat any visual difference as a regression. I would not bulk-approve snapshots from the current build for that reason. Preserve a few as diagnostic artifacts, but require a human-checked expectation before calling one a test oracle.
Trace one request before choosing assertions. Record where the bytes enter—perhaps an HTTP multipart/form-data upload with MIME type image/jpeg—and where decoding, orientation, resizing, inference, filtering, and serialization happen. OpenCV’s imread convention is BGR, while Pillow commonly exposes RGB images; a channel swap can leave a plausible-looking picture yet change a model’s predictions. JPEG EXIF orientation can likewise make an otherwise correct box refer to the wrong coordinate frame. These are boundary contracts worth testing separately, because a final screenshot does not identify which boundary failed.
The examples in Computer Vision in Software Development: Practical Uses can suggest useful fixture shapes, but examples of applications cannot establish what this inherited build ought to return. Select fixtures from actual input paths instead: an uploaded JPEG with orientation metadata, an image without detections, and an image containing an object near an edge. For each, write down whether coordinates belong to the original image or the resized tensor. If nobody can answer, that ambiguity is the first defect your tests should expose.
Metamorphic checks give you a starting oracle without pretending the old build is correct
You may not know the right answer for every photograph, but you can often state how an answer should behave after a controlled change. A modest brightness increase should leave the location of a high-contrast object alone, provided the production requirement includes that lighting range. Re-encoding the same image should not change a returned class merely because the JPEG encoder changed, provided the re-encoding does not erase the object. These are metamorphic checks: they compare related inputs instead of treating the current output as truth.
Here is a small test that actually runs with pytest 8.3.3, NumPy 1.26.4, and opencv-python 4.10.0.84. Save it as test_box.py and run python -m pytest -q test_box.py. Its toy detector makes the property visible; replace detect_box with your repository’s preprocessing and detection boundary before claiming coverage of production code.
import cv2
import numpy as np
def detect_box(image):
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
_, mask = cv2.threshold(gray, 100, 255, cv2.THRESH_BINARY)
contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
return cv2.boundingRect(max(contours, key=cv2.contourArea))
def test_brightness_preserves_box():
image = np.zeros((64, 64, 3), dtype=np.uint8)
cv2.rectangle(image, (12, 18), (40, 46), (180, 180, 180), -1)
brighter = np.clip(image.astype(np.int16) + 20, 0, 255).astype(np.uint8)
assert detect_box(image) == detect_box(brighter)
The synthetic fixture yields a measured bounding box of 29 by 29 pixels; that measurement follows from OpenCV’s inclusive rectangle endpoints, not from a claim about model accuracy. The grayscale threshold of 100 is a value to tune for this example, not a sensible universal production setting. Once the test calls real code, choose the brightness change from camera data: an arbitrary transformation can create a failing test for a condition the product never promised to handle.
Metamorphic tests also need limits. A horizontal flip may preserve an object’s class, but its expected x-coordinate must be transformed; asserting an identical box would encode a mistake. A crop can remove context required by a classifier, so class invariance after every crop is not a fair demand. Put each relationship in words beside the test, including the allowed input range and coordinate transform. That gives the next QA engineer a reason to challenge an assertion rather than merely update its expected value.
Contract tests beat model-score tests for the first round of protection
The popular alternative to snapshots is to start with a large labeled set and a single model score. I would postpone making that the primary regression gate, because a good aggregate score can hide a broken upload path, coordinate conversion, or empty-result response. COCO mAP@[0.50:0.95] averages performance across intersection-over-union thresholds, but it does not verify that your API expresses boxes in the units its callers expect. Use mAP to compare detector quality after the request-to-response path has a contract.
Write that contract at the seams. For an uploaded image, assert the decoder reports the expected width, height, channel order, and EXIF-corrected orientation before inference. For a detection response, assert that every box has finite numeric coordinates, lies inside the documented coordinate frame, and satisfies its stated ordering. A JSON Schema Draft 2020-12 schema can catch missing fields and wrong types; a small pytest assertion must still check cross-field rules such as x_min < x_max. Test the empty-detection response explicitly, because callers often take a different branch when the list is empty.
Keep inference separate enough to tell which kind of change broke a test. Pin the ONNX Runtime version and the model artifact checksum in CI, because a changed runtime or model makes output differences hard to attribute to application code. Run a few fixtures through the real runtime to catch wiring errors, while using a controlled detector result to test filtering and serialization. The controlled result is cheaper and more precise for a serializer test; the real-runtime fixture wins when the question is whether preprocessing still feeds the model the expected tensor.
Treat Computer Vision in Software Development: Key Use Cases as a prompt for choosing distinct input classes, not as a source of expected outputs for this repository. A test set assembled around appealing examples can miss the images your service actually rejects. Inspect production-safe samples, incident reports, and request validation rules before deciding which cases deserve permanent fixtures. Where images cannot leave an environment, record reproducible metadata and create synthetic boundary cases rather than silently omitting that path from testing.
Use snapshots selectively, where their maintenance cost buys information
Pixel snapshots versus semantic invariants is not a contest with one winner in every situation. Pixel snapshots win for a rendering component whose specified output is an image—such as a fixed overlay exported for review—because a changed pixel is then potentially the defect. Their cost is fixture storage and review of differences caused by fonts, encoders, and rendering environments. Semantic invariants win for detection APIs and preprocessing pipelines because they assert the caller-visible behavior through benign pixel variation. Their cost is design work: someone must define valid transformations, tolerances, and coordinate frames.
If a rendered output really needs comparison, choose the metric to match the failure. Exact pixel equality detects any change but produces noise after harmless rendering shifts. SSIM from scikit-image can tolerate some local differences, but a high similarity score can conceal a small, consequential missing annotation. An ImageHash perceptual hash is useful for finding broadly similar images, not for certifying the position of a small box, because its compact representation discards spatial detail. None should replace an assertion on the box returned to a caller when that box is the contract.
Make failures reviewable rather than merely green or red. Store the input, response JSON, model checksum, and a difference image for the small number of end-to-end fixtures; Git LFS or DVC can keep large binary fixtures out of ordinary source diffs. Log decoder failures and empty detections as separate categories, because combining them makes a pipeline outage look like a model-quality problem. Once tests run reliably, inspect the 95th-percentile request latency as a measured operational metric alongside correctness; a detector that returns the right answer only after the request deadline is still unusable to its caller.
Start tomorrow with one request that currently succeeds and one that returns no detections. Trace both from image bytes to response JSON, then write down the coordinate frame and empty-result shape before adding a test. If those facts are undocumented, ask the caller to confirm them. That small contract gives your first failing test a meaning; a folder of unexamined golden images does not.
