nanolab

A harness for deciding whether a small vision model is worth shipping.

The models are not the point. This harness has killed more candidates than it has shipped, which is the behaviour you want from a harness. It exists because the most common failure in small-model work is not a bad model — it is a good-looking number that a single threshold would have matched.

from nanolab import scalar_baseline, verdict, report
from nanolab.answerability import answerable, zoom_for, camera_spec

1. The bar: cheap baselines that are actually hard to beat

cheap_stats computes ten statistics per image — six global (mean, std, Laplacian variance, high-frequency ratio, gradient, entropy) and four spatially aware (centre-vs-edge, radial slope, row and column FFT peaks).

The spatial four exist because of a measured mistake. With global statistics only, a vignetting task showed a baseline of 0.707 against a model's 0.967 — an apparent +0.261 margin. Adding a radial-slope statistic took the baseline to 0.907 and the margin to +0.060. A baseline that cannot see where anything is loses to a CNN by default, and flatters it.

For multi-class problems a single threshold is too weak a bar (the implementation uses two cuts, so three bins, and cannot express six classes). Use transferred_baseline plus a small linear model over the same statistics — a ~66-parameter logistic. On one task that raised the bar from 0.438 to 0.543 and cut the model's apparent margin from +0.183 to +0.077.

2. The gates: four, in order of what they rule out

verdict() is the single place the logic lives, because re-implementing gates is how gates get silently dropped.

1. Utility — lift over the majority class ≥ 0.15. Statistically real is not the same as useful.
2. Ordinal — for ordered targets, mean error below a bin threshold.
3. Scientific — held-out model vs the in-sample scalar. Deliberately optimistic. Asks: is there structure beyond a low-order statistic?
4. Engineering — transferred model vs transferred scalar. Asks: what should actually ship?

Gates 3 and 4 disagree, and that is the point. On block-grid detection the in-sample scalar read 0.954 while the same threshold transferred read 0.565.

3. Answerability: is the task possible at all?

Based on Johnson's criteria (John Johnson, 1958): ~2 px across a target to detect it, 8 to recognise its type, 12.8 to identify a specific one. Measuring CNN attribute classification against per-attribute AP reproduces the recognition figure — accuracy degrades sharply below ~8 px on the diagnostic feature, and the effect survives controlling for training-set size (+0.074 / +0.112 / +0.144 within support bands).

answerable(feature_mm=6, reference_mm=1700, reference_px=0.25*720)
# (False, 0.6, 8.0)   -- a nasal cannula from a bedside 720p camera. Not a model problem.

zoom_for(feature_mm=5, input_px=224)
# 140.0   -- crop to 140 mm to put 8 px on a 5 mm feature. Zoom, do not upscale.

This is how we established that IV lines and nasal cannulae are hopeless from a room camera at any practical resolution — eight pixels on a 6 mm cannula at bedside framing needs a 9,067-pixel-tall sensor. That conclusion cost one line of arithmetic instead of a labelling campaign.

4. Diagnostics: does resolution buy anything?

resolution_sweep sweeps input size at constant parameter count (valid only for global-average-pooled architectures, else it confounds pixels with capacity).

A result that held in three independent domains — garment attributes, clinical attire, clinical scenes: cheap baselines are flat with resolution while learned models are not.

domain	model 64 → 256	baseline 64 → 256	margin
clinical attire (mask head)	0.401 → 0.587	0.280 → 0.282	+0.121 → +0.305
clinical scene (hard task)	0.640 → 0.699	0.612 → 0.620	+0.028 → +0.079

Global aggregates wash out detail however many pixels they are given. Resolution is precisely where a model earns its keep over a threshold — and if your margin does not grow with resolution, the model may not be doing anything a statistic could not.

5. Splitting: by group, never by sample

Adjacent samples are near-duplicates. A random split puts them on both sides and reports a number that means nothing — one quality-gate result went from a meaningless 1.000 to an honest +0.238 when split by group.

Two traps we hit and you will too:

• Group splits must require both classes on both sides. A size-only split once produced a test set that was 93% one class, and the model scored 0.070 — far below chance.
• Weight by test-set size when groups differ in size. Unweighted averaging over unequal splits reported +0.215 where the sample-weighted figure was +0.129. Constructed units (one image, one patch) are uniform and safe; natural units (a run, a session, a patient) are not.

Installation

pip install -e .

Why publish a harness rather than a benchmark

A benchmark tells you who won. This tells you whether the contest was worth entering. Of the models built with it, several were refused for reasons worth naming: occupancy lost to a threshold on thermal image variance (0.85 vs 0.77); shot-scale lost to a threshold that was itself barely above chance; a studio-vs-street classifier scored 0.994 where one entropy threshold got 0.957.

All three looked like results until the baseline was computed.