docs/EVALUATION.md

# Evaluation

**Status tags used in this document:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` ·
`NOT RUN` · `BLOCKED` · `DEFERRED` · `REJECTED` · `OPEN` · `RESOLVED` · `CLOSED`.

This document describes **how** every number in [`BENCHMARKS.md`](BENCHMARKS.md) was produced, and it
enforces the project's evaluation-honesty rules. It is deliberately conservative: a metric that was not
measured is tagged `NOT RUN`; a metric that got **worse** is shown getting worse; a metric whose
acceptance ruling is undecided is shown as `OPEN`.

It is written to be read in full. It is not a summary. Every protocol, threshold, sample count and
artifact key path below was read from a file in the source repository; where the evidence does not
establish something, the text says so explicitly rather than filling the gap.

**Companion documents.**

| Document | What it covers |
|---|---|
| [`BENCHMARKS.md`](BENCHMARKS.md) | The headline metric table, with source artifacts |
| [`LIMITATIONS.md`](LIMITATIONS.md) | An honest catalogue of everything not done or done poorly |
| [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) | The reproduction contract, the invariants, the commands |
| [`MODELS.md`](MODELS.md) | The six released artifacts and their backbones |
| [`TRAINING.md`](TRAINING.md) | How each artifact was trained |

---

## 1. Why this document exists, and the strategy it serves

### 1.1 Evaluation-first strategy

The project's master architecture plan (`Implementation and Architecture plan.md`, §2 "Evaluation-First
Strategy") fixes the engineering priority order as:

```
1. Optical-SAR joint reasoning
2. Change analysis
3. Single-image VQA
4. Grounding
5. Captioning
6. Agent/routing/evidence/reliability
```

That is an **engineering priority, not an official score ranking**. The plan is explicit (§2.1) that it
is not a claim about how a competition would weight the tasks. The plan also fixes (§2.2) which metrics
belong to which task family:

| Task family | Plan §2.2 metric set |
|---|---|
| VQA | exact match; normalized exact match; F1 where appropriate |
| Captioning | the benchmark-prescribed metrics; locally additionally BLEU, ROUGE-L, CIDEr, BERTScore |
| Grounding | IoU; Recall@IoU; mAP where applicable |
| Change | precision; recall; F1; IoU; mIoU |
| Change VQA | answer accuracy; semantic match if the benchmark specifies it |
| Optical-SAR | *task-dependent*: classification accuracy/F1, VQA accuracy, region IoU, mask IoU, change metrics |

The plan states a hard prohibition on the optical-SAR row (§2.2): **"Do not invent an official multimodal
metric."** That prohibition is implemented in code, not just documented — see §3.5.

### 1.2 Leakage prevention, the test firewall and the hidden firewall

The plan devotes three sections to the machinery that makes a number trustworthy at all:

- **§36 Data Leakage Prevention** — every sample carries `dataset_id`, `scene_id`, `sample_id`,
  `source_scene`, `geographic_hash`, `acquisition_date`, `sensor`, `sha256`, `split`; splits are assigned
  by **scene**, never by randomly splitting neighbouring patches.
- **§37 Test Set Firewall** — `evaluation/public_test/` is immutable; training code may not import it;
  evaluation code may read it only in *evaluation mode*; **no cache of test answers is permitted**.
- **§38 Hidden Set Firewall** — there must be **no** `hidden_test_mode` in training; the hidden interface
  accepts input imagery and produces the standard result schema, with no knowledge of the hidden answer.

All three are enforced as **mechanisms** rather than conventions (§3.2–§3.4 below). The reason is stated in
the source: *"a rule enforced by convention is a rule that holds until someone is in a hurry"*
(`evaluation/public_test/corpus.py`).

### 1.3 Stop/go gates and the completion checklist

The plan's §72 Stop/Go Gates define the checkpoints a benchmark must pass before it is allowed to
contribute to a final evaluation. **Gate 5** is the one that governs this document:

```
Gate 5 — proceed to final evaluation only when:
  [ ] all prompts frozen
  [ ] all thresholds frozen
  [ ] model revisions frozen
  [ ] config hash recorded
  [ ] public test isolation verified
```

Gate 3 governs grounding (`RemoteCLIP loads` / `VRSBench boxes parse correctly` / `coordinate conversions
tested`). The plan's §79 Completion Checklist enumerates, under `EVALUATION`: manifests, leakage scans,
benchmark adapters, metrics, normalization, immutable public test, reproducible run manifest.

**Status of the checklist for the evaluation family, honestly:**

| Checklist item (§79) | State |
|---|---|
| manifests | `IMPLEMENTED` + `MEASURED` (`evaluation/manifest_freeze.json` exists) |
| leakage scans | `IMPLEMENTED` (`evaluation/leakage.py`) — enforced by raising, not warning |
| benchmark adapters | `IMPLEMENTED`, registry empty at import — adapter-based benchmark runs are `NOT RUN` |
| metrics | `IMPLEMENTED` + `MEASURED` (see §4) |
| normalization | `IMPLEMENTED` (`evaluation/normalize.py`); identity-only, deliberately |
| immutable public test | `IMPLEMENTED` (seal/verify machinery) — the corpus directory is **empty**, reported `available=False` |
| reproducible run manifest | `IMPLEMENTED` (`evaluation/run_manifest.py`) — populated from measurement, never placeholders |

### 1.4 The single most important rule

**Do not fabricate.** A number with no artifact does not appear in these docs. Where evidence is missing,
this document writes `UNKNOWN — not established from the available evidence`.

---

## 2. The eight evaluation-honesty rules, in full

These eight rules are the spine of the document. Each is stated, then given its rationale, then given its
enforcement (by tooling, by test, or by explicit convention), and then — where relevant — a counter-example
showing what would go wrong if it were violated.

### Rule 1 — Evidence before claims

**Rule.** Every reported number has an artifact path. A number with no artifact does not appear in the docs.

**Rationale.** The failure mode this prevents is the *plausible number*: a figure that looks like a result,
appears in a summary, and is reproduced forever after because nobody re-derives it. The project hit this
class of bug repeatedly and records the hits (see `docs/PHASE12_115_METRIC_COMPUTED.md` §8, which documents
an earlier revision that published `2,179 passed` computed as `2,171 + 8` — arithmetic presented as a check,
where the total was never measured).

**Enforcement.** [`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py) walks each quoted
claim to its source artifact and compares at the printed precision. It is committed with its output
(`../tools/readme_metrics_report.txt`) and exits non-zero on any mismatch. See §7.6.

**What it forbids here.** No system-level end-to-end accuracy, no composite score, no "overall" number.
There is no such artifact, so there is no such number (§8).

### Rule 2 — Two protocols are never collapsed

**Rule.** Grounding is reported under the **canonical** protocol **and** the **matched6** protocol. Never
quote one alone.

**Rationale.** The two protocols are different measurement procedures over the same 16,159 VRSBench records,
and they give materially different answers: canonical mean best IoU **0.2838**, matched6 **0.2566**. Quoting
0.2838 alone would be selective; quoting 0.2566 alone would understate the shipped decode. Both are true
statements about different procedures.

**Enforcement.** Both protocols have their own artifact file, both are in the verification tool's claim list
(§7.6), and both appear in the headline table.

**Counter-example.** A reader who saw only `0.2838` would conclude the head is ~11 % better than it is under
the stricter matching protocol. That difference is larger than several of the margins the project treats as
decision-relevant elsewhere.

### Rule 3 — Two test sets are never collapsed

**Rule.** Change-VQA is reported on `test` **and** `test2`.

**Rationale.** The two sets disagree, and they disagree in the direction that flatters a selective report:
`test` accuracy **0.697626** / macro-F1 **0.378373**; `test2` accuracy **0.651469** / macro-F1 **0.372309**.
The `test` numbers are the higher pair.

**Enforcement.** All four values are in the verification tool's 20 claims; the artifact
(`artifacts/change_vqa/run/PROMOTION.json`) records all four under `verification`.

**Counter-example.** Quoting `0.697626` alone overstates accuracy by 4.6 pp against the second test set.

### Rule 4 — `accuracy` never travels without `macro-F1` for imbalanced multi-class heads

**Rule.** For the imbalanced multi-class heads (optical-SAR fusion, change-VQA), the macro-F1 is reported
alongside accuracy, always.

**Rationale.** Optical-SAR fusion reaches **0.931** accuracy with **0.434161** macro-F1. Those two numbers
describe very different things: the model is accurate on frequent classes and weak on rare ones. Accuracy
alone would read as "solved"; macro-F1 alone would read as "broken". Neither reading is correct on its own.

**Enforcement.** The pair is quoted together in every table; the verification tool asserts both.

**A subtlety that is itself documented as a trap.** The macro-F1 denominator is **all 19 class slots**, not
the 14 classes present. Five classes absent from the scored split contribute exactly `0.0`, which pulls the
mean down without describing any prediction the head made. The artifact records
`macro_f1_denominator: "all 19 classes (absent classes contribute 0.0)"`, plus `classes_present` and
`classes_absent`, precisely so the figure cannot be misread. The two denominators give two numbers:
`0.434161` (19 slots) and `0.589218` (14 present classes). Both are reproducible; only the 19-slot figure is
the pre-registered definition. See `docs/PHASE12_115_METRIC_COMPUTED.md` §3.5.

**A correction recorded rather than buried.** That same document records that an earlier revision of §3.5
over-generalised the absent-class argument to all seven zero-valued slots. The honest reading is **both**
statements: part of the low macro-F1 is populational (5 absent classes), *and* there is genuine per-class
failure — classes **5** and **6** are **present** in the scored split and score `0.0` (real total misses).
A class that is absent appears in `classes_absent`; classes 5 and 6 do not. The document states that the
error "was wrong in the flattering direction" and was caught by an independent probe that disagreed.

### Rule 5 — Validation is not test

**Rule.** The router figure is labelled **validation, ungated, n = 86**. The test split was **NOT RUN**.

**Rationale.** The router's `artifacts/router/threshold_sweep_val.json` records `split: "val"`,
`n_val: 86`, `n_val_examples_scored: 86`, `n_test_examples_scored: 0`, and `test_split_touched: false`.
The plan's §59 Testing section sets an engineering acceptance target of `task accuracy ≥ 95 %` measured on
`500 validation queries`, `100 hard negatives`, `50 unsupported queries`. The router corpus has **86** val
examples and **0** hard negatives in val (they are held out to test by design). So the recorded number is
**corpus-limited**: `corpus_limited: true`, with the artifact's own note saying this is *"NOT a calibration"*
and that *"the corpus was NOT padded with generated queries"*.

**Enforcement.** The label "overall **ungated** accuracy" is used everywhere; the artifact note travels with
the number; the verification tool asserts `corpus_limited` and `n_val`.

**Counter-example.** Presenting `0.965116` as a test result would be presenting a number measured on 86
examples — with zero hard negatives — as if it were measured on the plan's 500/100/50 corpus.

### Rule 6 — A negative result stays negative

**Rule.** Calibration ECE **worsened** (`0.013755 → 0.014929`) and is shown worsening. It is retained only
because it is part of the frozen configuration, **not** because it helped.

**Rationale.** The fitted temperature is `T = 0.9772732` and `ece_improvement = −0.001174` — negative. The
NLL improved microscopically (`0.689741 → 0.689631`, `nll_improvement 0.00011`) but the calibration error
metric got worse. Reporting the NLL improvement while omitting the ECE regression would be a selective
reading of the same artifact.

**Enforcement.** Both ECE values are in the verification tool's 20 claims; the tool prints
`(negative => calibration did NOT help)` next to `ece_improvement`; the reliability curve plotted on the
Benchmark page is explicitly labelled as the **pre-scaling** diagram, and the calibrated curve is **not
plotted** at all.

### Rule 7 — `USABLE ≠ ACCEPTED`

**Rule.** The VLM adapter's metrics are real (exact_match **0.963**, F1 **0.96432**); its status is
**ACCEPTANCE-REJECTED**.

**Rationale.** The artifact `artifacts/vlm/phase6_closure.json` states the distinction in its own words:

> *"'Verified' answers: is this artifact the one we trained, and does it work? 'Accepted' answers: did it
> clear the bar predeclared before we looked? Both are true, and they are different questions."*

The aggregate endpoint improved by **+49.5 pp** over the unadapted baseline, yet the run is `REJECTED`
because the pre-registered per-class guardrail (V2) failed on one class. A model can be a working
engineering artifact and a rejected research result at the same time.

**Enforcement.** The verification tool asserts that the artifact's `headline` contains the literal string
`ACCEPTANCE-REJECTED`. The deployed caption/VQA path therefore uses the **unadapted** SmolVLM; the adapter is
attachable only via the `SATQUERY_VLM_ADAPTER` environment variable (resolved in `specialists/vqa/model.py`).

### Rule 8 — No composite / vanity score

**Rule.** There is no single headline accuracy for the system, and none is invented by averaging the
per-task numbers.

**Rationale.** The plan (§63 Metric Normalisation) fixes the aggregation surface as
`aggregate: official_weights: null` "until the organizers publish them", and states that the specification
**explicitly prohibits inventing an official aggregate formula**. Averaging per-task metrics with invented
weights would be exactly that.

**Enforcement.** `evaluation/normalize.py` defines `official_weights: None` and an `aggregate()` function
that **raises** while it is `None`. The evaluation runner *calls* `aggregate()` on every run and records the
refusal in the report's `aggregate_score` block, so the prohibition is exercised rather than merely
documented. See §3.5.

---

## 3. The evaluation architecture

The evaluation subsystem is the single place where a benchmark score could be produced, and therefore the
single place where a fabricated score could be produced. Its design concentrates on making fabrication
structurally impossible. This section walks the machinery.

```
evaluation/
├── __init__.py
├── leakage.py                     # split-by-scene, duplicate rejection, the public-test firewall,
│                                  # the hidden-data guard
├── manifests.py                   # DatasetManifest / SampleRecord / RunManifest
├── manifest_freeze.json           # the frozen dataset manifest
├── prompt_freeze.json             # the frozen prompt set
├── normalize.py                   # §63 normalisation + the aggregate prohibition
├── run_manifest.py                # populates a RunManifest from measurement, never placeholders
├── runner.py                      # manifests → adapters → metrics → normalise → report
├── benchmark_adapters/
│   ├── __init__.py                # registry; empty at import by design
│   ├── base.py                    # BenchmarkAdapter contract + BenchmarkStatus
│   ├── _common.py                 # shared machinery (hashing, provenance, corpus resolution)
│   ├── declared.py                # the four declared benchmarks (data, no load())
│   ├── levir_cd.py                # change
│   ├── vrsbench.py                # grounding
│   ├── bigearthnet_s1.py          # optical-SAR
│   ├── change_vqa.py              # change-VQA
│   ├── fixture_vqa.py             # the one synthetic adapter (must be registered by hand)
│   ├── heldout.py                 # held-out split support
│   └── scorecard.py               # the six-state project scorecard
├── metrics/
│   ├── __init__.py                # narrow re-export surface (pinned by a test)
│   ├── change.py                  # precision / recall / F1 / IoU / mIoU (pooled + macro)
│   ├── grounding.py               # IoU, Recall@IoU, greedy matching, box-convention conversion
│   ├── vqa.py                     # exact match / normalized exact match / token F1 / answer accuracy
│   └── caption.py                 # BLEU / ROUGE-L / CIDEr / BERTScore (capability-gated)
└── public_test/
    ├── __init__.py
    └── corpus.py                  # the immutable, sealable public-test corpus
```

### 3.1 The evaluation runner and its status ladder

`evaluation/runner.py` is the single place a benchmark score is produced. Its status ladder, applied in
order per benchmark (`EvaluationRunner.run_one`), is:

1. **no adapter registered** → `NOT_RUN`, with `detail.registry` recording the registered set and the reason.
2. **corpus unavailable** (`BenchmarkNotAvailableError`) → `NOT_RUN`, with `detail.corpus` carrying the
   exception's context.
3. **`load()` or `scorer()` raised** → `FAILED`, with the traceback attached.
4. **adapter claims availability but yields no samples** → `FAILED` (not a zero-sample success — "an adapter
   that is broken" must not look like "a benchmark that scored nothing").
5. **no scorer supplied** → the runner returns without inventing one.
6. **otherwise** → the adapter's own verdict on its material, via `corpus_kind()`.

The status enum is `BenchmarkStatus` in `evaluation/benchmark_adapters/base.py`:

| Status | Meaning (from `_STATUS_LEGEND`) | `is_scored` |
|---|---|---|
| `NOT_RUN` | No execution took place: no adapter, or corpus absent. No number is reported. | no |
| `FAILED` | Execution was attempted and raised. Traceback attached. Any partial value is not reported. | no |
| `DEGRADED` | Completed against a corpus the adapter does not attest as the benchmark's own (a subset or proxy). The numbers describe that corpus. | no |
| `FIXTURE` | Completed against generated material. The arithmetic is real; the subject is not the benchmark. Never quote as a benchmark score. | no |
| `REAL` | Completed against the benchmark's own corpus, as attested by the adapter. **The only status that yields a benchmark score.** | **yes** |

The critical boundary is in `EvaluationRunner.run`: **only `REAL` results are collected into
`report["real_metrics"]`**. `FIXTURE` and `DEGRADED` results go into their own keys
(`fixture_metrics`, `degraded_metrics`). A caller who wants a benchmark number has to ask for
`real_metrics`, which is empty whenever no real corpus was present.

A design change is recorded in the runner's own docstring: the classification step was originally a two-way
branch on `is_official()`, which made `DEGRADED` **unreachable** — so a real-but-partial corpus was labelled
`FIXTURE`, i.e. called synthetic. `corpus_kind()` now answers the question directly, and its base
implementation reproduces the old two-way result exactly, so no adapter changed meaning when the fix landed.

**The empty registry is deliberate.** `evaluation/benchmark_adapters/__init__.py` states: *"The registry is
empty at import."* Nothing is registered as a side effect of importing the module. `register_default_adapters()`
is the explicit act that wires up the four real-corpus adapters (`levir_cd`, `vrsbench`, `bigearthnet_s1`,
`change_vqa_test`). The synthetic fixture is deliberately **absent** from `DEFAULT_ADAPTER_FACTORIES`:
including it would mean the convenience function quietly added a benchmark whose scores describe a generator.

**Measured fact.** The adapters package docstring records (revised 2026-09-23) that three of the four
corpora are on disk and predate the package:

| Benchmark | Corpus on disk | Size |
|---|---|---|
| `levir_cd` | `data/levir/` | 2,048 test tiles |
| `vrsbench` | `training/data/vrsbench/` | 16,159 eval records |
| `bigearthnet_s1` | `data/bigearthnet_v2/reben/...` | 28,000 S1 patches |
| `change_vqa_test` | `data/cdvqa/annotations/` | 39,686 Test questions |

So *"the gap was never the data. It was the **scoring half**."* Adapter-based benchmark runs remain
`NOT RUN` in this release (see §8); the metric artifacts in §4 were produced by the specialists' own
evaluation scripts, not by the adapter runner.

### 3.2 Leakage prevention

`evaluation/leakage.py` states its rules at the top of the file:

```
1. Split by SCENE, never by sample/tile.
2. No scene may appear in more than one split.
3. Duplicate content (same sha256) is rejected.
4. Public test sets are immutable and off-limits to training code.
5. Hidden data must never influence thresholds, prompts, or routing.
```

and adds the enforcement principle: *"Every function here either returns a clean result or raises. None of
them 'warn and continue' — a leakage warning that scrolls past is indistinguishable from no check."*

**`assign_splits_by_scene(records, *, train_ratio=0.8, val_ratio=0.1, seed=42)`.** This is the function that
prevents the classic remote-sensing leakage bug — slicing a large scene into tiles and then randomly
splitting tiles, so that near-duplicate adjacent tiles leak the test set into training. Implementation
notes recorded in the source:

- Scenes are shuffled with a seeded RNG so the assignment is reproducible.
- Scene **order** is sorted first, making the shuffle independent of input order.
- A scene's records all receive the same split — *"enforced, not hoped for"*.
- Records are **emitted in sorted scene order**, not shuffled order: *"the seed controls which scene goes to
  which split; it must not control the order of the returned list, or downstream code taking `records[:N]`
  would silently depend on the seed."*
- Guards raise `LeakageError` for `train_ratio ∉ (0,1)`, `val_ratio ∉ [0,1)`, and `train_ratio + val_ratio ≥ 1`.

**`audit_manifest(manifest, *, require_all_splits=True)`** performs the full audit and raises `LeakageError`
when a hard rule is broken. It checks: no scene in more than one split; no duplicate `sha256`. Two advisory
warnings are recorded (not raised): a split set that is not exactly `{train, val, test}`; and samples whose
`scene_key` fell all the way back to `sample_id` (no `scene_id`, `geographic_hash` or `source_scene`), for
which *"scene-level isolation is vacuous"*.

**`assert_no_scene_overlap(train, test)`** is the hard assertion that two collections share no scene.

**`PublicTestFirewall(public_test_root)`** blocks training-time reads of the public test tree with two
layers: *path inspection* (any read whose resolved path is under the public test root is refused while the
firewall is armed) and *manifest inspection* (a manifest containing test-split records may not be handed to
a training loop). The firewall is **armed by default**; only the evaluation runner disarms it.

**`assert_no_hidden_access(config)`** guards the hidden-evaluation contract (plan §38). It raises unless
`evaluation.hidden_data_access is False` **and** `evaluation.official_aggregate_weights is None`. The
docstring: *"If someone flips that flag to True, evaluation stops here rather than silently tuning on hidden
material."*

### 3.3 The public-test firewall and the immutable corpus

`evaluation/public_test/corpus.py` implements plan §37 as four mechanisms:

| Plan §37 rule | Mechanism |
|---|---|
| Public test sets are immutable | `PublicTestCorpus.seal()` writes a manifest of every file's `sha256` and byte count; `verify()` recomputes and raises on **any** difference |
| Training code may not import them | a corpus is always bound to a `PublicTestFirewall` root, and `read()` refuses any path that escapes the corpus root |
| Evaluation code may read them only in evaluation mode | `PublicTestCorpus` requires an explicit `EvaluationMode` token; training code does not have one |
| No cache of test answers is permitted | there is **no writer** for gold answers anywhere in the module; `forbid_answer_cache()` makes the absence checkable |

**The `EvaluationMode` token is deliberately not a bool.** The docstring: *"`open_public_test(corpus,
mode=True)` would be far too easy to pass from training code by reflex; requiring an object of this type
means the caller had to go and find the constructor."* `evaluation_mode(reason)` requires a non-empty reason,
which travels into every read's record: *"'Why is this code reading the test set?' is the question the
firewall exists to make answerable."*

**`verify()` distinguishes four deltas** because they mean different things: `changed` (content differs —
the gold answers may no longer hold), `added` (new material appeared), `removed` (material disappeared —
results computed against it are no longer reproducible), and `resized` (content identical but byte count is
not — *"should be impossible and therefore signals a bug"*). A `verify()` that only checked hashes of files
still present would miss `removed`, *"which is the one that silently invalidates a published number."*

**Sealing an empty corpus is refused.** The docstring: *"Sealing an empty corpus would produce a seal that
verifies forever, which reads as 'the test set is intact' when the truth is 'there is no test set'. That is
the exact class of false assurance this module removes."*

**The corpus directory is currently empty** (only a `.gitkeep`). The module reports this as
`available=False`, not as an empty passing test set: *"an empty corpus and a corpus that passed the firewall
must not look the same."* `open_public_test()` raises `PublicTestSealError` when the corpus holds no material.

### 3.4 The run manifest

`evaluation/run_manifest.py` exists because a `RunManifest` was being constructed with four fields left at
their defaults (`code_revision=None`, `environment={}`, `prompt_versions={}`, `started_at=None`) — *"a
manifest in name only: it cannot answer 'which code, on which machine, with which prompts'."*

The rule the module follows is stated at the top:

> **Never fabricate. Record the absence.** Every field here is either a value that was genuinely measured,
> or an explicit statement that it was not available. There is no default that looks like data.

Concretely: an unreadable git commit becomes a `code_revision` string that says so, plus the content
snapshot digest that **is** available; a missing prompt module is reported as `"unavailable"`, not as an
empty string that reads like "no prompts were used"; an environment value that cannot be probed is recorded
as `"unavailable"`. The module's own words: *"A placeholder is worse than a gap, because a gap is visibly a
gap."*

Measured facts recorded by the module:

- `environment_block()` records Python version, implementation, platform, machine, processor, executable,
  cwd, repo root, and — when torch is importable — torch version, CUDA availability, CUDA/cuDNN versions,
  GPU count and GPU names. When torch is absent or broken, every torch key is set to
  `"unavailable (<ExceptionType>)"` rather than omitted, *"because an omitted key is indistinguishable from
  'this run needed no torch'."*
- `prompt_versions()` enumerates the prompt modules (`specialists.vqa.prompts`,
  `specialists.optical_sar.prompts`) rather than globbing them, *"so a specialist that stops exporting a
  version should show up as MISSING, not silently drop out of the manifest."* A prompt change is a behaviour
  change that no config hash captures, which is why it belongs in the manifest.
- `code_revision_identifier()` names a dirty working tree explicitly: *"a commit hash from a dirty tree does
  NOT identify the bytes that ran, and a manifest that omits that fact overstates what it proves."*
- `manifest_completeness()` is a **report, not a gate** — *"this module does not decide whether a manifest is
  good enough, because no quality threshold is defined anywhere in the plan."*
- `ended_at` is **deliberately not defaulted**: *"an unset end time must stay unset rather than becoming
  'now', which would claim the run took zero seconds."*

### 3.5 Metric normalisation and the aggregate prohibition

`evaluation/normalize.py` implements plan §63 literally and is *"deliberately small"*:

- The transform set is **minimal**: the only registered transform is the **identity** map, and only for
  metrics whose native range is already `[0, 1]` — `iou`, `miou`, `f1`, `precision`, `recall`, `accuracy`,
  `map`. Each is listed explicitly in `TRANSFORMS`.
- A metric that is **not** registered **raises** `UnregisteredMetricError`. *"It is never guessed at, never
  clamped, and never inverted."* A lower-is-better metric (e.g. an error rate) is **not** handled, because
  doing so *"would require a bound this module has no authority to choose."*
- A value outside `[0, 1]` after its transform **raises** `NormalisationRangeError`; it is never clamped
  silently.
- `official_weights: None = None` with **no default**. `aggregate()` **raises**
  `OfficialWeightsUnavailableError` while it is `None`.

The runner makes the prohibition **exercised, not merely documented**. `aggregate_or_explain()` calls
`aggregate()` and records the refusal:

```
aggregate_score = {
  "available": False, "score": None, "weights": None,
  "error": "OfficialWeightsUnavailableError",
  "reason": "...",
  "note": "No official aggregate exists. Plan section 63 prohibits inventing an aggregate formula,
           so no single score is reported. Per-metric normalised values remain available above."
}
```

The docstring explains why the call is made rather than short-circuited on the `None` check: *"if someone
later assigns weights without implementing a formula, this call still raises, and the report says so. A bare
`if official_weights is None` would have hidden that."*

`normalise_metrics()` also records failures instead of dropping them: a metric that fails normalisation is
**absent** from `normalized` and **present** in the `normalisation` records with its error, *"so a consumer
cannot mistake 'could not be normalised' for 'was not measured'."* Non-numeric values raise rather than being
coerced: *"a metric whose value is a string is a bug in the adapter, and `float('abc')` failing deep inside a
report writer is a worse way to find out."*

### 3.6 The metric implementations

All metric code is deliberately boring: boolean-array arithmetic, every division guarded, tested against
hand-computed values. The grounding module states the principle: *"Reference values are hand-computed in the
tests rather than taken from a library, because a metric that agrees only with itself proves nothing."*

**`evaluation/metrics/change.py` — change detection.** Pure NumPy; no model, no torch, no dataset. Two
aggregation conventions are computed and labelled:

- **pooled**: sum the confusion counts across every tile, then compute one precision/recall/F1/IoU.
- **macro**: compute per tile, then average.

They differ when tile sizes or change fractions differ and can disagree by several points, so *"rather than
pick one and hope, both are computed and labelled."* Empty-change tiles (all background) are **excluded from
the macro average** — otherwise they contribute a recall of `0.0` *"for a property of the dataset rather
than of the model"* — and are counted separately in `n_images_with_change`. Binarisation uses `>=` rather
than `>`: *"a pixel at exactly the threshold is a change. The asymmetry matters for reproducibility, so it is
pinned by a test."* `miou` averages the change-class IoU and the background IoU.

**`evaluation/metrics/grounding.py` — grounding.** Pure NumPy. Coordinate convention is `[x1, y1, x2, y2]`
in the same frame for prediction and target; internally SatQuery uses normalised 0–1
(`CoordinateSystem.NORMALIZED_0_1`), and VRSBench's 0–100 annotations are converted at the adapter boundary,
**never here**: *"Mixing frames here would produce a plausible-looking IoU that means nothing."* Matching is
**greedy, highest-IoU-first**, not Hungarian — *"greedy is what VRSBench-style referring evaluation does and
it is order-independent because the sort is on the IoU value, not on the prediction order."* Ties break on
`(pred_index, target_index)` so the result is deterministic. A pair where either box has zero area yields
`0.0`, not NaN: *"A degenerate prediction is a wrong prediction, not an undefined one."* Recall is
**macro-averaged over images** (mean of per-image recall), not pooled: *"macro is the convention for
referring evaluation because one crowded image should not dominate the score."* Images with no targets are
excluded from the recall average but still counted in `n_images`. `benchmark_to_normalized(box, scale=100.0)`
is a **named function** rather than an inline division, because *"treating those numbers as pixels or as 0-1
is a silent, catastrophic error."*

**`evaluation/metrics/vqa.py` — VQA / Change-VQA text metrics.** Pure stdlib (`re`, `collections`). The
normalisation rule set is the **VQA-v2** convention, published as an inspectable `NORMALIZATION_RULES` tuple:

```
R1 lowercase the text
R2 fix common contractions      ("dont" -> "don't")
R3 map number words to digits   ("two"  -> "2")
R4 remove articles              ("a", "an", "the")
R5 strip punctuation            (each punctuation char becomes a space; a period or comma
                                 BETWEEN TWO DIGITS is preserved, so "3.14" and "1,000" survive)
R6 collapse whitespace and strip
```

`exact_match` is **raw** string equality (case- and punctuation-sensitive); `normalized_exact_match` applies
`normalize_answer` to both sides first. Both return a float in `{0.0, 1.0}` so they compose with averaging
without a cast. `token_f1` uses the **multiset** intersection, so `"cat cat"` against `"cat"` scores `2/3`,
not `1.0` — *"a set intersection would ignore repetition and inflate the score."* `answer_accuracy` is the
fraction of normalised-exact matches and raises if `preds` and `golds` differ in length, *"because a
mismatch means the caller has mis-aligned questions and answers, and scoring the overlap silently would hide
that."* Degenerate cases are documented and pinned: both sides empty → `1.0`; exactly one side empty → `0.0`.

The module records a judgement call plainly: the rule set is *"a CONVENTION CHOICE. The plan does not
mandate it"*, implemented because *"some documented, deterministic rule set is required for 'normalized
exact match' to mean anything, and VQA-v2 is the standard one."* It says the set *"should be ratified — or
replaced — before these numbers are treated as a benchmark figure."*

**`evaluation/metrics/caption.py` — caption metrics.** Implements the plan §2.2 caption set (BLEU, ROUGE-L,
CIDEr, BERTScore) over pinned third-party packages (`sacrebleu==2.4.3`, `rouge_score==0.1.2`,
`pycocoevalcap==1.2`, `bert-score==0.3.13`). `score_captions()` **raises** for a requested-but-unavailable
metric rather than falling back, *"because a silent fallback is how 'BLEU' ends up meaning 'token F1'."*
Measured capability on the authoring machine (2026-09-23): BLEU available (corpus BLEU 100.0 on identical
input, 41.14 on a near-match); ROUGE-L available (1.0 identical, 0.833 near-match); CIDEr **unavailable**
(no `java` on PATH for the PTB tokenizer); BERTScore **unavailable** (the local HuggingFace cache holds a
corrupt `models--roberta-large` entry whose `config.json` is 0 bytes, which `transformers` prefers over a
download). **No caption score is published anywhere**: *"Producing one requires a corpus of predictions and
references, which is a separate act."*

**The `evaluation.metrics` package surface is deliberately narrow and pinned.** `change` and `grounding` are
imported as submodules, not re-exported flat, because both publish the public names `score_one` and
`score_dataset` — *"a flat re-export would silently shadow one module with the other."* The pin is a test:
`tests/unit/test_vqa_metrics.py::test_package_reexports_the_vqa_public_names` asserts
`set(metrics.__all__) == set(vqa.__all__)` exactly.

### 3.7 The project scorecard

`evaluation/benchmark_adapters/scorecard.py` defines a six-state vocabulary so a reader meets the strongest
evidence first and the two kinds of absence last:

| State | Meaning (from `STATE_MEANINGS`) | Score-bearing? |
|---|---|---|
| `REAL` | ran against the benchmark's own corpus; **the only state that may back a published score** | **yes** |
| `DEGRADED` | ran against a real but partial or proxy corpus; the number describes that material | no |
| `FIXTURE` | ran against self-generated material; proves the machinery, never a benchmark score | no |
| `FAILED` | execution was attempted and raised | no |
| `NOT_RUN` | not attempted: no adapter registered, or deliberately skipped | no |
| `RESOURCE_BLOCKED` | not attemptable here: the immutable evaluation corpus does not exist | no |

`SCORE_BEARING_STATES = frozenset({"REAL"})` — *"Which states may carry a number a reader is allowed to
quote. Exactly one."*

---

## 4. Per-task evaluation protocols

Each subsection states the split, the sample count, the thresholds, the protocol, the decode variants where
they exist, the artifact, and the **exact key paths** at which each quoted number lives.

### 4.1 Change detection — `VERIFIED`

**Artifact.** `artifacts/change/eval_test/eval_result.json`

| Property | Value | Key path |
|---|---|---|
| Split | LEVIR-CD-256 test | `n` |
| n | 2,048 | `n` |
| n with change | 935 | `n_images_with_change` |
| Threshold | 0.50 (frozen in config) | `metrics.threshold` |
| Device | `cuda` | `device` |
| Config hash | `78f1e3700da15aa1` | `config_hash` |
| Checkpoint config hash | `78f1e3700da15aa1`, checked | `checkpoint_config_hash`, `checkpoint_config_hash_checked` |
| Config drift | `false` | `config_drift` |
| Wall time | 55.359 s | `metrics.seconds` |

**Pooled metrics** (sum counts across tiles, then compute) — all under `metrics.pooled.*`:

| Metric | Value | Key path |
|---|---|---|
| IoU (change class) | **0.8122** | `metrics.pooled.iou` |
| mIoU (change + background) | 0.9007 | `metrics.pooled.miou` |
| F1 | **0.8964** | `metrics.pooled.f1` |
| Precision | 0.9195 | `metrics.pooled.precision` |
| Recall | 0.8745 | `metrics.pooled.recall` |
| TP / FP / FN / TN | 5,978,997 / 523,658 / 858,407 / 126,856,666 | `metrics.pooled.tp` … `.tn` |
| n_pixels | 134,217,728 | `metrics.pooled.n_pixels` |

**Macro metrics** (per tile, then average over tiles **with change**) — under `metrics.macro.*`:

| Metric | Value | Key path |
|---|---|---|
| IoU | 0.7180 | `metrics.macro.iou` |
| **mIoU** | **0.8457** | `metrics.macro.miou` |
| F1 | 0.7962 | `metrics.macro.f1` |
| Precision | 0.8506 | `metrics.macro.precision` |
| Recall | 0.7757 | `metrics.macro.recall` |

The macro block carries the same pooled confusion counts (`tp`/`fp`/`fn`/`tn`), because `ChangeReport.macro`
is a `ChangeScores` whose `counts` field is the pooled total — see `evaluation/metrics/change.py`,
`score_dataset`.

**Why both pooled and macro.** The test split is only ≈ 5 % changed pixels — `metrics.mean_change_fraction`
is **0.0509**. The per-image change-fraction distribution (`change_fraction_quantiles`) is
`min 0.0`, `p50 0.0`, `p90 0.197205`, `max 0.684937`. Pooled IoU (0.8122) and macro IoU (0.8457) answer
different questions about that imbalance; the macro figure is the `miou` convention (mean of change-class and
background IoU, per tile, averaged over tiles with change).

**Full confusion counts are shipped** so any metric can be recomputed without re-running the model — see
`evaluation/metrics/change.py::scores_from_counts`.

**Model identity.** The artifact embeds the checkpoint's own config (`checkpoint_embedded_config`):
`encoder: resnet18`, `encoder_channels: [64, 128, 256, 512]`, `frozen_encoder: false`,
`pretrained_used: true`, `sa_mode: PAM`, `width: 128`, `attention_budget_bytes: 268435456`. The evaluation
environment is recorded under `environment`: Python `3.12.13`, torch `2.10.0+cu128`, `cuda_available: true`,
platform `Linux-6.12.90+-x86_64-with-glibc2.35`.

**Released artifact.** `change/head.pt` — 63,231,009 bytes, sha256
`c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa`.

**Why this is the only `VERIFIED` headline.** It is the only task whose artifact is a full-split evaluation
against an immutable public split, at a frozen threshold, with a checked config hash and no drift.

### 4.2 Grounding — `MEASURED`, two protocols × three decode variants

**Artifacts.**

- canonical: `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`
- matched6: `artifacts/grounding/remoteclip_grounding_v001/eval_result_matched6.json`

**Shared protocol.** Split: VRSBench eval, `n_eval_records` **16,159**. Resolution `224` (frozen;
`resolution_frozen: true` in config). Token grid `grid: 7` (7 × 7 = 49 cells). Head decode
(`head_decode`): `nms_iou: 0.5`, `score_threshold: 0.4`, `top_k: 20` in the canonical run and `top_k: 6` in
the matched6 run — the protocol difference. `frozen_config_evaluation: true`, `limited_run: false`,
`config_drift: false`. Environment: CPU, Python `3.11.16`, torch `2.14.0+cpu`,
`Windows-10-10.0.26200-SP0`.

**Box convention.** VRSBench stores boxes normalised to **0–100**; SatQuery stores **0–1**. The conversion is
a declared config value, `grounding.benchmark_box_scale: 100.0`, and a named function
`benchmark_to_normalized()` — *"so the conversion cannot be applied twice or forgotten."*

**Results — canonical** (`results.*`):

| Decode | mean_best_IoU | Key path | recall@0.10 | recall@0.25 | recall@0.50 | latency ms/img |
|---|---|---|---|---|---|---|
| `head_threshold` (shipped) | **0.2838** | `results.head_threshold.mean_best_iou` | 0.6882 | 0.5047 | **0.2198** | 2.205 |
| `head_argmax` | **0.1215** | `results.head_argmax.mean_best_iou` | 0.3183 | 0.2088 | 0.0795 | 0.655 |
| `zero_shot_matched` (baseline) | **0.0972** | `results.zero_shot_matched.mean_best_iou` | 0.3298 | 0.1188 | 0.0234 | — |

The `recall` sub-dict is keyed by the threshold **as a string** — `"0.10"`, `"0.25"`, `"0.50"` — which is why
the verification tool's dotted-path resolver prefers the **longest** matching key at each step (§7.6). The
recall@0.5 key path is `results.head_threshold.recall.0.50`.

**Results — matched6** (`results.*`):

| Decode | mean_best_IoU | Key path | recall@0.10 | recall@0.25 | recall@0.50 |
|---|---|---|---|---|---|
| `head_threshold` | **0.2566** | `results.head_threshold.mean_best_iou` | 0.6315 | 0.4545 | **0.1938** |
| `head_argmax` | 0.1215 | `results.head_argmax.mean_best_iou` | 0.3183 | 0.2088 | 0.0795 |
| `zero_shot_matched` | 0.0972 | `results.zero_shot_matched.mean_best_iou` | 0.3298 | 0.1188 | 0.0234 |

**Reading.** The head clears the zero-shot baseline, but only the threshold decode is meaningfully above it;
the argmax decode (0.1215) is barely better than zero-shot (0.0972). The absolute level is modest either way:
**grounding is useful, not solved.** The zero-shot baseline reference is also recorded inside both artifacts
as `phase7_reference` (`mean_best_iou: 0.0972`, `recall_at_0.50: 0.0234`, source
`docs/PHASE7_RESOLUTION_DECISION.md`).

**Both protocols are also recorded in a paired analysis** (see §5): the pre-registered resolution experiment
scored **the same 16,159 samples** at both resolutions, which makes the paired test the stronger statistic
because it removes between-object variance.

**Head identity.** `grounding/head.pt` — 12,639,041 bytes, sha256
`93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb`, over the frozen RemoteCLIP ViT-B/32
encoder (revision `bf1d8a3ccf2d`). Per-cell feature = `concat([patch, text, patch·text, global_pool])` =
`4 × 512 = 2048` (finding P7-1: the transformer width is 768, but `visual.proj` maps to a projected dim of
**512**). Cells are assigned by ground-truth box centre (`decode: cell_relative`). The objectness BCE is
weighted **20×** (`positive_confidence_weight: 20.0`) because only ~1 of 49 cells is positive; unweighted,
*"the optimum is 'no object' everywhere; this weight is what stops that collapse."*

**Two additional pre-registered smoke files** exist for the resolution experiment and are **not** evidence:
`grounding/per_sample_224_smoke.jsonl`, `grounding/per_sample_448_smoke.jsonl`,
`grounding/resolution_experiment_smoke.json`. At n = 12, n = 40 and n = 6 the smoke runs reported
`Recall@0.5 = 0.0000` at **both** resolutions and emitted a degeneracy warning; at full scale the metric is
non-zero (0.0234 / 0.0212), so the note correctly did not fire. The sub-floor runs were never treated as
evidence.

### 4.3 Optical-SAR fusion — `MEASURED`, ruling `OPEN`

**Artifact.** `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`

| Property | Value | Key path |
|---|---|---|
| Tool | `fusion_115_metric` | `tool` |
| Metric | `pre_registered_11.5` | `metric` |
| Definition | fusion-head accuracy and macro-F1 over the 19-class label space on the held-out split | `definition` |
| Split | `test` (held out; never used for selection) | `split` |
| n scored | **4,000** | `n_scored` |
| **Accuracy** | **0.931** | `accuracy` |
| **macro-F1** | **0.434161** | `macro_f1` |
| Loss | 0.254592 | `loss` |
| num_classes | 19 | `num_classes` |
| macro-F1 denominator | all 19 classes (absent classes contribute 0.0) | `macro_f1_denominator` |
| classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` |
| classes absent | `[1,11,14,15,16]` | `classes_absent` |
| Deciding statistic? | **`false`** | `is_deciding_statistic` |
| Head sha256 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` | `head_sha256` |
| Head bytes | 14,427,457 | `head_bytes` |
| Cache | `artifacts/optical_sar/fusion_features/test.npz`, arm A | `cache_path`, `cache_arm` |

**The 19-term per-class F1 vector** (`_per_class_f1`) is shipped so the macro-F1 is auditable:

```
[1.0, 0.0, 0.4280155642023346, 0.5714285714285714, 0.8871595330739299, 0.0, 0.0, 0.8,
 0.928652321630804, 0.9397590361445783, 0.18181818181818182, 0.0, 0.2, 0.4375, 0.0,
 0.0, 0.0, 0.8767123287671232, 0.9980101702409905]
```

**The artifact's own advisory**, verbatim:

> *"This tool reports ONE head's held-out accuracy and macro-F1. It selects no head, ranks nothing and
> compares no arms. Whether this constitutes a Phase 12 pass is the owner's ruling."*

**Two readings, both required.** `docs/PHASE12_115_METRIC_COMPUTED.md` establishes, with four independent
reproductions, that the numbers are correct — and then establishes what they do **not** mean:

- **Reproduced outside the tool** (direct call to `evaluate_fusion_head`): `n=4000, accuracy=0.931,
  macro_f1=0.43416082670034284`.
- **Reproduced without the trainer's helpers at all** — rebuilding the input tensor by hand
  (`concat([optical_gap, sar_gap, joint_gap], axis=1)` then `concat([…, optical_mask, sar_mask], axis=1)`,
  widths `(4000,768)×3 + (4000,12) + (4000,2)` = **2318**) — gives **accuracy 0.931** again. This checks the
  feature *ordering*, which a helper-based re-run would not.
- **Reproduced by a third-party implementation** — `sklearn.metrics.accuracy_score = 0.931`,
  `f1_score(average="macro", labels=range(19), zero_division=0) = 0.43416082670034284`.
- **The accuracy is not a constant predictor.** The test split's majority class holds **2,264 / 4,000 =
  0.566**; the head scores **0.931**.

**The per-class spread is real, in both directions.** Per-class recall on the held-out split (from
`docs/PHASE12_115_METRIC_COMPUTED.md` §3.4): class 18 (n=2264) **0.997**; class 9 (n=246) **0.951**;
class 17 (n=34) **0.941**; class 8 (n=453) **0.905**; class 4 (n=846) 0.809; class 2 (n=63) 0.873;
class 0 (n=13) 1.000; class 13 (n=18) 0.778; class 10 (n=4) 0.500; class 3 (n=39) 0.410; class 12 (n=9)
0.111; class 5 (n=4) **0.000**; class 6 (n=1) **0.000**; class 7 (n=6) 1.000.

Measured test-set label distribution:

```
[13, 0, 63, 39, 846, 4, 1, 6, 453, 246, 4, 0, 9, 18, 0, 0, 0, 34, 2264]
```

- **14 of 19 classes present; five have zero samples.**
- Top-to-bottom ratio **2,264 : 1** (class 18 vs class 6).
- The median of the 14 present classes is **0.6857** against an accuracy of **0.931** — *"the signature of
  prediction dominated by frequent classes."*

**The denominator trap.** `0.434161` and `0.931000` sit next to each other, which invites the reading "the
head is accurate on common classes and catastrophic elsewhere." Two different numbers come from the same
per-class scores:

| Averaged over | Value |
|---|---|
| **all 19 slots** — the pre-registered definition | **0.434161** |
| the 14 present classes only | 0.589218 |

*A reader who computes the second and compares it to the recorded scalar will conclude the recorded figure is
wrong. It is not — it is the 19-slot mean, and the trainer's `_macro_f1` divides by `num_classes` by
construction.* A regression test (`test_macro_f1_averages_over_all_slots_not_present_ones`) pins the
arithmetic.

**The caveat that governs how this number may be used.** The cache metadata records
`label_policy = require_single_label`, `n_skipped_by_policy = 0`. reBEN v2.0 is a **multi-label** corpus while
the frozen head is a **single-label 19-class softmax** trained with `cross_entropy`. The extraction therefore
restricted to single-label patches, which *"preserves the frozen architecture and the 19-class space exactly
[but] changes the **evaluation population**."* Measured context: single-label patches are **17.57 %** of the
corpus (**96,537 of 549,488**), and under this policy the rarest class survives as **1 patch**, a
**59,204 : 1** imbalance.

So this metric **may not** be presented as: a multi-label BigEarthNet/reBEN result; comparable to published
BigEarthNet numbers (almost all multi-label); or a statement about all 19 classes (5 have no test samples).
It **may** be presented as: the pre-registered 11.5 metric, as computed under the single-label extraction
policy the frozen architecture requires, on the held-out split. The `label_policy` choice remains an **open
owner decision**.

**Reproduction and exit codes.** `scripts/eval_fusion_115.py` exits `0` when a metric was computed and `2`
when it refused (missing artifact, empty split, wrong `config_hash`, or a head/cache dimension disagreement —
*"a category error, not a result"*). The tool never reads the validation split when scoring `test`; a test
pins that separation. The trainer's `pre_registered_metric_computed = false` flag is **correct about the
trainer** and is deliberately not "fixed": the trainer fits on `train`/`val` and never opens the test split,
so the held-out split cannot be contaminated by the search over 10 runs, 2 arms and 5 seeds.

**A guard that is load-bearing.** Removing the cache `config_hash` guard makes the tool print
`ACCURACY: 0.000000` for a cache from a different experiment — *"a silent wrong number that looks like a
result."* The guard is pinned by `tests/unit/test_eval_fusion_115.py` (8 tests, 0 deleted, 0 weakened at the
time of that document; §8 of the same document records the module at 10 passed in the suite run).

### 4.4 Change-VQA — `MEASURED`, ruling `OPEN`

**Artifact.** `artifacts/change_vqa/run/PROMOTION.json`

**Two test sets — never collapsed:**

| Split | accuracy | macro-F1 | Key path |
|---|---|---|---|
| `test` | **0.697626367** | **0.378373275** | `verification.test_accuracy`, `verification.test_macro_f1` |
| `test2` | **0.651469262** | **0.372308516** | `verification.test2_accuracy`, `verification.test2_macro_f1` |

The `test` numbers are the higher pair. Both are reported; quoting only `test` would overstate the result.

**Baselines recorded in the same artifact** (`verification`): `global_majority_baseline_test: 0.311546`,
`global_majority_baseline_test2: 0.178728`. Sample counts: `n_scored_test: 39686`,
`n_scored_test2: 31036`. `mask_gain: 0.0`.

**The artifact's own ruling**, verbatim from `verification.metric_ruling`:

> *"OPEN — the plan leaves the accuracy/macro-F1 interpretation owner-gated. No official aggregate metric is
> asserted here."*

**Selection and training provenance** (`identity`): `epoch_selected: 8`, `selected_on: "Val answer accuracy"`,
`val_answer_accuracy: 0.700018`, `stop_reason: "early_stopping"`, `seed: 42`, `config_hash:
78f1e3700da15aa1`, `dataset_id: cdvqa`, `feature_spec: change_feat_v1`,
`change_cache_spec: c801326f85a185f8`, `text_cache_spec: d2801ea1a314354a`,
`preprocessing_version: change_vqa_preproc_v1`.

**Promotion gate evidence** (`artifact` + `source` + `verification`): the head is `artifacts/change_vqa/run/head.pt`,
sha256 `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a`, 5,822,809 bytes, architecture
`change_vqa_head_v1`, **1,453,912 parameters**, `satquery_trained: true`, `eval_mode: true`,
**`non_finite_tensors: 0`**, `weights_modified: false`, `byte_identical_to_source: true`. The hash **agrees
across** `model_metadata.json`, `run_record.json` and `hashes.json (run.checkpoint_sha256)`. The verification
block records `checks_passed: 93`, `checks_failed: 0`, `checks_unverified: 0`.

**Frozen dependency** (`frozen_dependency`): the head's change features are backed by the frozen STANet change
detector `artifacts/change/levir_change_v001/head.pt`, sha256
`c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa`, 63,231,009 bytes,
`verified_byte_exact_vs_local: true`.

**Serving wiring** (`serving_wiring`): the expected path in code is `artifacts/change_vqa/run/head.pt`,
declared in `app/serving.py:76-78` (`CHANGE_VQA_HEAD`) and also defaulted by `scripts/train_change_vqa.py:54`
(`DEFAULT_OUT`) and `scripts/evaluate_change_vqa.py:74` (`DEFAULT_CHECKPOINT`). `code_change_required: false`.

**What promotion does and does not do** (`state`): before promotion the artifact was
`TRAINED_UNVERIFIED`; after promotion it is `PROMOTED`. The artifact's own note: *"Promotion records
provenance and wires the serving path. It does not itself confer VERIFIED status; that is the maintainer's
ruling."*

**The head was trained outside this repository** (Kaggle GPU). The head trains on **cached change + text
features**, not raw imagery. The raw CDVQA loader loads examples but has no training loop of its own, and the
two paths are not conflated. See §8 for what that means for reproduction.

### 4.5 VLM adapter — `MEASURED`, `ACCEPTANCE-REJECTED`

**Artifact.** `artifacts/vlm/phase6_closure.json` (`status: CLOSED`)

**Headline** (`headline`), verbatim: *"Phase 6 is closed. The Run 1 LoRA adapter is promoted to the
production VLM adapter: USABLE and VERIFIED, but ACCEPTANCE-REJECTED."*

**Usable metrics** (`why_usable_verified.adapted_test`) on a **frozen 1,000-question subset**:

| Metric | Value | Key path |
|---|---|---|
| exact_match | **0.963** | `why_usable_verified.adapted_test.exact_match` |
| F1 | **0.96432** | `why_usable_verified.adapted_test.f1` |
| precision | 0.963391 | `why_usable_verified.adapted_test.precision` |
| recall | 0.965251 | `why_usable_verified.adapted_test.recall` |
| n | 1,000 (`n_available: 1000`) | `why_usable_verified.adapted_test.n` |
| confusion | tp 500, fp 19, fn 18, tn 463 | `why_usable_verified.adapted_test.confusion` |

Aggregate test delta: `why_usable_verified.aggregate_test_delta_pp = 49.5` (+49.5 pp over the unadapted
baseline). `why_usable_verified.gate_d_reproduced_run1_adapted_control_exactly: true`.

**Why acceptance was rejected** (`why_acceptance_rejected`), with `decision_split: "test"`:

- `test baseline = 46.80 pp, adapted = 96.30 pp, delta = +49.50 pp; required (V1) >= +5.00 pp` — **V1 passed**.
- `V2 (v002) failed: 1 class(es) lost >= 4 questions with z >= 1.96 on test` — **V2 failed**.

The failing class is **Mixed forest**: `adapted_pp: 87.8788`, `baseline_pp: 100.0`, `drop_pp: 12.1212`,
`lost_questions: 4`, `n_questions: 33`, `z: 2.1335`. Class-level summary: `n_classes_failed: 1`,
`n_classes_held: 5`, `n_classes_improved: 11`, `n_classes_total: 19`.

The artifact records the residual risk explicitly (`residual_risk`): *"The verdict rests on 4 questions in
one class of 33 … With no n >= N floor in V2, a 33-question class can flip the verdict of a run whose
aggregate endpoint improved by 49.5 pp. Reported, not resolved."* It also records why this is not a
split artefact (`why_not_a_split_artefact`): the same class degraded on the val split in run 1
(drop 6.4516 pp, n=31), *"so this is a property of the adapter, not an accident of one subset."*

**The pre-registered thresholds** (`why_acceptance_rejected.thresholds_used`): `accept_min_delta_pp: 5.0`,
`accept_min_delta_pp_ceiling: 2.0`, `ceiling_baseline_pp: 95.0`, `max_class_drop_pp: 1.0` (v001),
`min_class_questions: 20`, `test_val_disagreement_pp: 10.0`, `primary_endpoint: "presence-question normalised
exact-match accuracy"`. The v002 criterion (`v2_criterion`) uses `min_class_drop_questions: 4` and
`class_drop_z: 1.96`, with `se_formula: "sqrt((p_b*(1-p_b) + p_a*(1-p_a)) / n)"` and
`z_formula: "(baseline - adapted) / se"`.

**A declared amendment, recorded honestly.** v002 amends v001's V2 only. The artifact records
`declared_after_first_run: true`, `declared_before_training: false` — i.e. the amendment was declared after
the first run and **not** before training — with the reason that v001's flat 1.0 pp threshold *"fires on 0.20
of a question and sits ~8x below the per-class standard error of the delta (2.6-10.6 pp on run 1)."* The
amendment states its 1.96 SE bar is below the contract's ~3 SE figure, so it is *"not a numerically stricter
bar."* V1/V1'/V3/V4 are unchanged from v001.

**Preserved, immutable verdicts** (`preserved_records`): three records, all `immutable: true` —

| Record | Split | Status | Role |
|---|---|---|---|
| `v001_val_rejected` | val | `REJECTED` | the ORIGINAL pre-registered verdict, preserved verbatim |
| `v002_independent_test_rejected` | test | `REJECTED` | the CONTRACT-FACING verdict |
| `v002_val_accepted_NOT_final` | val | `ACCEPTED` | recorded for completeness ONLY — **not** final acceptance |

The third record is explicitly **not** final: it *"decides on the same val subset that motivated v002, which
§7.4 condition 3 forbids."*

**BERTScore is unavailable and recorded as unavailable**, not as a zero or a substitute:
`thresholds_used.bertscore = {"available": false, "reason": "roberta-large is not in the local HuggingFace
cache; BERTScore cannot be computed offline..."}`. BLEU/ROUGE/BERTScore are excluded for the stated reason
that *"target answers are one token; BLEU/ROUGE are meaningless at that length and BERTScore is unavailable
offline."*

**The forward rule** (`forward_rule`) — *"Any future improved adapter MUST be a new experiment/version. It
MUST NOT rewrite, amend, or supersede Run 1's records."* `run1_is_frozen: true`. New work requires a new
artifact, a new run manifest, a new experiment/version identifier, the predeclared acceptance rule applied
as-is (or a new rule version declared before the run it judges), and its own independent test-split
adjudication.

**Production adapter shape** (`production_adapter`): base `HuggingFaceTB/SmolVLM-500M-Instruct` at revision
`a7da5b986cb5`; `use_status: USABLE_VERIFIED`; `acceptance_status: REJECTED`; `lora_rank: 16`,
`lora_alpha: 32`, `lora_target_module_count: 224`, targeting `q_proj`, `k_proj`, `v_proj`, `o_proj`,
`gate_proj`, `up_proj`, `down_proj`; `trainable_params: 8683520`; `trainable_fraction: 0.01682312`;
`precision_recorded: fp16`. `vision_tower_untouched` records that `trainable_subtrees` is exactly
`{'model.text_model': 8683520}`, so the vision model (86,433,024) and connector (11,796,480) are in
`frozen_params` — *"The contract's vision-tower hazard did not occur."*

**How it is enabled** (`how_enabled`): by environment variable, not a code change —
`SATQUERY_VLM_ADAPTER`, resolved in `specialists/vqa/model.py` (`ADAPTER_ENV_VAR` at line 41; resolution
order explicit arg → env var → none at line 254; attached via `PeftModel.from_pretrained` at line 302).
**The deployed caption/VQA path therefore uses the unadapted SmolVLM.**

**Two known traps recorded in the artifact** (`known_traps`), both about `adapter_sha256` naming two different
values: `training/vlm/artifact.py` computes a **TREE HASH** over the `{relpath: sha256}` weight map
(`5c6b8631…`), while `specialists/vqa/model.py::_adapter_sha256` computes the **FILE** sha256 of
`adapter_model.safetensors` (`07c76a75…`). *"Recomputing one and comparing it to the other yields a false
'artifact was altered' conclusion."* And: *"The promoted adapter is NOT checkpoint-2000. The three weight
files have three different digests: top-level `07c76a75…`, checkpoint-1500 `7273588e…`, checkpoint-2000
`bf249943…` So 'just use the last checkpoint' is not equivalent to this artifact."*

**Verification provenance** (`artifact_verification`): verdict `verified`, `loadable_via_production_path:
true`, `tree_hash_matches: true`, `weights_file_sha256_matches: true`, manifest check clean (14/14 files),
`trainable_params` measured two independent ways (`loaded_minus_frozen_equals_measured: true`). Three
informational findings are recorded (`F2_adapter_sha256_name_collision`, `F3_source_zip_scope`,
`F4_promoted_adapter_is_not_a_checkpoint`). The record was amended on review to add the F4 provenance
finding, *"No verification result changed: verdict remains 'verified'."*

**What closure does not claim** (`what_closure_does_not_claim`): that Run 1 was accepted; that the Mixed
forest regression is resolved; that a new adapter exists or is planned; that the v002 rule or verdict was
altered.

### 4.6 Router — `MEASURED`, `TEST NOT RUN`

**Artifact.** `artifacts/router/threshold_sweep_val.json`

| Property | Value | Key path |
|---|---|---|
| Split scored | `val` | `split` |
| n val | **86** | `n_val` |
| n val scored | 86 | `n_val_examples_scored` |
| n test scored | **0** | `n_test_examples_scored` |
| Test split touched? | **`false`** | `test_split_touched` |
| **Overall ungated accuracy** | **0.965116** | `overall_ungated_accuracy` |
| Corpus limited | `true` | `corpus_limited` |
| Corpus total / groups | 576 / 54 | `corpus_total`, `corpus_groups` |
| Split sizes | train 410 / val 86 / test 80 | `split_sizes` |
| Shipped threshold | 0.7 | `shipped_threshold` |
| Val min per-class support | 8 (caption) | `val_min_support` |
| Hard negatives in val | 0 | `hard_negatives_in_val` |
| Plan min val queries | 500 | `plan_min_val_queries` |
| Plan min hard negatives | 100 | `plan_min_hard_negatives` |

**Val task counts** (`val_task_counts`): caption 8, change 20, grounding 14, optical_sar 10, unsupported 19,
vqa 15.

**The artifact's own note**, verbatim:

> *"corpus-limited: val n=86 vs plan >=500. This is NOT a calibration -- the corpus is synthetic and too
> small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to
> TEST by design). Selecting a threshold here yields a justified default, not a calibrated value. The corpus
> was NOT padded with generated queries. Backlog P1-9's 'n=80' is the TEST split; the sweep target is val
> n=86. The test split was NOT touched."*

**The threshold sweep** (`rows`, 50 thresholds from 0.50 to 0.99) reports `coverage`,
`covered_task_accuracy`, `fallback_rate` and `n_covered` at each threshold. The shipped row
(`shipped_row`) is threshold 0.70: coverage 0.848837, covered_task_accuracy 0.972603, fallback_rate 0.151163,
n_covered 73. The row selected by the sweep's own criterion (`select_by: "covered_accuracy"`) is
threshold **0.76**: coverage 0.790698, covered_task_accuracy **1.0**, fallback_rate 0.209302, n_covered 68.
The delta against the shipped threshold (`delta_vs_shipped`) is `coverage: -0.0581`,
`covered_task_accuracy: 0.0274` — i.e. the sweep's selection trades 5.8 pp of coverage for 2.7 pp of covered
accuracy. **The shipped threshold remains 0.70.**

**Adapter identity** (`adapter_encoder`, `adapter_config_hash`): frozen
`sentence-transformers/all-MiniLM-L6-v2` at revision `1110a243fdf4`, 22,713,216 parameters, `max_length: 128`;
adapter config hash `615478910dc266bf`.

**Why the number is labelled "ungated".** The plan's §59 engineering acceptance target is
`task accuracy ≥ 95 %` on 500 validation queries / 100 hard negatives / 50 unsupported. The recorded corpus is
86 val examples with **0** hard negatives. A router can reach a high accuracy on easy queries while failing
exactly on the hard-negative families the plan calls out (§5 "Hard negatives … This is important"). The
number is therefore reported as **ungated**, on a **corpus-limited** split, and the **test split is
`NOT RUN`**.

**Released artifact.** `router/adapter.pt` — 211,961 bytes, sha256
`8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c`, architecture
*"task/modality adapter over frozen MiniLM embeddings (~50,822 params)"*. The config records the frozen
encoder's training setup (`router.training`): 60 epochs, batch size 64, learning rate 0.001,
weight decay 0.01, task loss weight 1.0, modality loss weight 0.3, binary loss weight 0.5,
`val_ratio: 0.15`, `hard_negatives_to_test: true` — splits are by **group** (template / hard-negative
family), never by example, so *"hard-negative families are placed in test so their accuracy measures
generalisation rather than memorisation"* (finding F4-3).

### 4.7 Calibration — `MEASURED`, not an improvement

**Artifact.** `artifacts/calibration_v001.json`

| Property | Value | Key path |
|---|---|---|
| Fitted on | `Val` | `provenance.fitted_on` |
| n samples | 16,441 | `provenance.n_samples` |
| Temperature | **0.9772731820958189** | `temperature_scaling.temperature` |
| **ECE before** | **0.013755** | `metrics.ece_before` |
| **ECE after** | **0.014929** | `metrics.ece_after` |
| **ece_improvement** | **−0.001174** | `metrics.ece_improvement` |
| NLL before / after | 0.689741 / 0.689631 | `metrics.nll_before`, `metrics.nll_after` |
| nll_improvement | 0.00011 | `metrics.nll_improvement` |
| Bins / classes | 15 / 19 | `metrics.n_bins`, `metrics.n_classes` |
| Held-out splits excluded | `[Test, Test2]` | `provenance.held_out_splits_excluded` |
| Objective | `mean_negative_log_likelihood` | `provenance.objective` |
| Optimizer | `golden_section_on_log_temperature`, 200 iterations | `provenance.optimizer`, `fit_diagnostics.iterations` |
| Space | `multiclass_logits` | `provenance.space` |
| Scope | the `change_vqa` specialist only | `scope.specialist` |

**Result: calibration did not improve — it moved slightly worse.** `ece_improvement` is negative. The NLL
improved microscopically, but the calibration-error metric got worse, and the honest reading is the ECE one.
The scaling is retained because it is part of the frozen configuration, **not** because it helped.

**Consumer contract** (`consumer_contract`): applied as `sigmoid(logit(z) / T)` for a scalar `z` and
`softmax(logits / T)` for a distribution; class `TemperatureCalibration`, module `evidence.confidence`;
read keys `temperature`, `fitted_on|split`, `artifact`, `n_samples`; resolved via
`load_calibration(config, base_dir='configs')`.

**Reliability diagram** (`reliability_diagram`): 15 equal-width bins over predicted-class confidence, with
per-bin accuracy, confidence, count and gap; `ece: 0.013755` (the **pre-scaling** value). The artifact's own
note: *"Equal-width bins over predicted-class confidence. ECE is bin-count sensitive and is not an aggregate
score."* Two bins (0.0–0.0667, 0.0667–0.1333) have `count: 0` and therefore `null` accuracy/confidence/gap.

**Scope caveat** (`scope.note`): *"This temperature calibrates the R-02 change-VQA head's answer confidence.
Other specialists emit their own raw scores and are unaffected."* So this artifact says nothing about the
calibration of any other specialist.

**Provenance** (`provenance`): the calibration is keyed to the change-VQA head `artifacts/change_vqa/run/head.pt`,
sha256 `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a`, `config_hash: 78f1e3700da15aa1`,
`dataset_id: cdvqa`, `feature_spec: change_feat_v1`. `type_mask_applied: false`.

### 4.8 Captioning — `IMPLEMENTED`, benchmark `NOT RUN`

The caption metric set is implemented and capability-gated (§3.6). **No caption benchmark score is
published** — no caption artifact exists under `artifacts/`, and the module states plainly: *"No caption
score is published anywhere by this change. Producing one requires a corpus of predictions and references,
which is a separate act."* The plan itself records the official caption definitions as `UNVERIFIED`
(`:4231-4237`), and ruling R-16 is open.

So captioning sits at: metrics `IMPLEMENTED`; capability measured (BLEU and ROUGE-L available, CIDEr and
BERTScore unavailable on the authoring machine); **benchmark `NOT RUN`**.

---

## 5. The grounding resolution experiment — a pre-registered rejection

This is the project's model of how a configuration decision should be made, and it is documented in full at
`docs/PHASE7_RESOLUTION_DECISION.md` (status `RESOLVED 2026-09-16`, decision **224 px**).

**The rule, fixed before the result was seen:**

```
448 WINS  if Recall@0.5 improves by >= 0.05 absolute
          OR mean best IoU improves by >= 0.05 absolute
224 WINS  otherwise
INCONCLUSIVE if fewer than 30 samples were scored
```

The artifact records `rule_changed_since_preregistration: false`.

**The run.** Full VRSBench eval split, **16,159 / 16,159 records scored at both resolutions**, Tesla T4,
`--all --device cuda --tag full`.

**The result.**

```
224 WINS
  recall@0.5 gain 448/224 : -0.0022
  bestIoU    gain 448/224 : -0.0147
  latency          ratio  : 1.59x
```

Neither component came close to the +0.05 margin. Both were **negative**.

**Measured detail.**

| Metric | 224 | 448 | delta |
|---|---|---|---|
| token grid | 7 × 7 = 49 | 14 × 14 = 196 | 4.0× tokens |
| attention cost (n²) | 1× | 16× | — |
| with boxes | 16159/16159 | 16159/16159 | — |
| **mean best IoU** | **0.0972** | 0.0825 | **−0.0147** |
| Recall@0.10 | **0.3298** | 0.2599 | **−0.0699** |
| Recall@0.25 | **0.1187** | 0.0944 | **−0.0243** |
| Recall@0.50 | **0.0234** | 0.0212 | **−0.0022** |
| latency mean | **20.0 ms** | 31.8 ms | 1.59× |
| latency p90 | **20.9 ms** | 32.9 ms | 1.57× |
| peak VRAM | **592.1 MB** | 599.8 MB | +7.7 MB |
| wall time | **~8.5 min** | ~11.2 min | 1.32× |

**448 is worse on every quality metric and slower. There is no axis on which it wins.**

**Best-IoU distribution** — the shift is a whole-distribution move toward the zero-overlap bucket, not a tail
effect:

| bucket | 224 | 448 |
|---|---|---|
| 0.00–0.10 | 10,829 | 11,957 |
| 0.10–0.25 | 3,412 | 2,677 |
| 0.25–0.50 | 1,540 | 1,182 |
| 0.50–0.75 | 336 | 307 |
| 0.75–1.01 | 42 | 36 |

**Paired analysis — independent confirmation.** Both resolutions scored the **same 16,159 samples**, so the
paired test removes between-object variance:

```
paired samples     : 16159
mean 224           : 0.0972
mean 448           : 0.0825
mean paired diff   : -0.0147   (95% CI -0.0160 .. -0.0134)
t statistic        : -22.63
CI excludes zero   : True
448 better on      :  1371/16159 ( 8.5%)
448 worse on       :  3372/16159 (20.9%)
identical          : 11416/16159 (70.6%)
```

**The paired test and the pre-registered rule agree.** There is no rule-versus-evidence disagreement to
escalate: both say 224, and the confidence interval excludes zero by a wide margin. The win/loss split is
itself informative: 448 wins on only 8.5 % of records and loses on 20.9 % — *"the finer grid is not merely
neutral, it is **actively harmful** on a fifth of the corpus."*

**The recall ladder, paired** (all three CIs exclude zero): 0.10 → −0.0699; 0.25 → −0.0243; 0.50 → −0.0022.
The gap **narrows as the threshold rises**, which is the signature of a method that cannot reach high IoU
either way.

**Degeneracy notes: none at full scale.** At n=12, n=40 and n=6 the smoke runs reported `Recall@0.5 = 0.0000`
at **both** resolutions and the script emitted a degeneracy warning. At full scale the metric is non-zero
(0.0234 / 0.0212), so the note correctly did not fire. *"The warning was therefore a genuine sub-floor
artefact, and the full run resolved it — which is exactly why the sub-floor runs were never treated as
evidence."*

**Why 448 did not help — the honest reading.** The zero-shot method selects a patch by text similarity and
returns that patch's box. At 224 a box is 1/7 of the image; at 448 it is 1/14. A finer grid is only better if
the target is small and the similarity peak lands on the correct fine cell. Two things work against that
here: the peak is not sharper at 448 (splitting each cell into four gives four chances to pick a wrong
sub-cell, and the similarity field on frozen features is smooth); and Recall@0.10 drops the most (−0.0699) —
*"if finer tokens genuinely localised better, the loosest threshold would benefit most."* The document states
plainly: *"This is the zero-shot baseline's limitation, not a property of RemoteCLIP."*

**What this establishes.** Grounding runs at **224**, frozen in `configs/base.yaml` (`grounding.image_size:
224`, `grounding.resolution_frozen: true`). Peak VRAM for the frozen encoder at 224 is **592 MB** — inside the
ZeroGPU free tier and inside a T4's 15 GB with room for a head and the VLM. Encoder latency at 224 on a T4 is
**20 ms/image** (5× faster than the CPU figure of 97 ms). The 224 localization floor is **1/7 of image width
per token**, documented as a known limitation of the zero-shot method.

**What this does NOT establish.** Whether the zero-shot baseline is good (it is not — mean best IoU 0.0972 and
Recall@0.5 0.0234 are **weak**; this is an ablation floor for the Phase 8 head, not a product). Whether a
trained head has the same resolution sensitivity — a learned head could in principle exploit finer positional
information a cosine-argmax cannot; re-opening the resolution question after Phase 8 would be legitimate
**if** the head's validation curve suggests it, and would be a **new pre-registered experiment, not a silent
retune**. And anything about hidden ISRO/SAC imagery — VRSBench is overhead optical; the hidden set is
Cartosat-2S + RISAT, a different distribution entirely.

**Reproduction.**

```bash
python scripts/exp_grounding_resolution.py \
    --vrsbench  <data-root> \
    --checkpoint <RemoteCLIP-ViT-B-32.pt> \
    --all --device cuda --tag full

python scripts/analyze_grounding_resolution.py --tag _full
```

Artifacts: `per_sample_224_full.jsonl`, `per_sample_448_full.jsonl`, `resolution_experiment_full.json` —
16,159 lines each. Every aggregate in the document is recomputable from the JSONL without re-running the
encoder.

**Action taken** (recorded in the document): `configs/base.yaml` retained `image_size: 224` and set
`resolution_frozen: true`, added the `grounding_head` block, extended `grounding_training`;
`allow_resolution_experiment` was **removed** — *"the experiment is complete, and a flag that says 'not yet
decided' is now false"*; `docs/ARCHITECTURE_FREEZE.md` marked the grounding row `RESOLVED`;
`tests/test_config.py` asserts the freeze fields; `specialists/grounding/head.py` trained at grid 7×7.

---

## 6. Behavioural evaluation (live validation)

Accuracy and behaviour are evaluated **separately**. The per-task protocols in §4 measure whether a model is
*correct*; this section measures whether the **deployed pipeline runs and routes correctly** on unseen
imagery and questions. It is behavioural evidence, and it is **not** an accuracy claim.

### 6.1 What was driven

The deployed stack was driven in a **headed browser** against production
(`https://satquery.pages.dev`), **one upload per case**, with per-case screenshots and recorded run
identifiers. Every case exercised the real path:

```
capabilities → assets → infer   (all on the Render orchestrator origin)
```

with a second `assets` call for the pair tasks. Each case produced a real `run_*` identifier, a live trace
bar, and a per-case screenshot.

**The result.**

| Property | Result |
|---|---|
| Independent full passes | **3** |
| Cases per pass | 8 (6 regression + 2 router-defect) |
| Passes at 8/8 | **3 of 3** |
| Live runs executed | **24** |
| Correct dispatches | **24** |
| Mock-node contamination | **0** on every live run |
| Trace fill | **94.4444 %** on every live run |
| Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) |

No run id is shared between passes.

### 6.2 The eight cases

**Phase A** re-runs the six capability cases (regression): vqa, caption, grounding, change, change_vqa,
optical_sar. **Phase B** is the router defect itself: each query uploads **one** asset, which is the exact
condition under which the old router collapsed to `vqa`.

| Case | Query | Expected | Dispatched | Run id (pass 3) |
|---|---|---|---|---|
| A1 | What type of terrain dominates this scene? | vqa | vqa | `run_fef26e91e7e6` |
| A2 | Describe the main visual characteristics of this scene. | caption | caption | `run_96281bdfcc08` |
| A3 | Where are the visible buildings in this image? | grounding | grounding | `run_e49adc8d319f` |
| A4 | What changed between the earlier and later image? | change | change | `run_aedc59cbcdc9` |
| A5 | Did the coastline advance between the two observations? | change_vqa | change_vqa | `run_62ca98d510be` |
| A6 | What land-cover characteristics … optical and SAR …? | optical_sar | optical_sar | `run_beacf6aa4e21` |
| **B1** | **Where are the built-up areas in this image?** | **grounding** | **grounding** | **`run_467ffa406f22`** |
| **B2** | **Where is the new airport?** | **grounding** | **grounding** | **`run_46980ba55c62`** |

**The defect, before and after.** `"Where are the built-up areas in this image?"` with one asset:

- **Before** (HEAD `9d57aed`, replayed through the shipped pre-fix functions):
  `reading=change, temporal=required → dispatched=vqa (wanted=change_vqa, substituted=true)`.
  `\bbuilt\b` matched the temporal regex and `area` matched inside `"areas"`.
- **After** (live): `reading=grounding, temporal=none → run_f0d7a90b5aa1` (pass 1),
  `[grounding] Located 6 candidate region(s) … Highest objectness 0.82.`

`"Where is the new airport?"` with one asset:

- **Before**: `reading=change, temporal=required → dispatched=vqa (wanted=change, substituted=true)`.
- **After** (live): `run_69e38a182a71`, 6 regions, highest objectness 0.83.

**The A5 discriminator note.** The interpretation panel shows the router's **reading**; the task actually
dispatched is the one the server tags on the answer. A5 reads `change` but answers `[change_vqa]` — the
documented quantifier upgrade, not a mismatch. The page's Answer block promises an answer, and the server's
`change` returns a spatial map with no language output; with one asset attached, `change` would instead be
refused outright (*"requires exactly 2 assets"*), so the console avoids asking for a pair-requiring task when
only one asset exists. This is **flagged, not failed**.

**Model-quality note (kept separate from deployment success).** The two verdicts are kept apart:

1. **Deployment/integration: PASS** — the full pipeline works on unseen imagery and questions.
2. **Model quality: MIXED** — caption and grounding are meaningful; change/change_vqa are plausible; VQA is
   weak-but-related (A1 answers "Grassland"); optical-SAR still returns a bare class index
   (`[optical_sar] Fused optical-SAR prediction: class_18 (margin 1.000; optical channels 4/12, SAR channels
   2/2)`), not a human label. The modality accounting in that answer again confirms the right channels reached
   the fusion head.

### 6.3 The three passes

| Pass | Deployed HEAD | Harness | Result |
|---|---|---|---|
| 1 | `ff46eba42b18` + `d413d3672311` | v1 (`fill_input`) | 8/8 |
| 2 | `2d7ae53b482d` | v2 asserting | 8/8 |
| 3 | `2d7ae53b482d` | v2 asserting (pre-discriminator-fix) | 8/8 (recomputed) |

Pass 1 ran against the post-fix frontend and the unchanged live backend (`SatQuery-Backend`
`89d80eaddec5`), with the tunnel agent connected (`agent_id: codespaces-fd1038`). Passes 2 and 3 ran against
the final HEAD `2d7ae53b482d`. Each pass produced fresh run identifiers, none shared with the others.

### 6.4 The harness false-positive bug — recorded in full

This is the most important paragraph in this section, because it is exactly the kind of silent
false-positive an evaluation harness must not have.

**What happened.** Pass 1's harness drove the query box with `fill_input()`. A later re-run attempt failed on
case 1 with `run_id=0002`, `mock_nodes=9`, `answer="No answer yet"`, and only a `capabilities` call — the
**mock path**.

**Root cause.** `fill_input` types with **real CDP key events**, and Chrome **drops synthesized key events
when the browser window does not hold OS focus**. The harness had **no assertion** on the query box, so it
clicked Run with the page's **default query** still in the box — and still recorded a "result". That is a
false pass: the harness dispatched the page's default query, not the case's query, and reported success.

**Measured directly.** With Chrome backgrounded, `press_key("Z")` left `#qtext.value` **unchanged**, while
`type_text("Q")` (CDP `Input.insertText`, which is **not** focus-gated) inserted fine.

**The fix.** The current harness **asserts form state before dispatch**:

- `q_ok` — the query box really holds the intended query (`q_got` records what it actually held);
- `obs_ok` — `#obsTail` reads `ready`;
- `t0_ok` — both frames are attached for pair tasks.

Every case in passes 2 and 3 carries `q_ok: true`, `obs_ok: true`, `t0_ok: true`, `no_mock_nodes: true`, a
real `run_*` id, trace fill 94.4444 %, the Hugging Face link in the DOM, and the `capabilities → assets →
infer` chain on the Render origin.

**Two further harness bugs, both causing false *failures* (not false passes):**

1. The answer `[task]` tag exists only for region tasks (vqa/caption answers are bare).
2. The intent panel is a concatenated string, so `task([a-z_]+)` must be matched **non-greedily** up to
   `modality`.

The harness now computes the dispatched task as `answer_tag` when present, else the reading.

### 6.5 The independent check that pass 1 was NOT infected

The bug in §6.4 raised the obvious question: was pass 1's 8/8 itself a false pass? It was checked, and it is
clean, on three independent grounds:

1. **Its intents are query-specific.** A1 reads `taskvqa…temporalnone`, **not** the default query's
   `taskchange…temporalrequired`. If the default query had been dispatched, the recorded intent would be the
   change intent — it is not.
2. **Its answers embed the query text.** Grounding answers read `[grounding] Located 6 candidate region(s)
   for 'Where are the built-up areas in this image?'` — the query text is inside the answer, so the intended
   query reached the server.
3. **A6's answer proves two files were uploaded** (`optical channels 4/12, SAR channels 2/2`), which the
   default single-asset path could not produce.

So pass 1's 8/8 stands. The failure mode is recorded here anyway, because it is the class of bug that
produces a green result from a red run.

### 6.6 The recompute-verdicts safety property

Pass 3 was launched with the harness build that still carried the two discriminator bugs (§6.4), so its raw
output says `SUMMARY 0/8 PASS`. The verdicts (8/8) are **recomputed from the recorded evidence** by
`recompute_verdicts.py`.

That is the intended safety property: **the recorded evidence (run id, `mock_nodes`, intent, answer,
assertions) is independent of the verdict computation**, so a harness bug can never silently turn a real
failure into a pass, and never costs a re-run to correct. The raw pass-3 records show
`task_dispatched: false` (the buggy check) while `answer_tag_agrees: true`, `no_mock_nodes: true`,
`live_run: true` — the recomputation reads the evidence, not the buggy predicate.

**Liveness notes recorded for anyone reproducing this** (`LIVE_VALIDATION_POSTFIX.md`):

- **`browser-use` block-buffers stdout even when redirected to a file**, so the output file stays at 0 bytes
  until the process exits — indistinguishable from a stall. The harness now forces line-buffering and prints
  `CASE_START <id>` per case.
- **`grep` block-buffers when piped**, so piping the harness through `grep` swallows all output if the
  pipeline is killed. Redirect to a file instead.

### 6.7 Known residuals after the live runs

- `"What is the new runway?"` still routes to `change` rather than `vqa` (the `new`-as-change heuristic fires
  on non-`where` questions). Strictly better than pre-fix, where `new` was unconditionally temporal.
  *"A lexical router cannot cleanly separate 'the new X' from 'what's new'."*
- `"How much built-up area was added?"` now routes to `vqa` (under-trigger) because `built` was dropped from
  the temporal set and `area` no longer matches inside `areas`.
- The old EO pair URLs still answer `200` from Cloudflare's **edge cache** (`CF-Cache-Status: HIT`) although
  the files are deleted — a cache-busted request returns `404`. Nothing references them.
- The Anatomy plate points at the 720×720 variant of an image the recorded run analysed at 730×730. The
  content is identical and the canvas scales it, but the page's *"the ACTUAL analysed image"* wording is very
  slightly loose.

---

## 7. Test-suite results

This section reports the test suites **in full, including the failures**. Nothing is hidden, and no failure is
claimed to be a regression when it is not.

### 7.1 Frontend live-wiring regression suite

`tests/unit/test_frontend_live_wiring.py` — **106 passed** (re-run this session).

The module proves two things that were **broken before the change** (quoted from the module docstring):

- **CORS.** The deployment allowed exactly one origin (`https://satquery.pages.dev`), so the frontend could
  not be driven from a local development server at all: every request from `http://localhost:8080` was
  refused with `Disallowed CORS origin`, and a developer had to deploy to Cloudflare to test a one-line
  JavaScript change. The fix adds explicit localhost origins while keeping production listed and keeping `*`
  rejected.
- **The task vocabulary.** The page's router and the server's `Task` enum must agree. They are two
  independently written vocabularies in two languages, and nothing previously checked that a value the
  frontend would send is a value the server accepts. The module reads **both** and compares them.

It also pins the `chang` word-boundary defect: `\bchang\b` cannot match `"changed"`, so the page's own default
question (*"What changed here?"*) routed to `vqa` instead of the change path. *"That is invisible to any test
that only checks 'is this a valid enum value' — both branches produce a valid value. The test therefore
asserts the ROUTE, not just the validity."*

The tests parse the shipped **source text**, because the frontend is plain ES5 with no build step and no
module exports, so there is no importable symbol; the parsing is anchored on **named tokens** rather than line
numbers, *"so it does not silently pass if the file is restructured."*

The test classes, read from the file, cover: the CORS allowlist; the CORS decision; the frontend↔server task
vocabulary; the default question reaching the change path; a location question **not** being a change
question; the architecture policy not reading a location as a change; the task respecting the pair
requirement; the live client using the real ingestion path; the page loading the live client; the honest
failure contract; the caption no longer calling a real upload illustrative; the frontend sending only the
assets the task requires; descriptive queries routing to caption; optical-SAR input validation; and server
error translation.

### 7.2 Doc/frontend suite

The doc/frontend suite (5 files) — **183 passed**. The five files are:

| File | What it pins |
|---|---|
| `tests/unit/test_frontend_guide_doc.py` | both documents agree on tasks, coordinate systems, evidence types, geometry shapes; no login screen; no secrets in the browser; the confidence-property trap; serialized analyses |
| `tests/unit/test_frontend_live_wiring.py` | see §7.1 |
| `tests/unit/test_api_contract_doc.py` | every JSON block parses and validates; no contract example fabricates an artifact URI; documented task/coordinate-system values are the real ones; every error code documented; the frozen config hash is recorded; the calibration caveat is recorded |
| `tests/unit/test_runbook_doc.py` | the runbook names only public serving entrypoints; quoted artifact sizes match the real files; capability list matches the registry; env vars are the architecture ones; undecided items declared undecided; **no claim that a deployment happened**; verified distinguished from designed |
| `tests/unit/test_deploy_config.py` | the deploy manifest declares the non-registry marker; the validator reports no errors; **the config-hash regression guard**; C-8 constraints (`torch_compile` false, cpu mode required, lazy load, single model cache) |

These suites are **documentation conformance tests**: they fail if a doc drifts from the code it describes.

### 7.3 Full `tests/unit` — the 5–6 environmental/ordering failures, reported honestly

Running the **entire** `tests/unit` suite in the authoring sandbox produces **5–6 failures**. They are
**environmental / ordering** failures, and they are attributable as follows:

| # | Failure | Attribution |
|---|---|---|
| 1–4 | `test_safe_delete_shim` — **4 failures** | the sandbox **delete guard** (environment) |
| 5 | one **ordering flake** in the router route test | test **ordering** (passes in isolation) |
| 6 | one **stale adapter test** (`optical_sar` absent when CROMA unshipped) | stale test — **CROMA is now shipped** |

**These are not regressions, and they are not hidden.** The evidence that they are not regressions is the
re-run: **re-running the affected files together gives 137 passed**.

**The `test_safe_delete_shim` failures in detail.** `tests/unit/test_safe_delete_shim.py` guards two defects
in the WorkBuddy Windows safe-delete shim (`cli/vendor/shim/sitecustomize.py`) that produced **phantom test
failures** in this repository. The module's own docstring records the mechanism:

- **Defect 1 — a verbatim temp path was not recognised as a temp path.** `_path_for_compare` returned
  `normcase(realpath(abspath(path)))`, which preserves the Windows verbatim prefix `\\?\`. `os.path.relpath`
  cannot relate `\\?\c:\...` to `c:\...`, so `_is_under_root` returned False and the OS-temp exemption in
  `_should_bypass_safe_delete` did not apply. Measured before the fix: *plain* temp subdir → bypass `True`;
  *verbatim* temp subdir → bypass `False` (the bug); non-temp subdir → bypass `False`. That is why routine
  pytest `garbage-*` collection — which walks
  `\\?\C:\Users\...\Temp\pytest-of-anish\garbage-*` — reached the bulk guard, tripped `confirmRequired` at 69
  entries against a threshold of 50, and **latched a rejection that then blocked every delete in the
  conversation**.
- **Defect 2 — a successful delete was reported as a failure.** `_platform_trash` raised whenever
  `SHFileOperationW` returned non-zero. Measured on the authoring host with
  `FOF_ALLOWUNDO|FOF_NOCONFIRMATION|FOF_NOERRORUI|FOF_SILENT`, calling shell32 directly: 4 regions × 3
  interleaved rounds returned `2, 2, 2, …` for non-temp and `0, 0, 0` for temp; 6 processes × 20 deletes
  returned `2 (120/120)` non-temp and `0 (120/120)` temp. The target is removed in **every** case and the
  Recycle Bin is populated and active (1,300+ `$I`/`$R` entries), so the return code is **not** a reliable
  "could not delete" signal.

The module states that the code is **intermittent** — *"across pytest invocations the same non-temp delete
sometimes returned 0, and in one run a temp delete returned non-zero. The precise Windows-internal trigger is
**NOT established** and is not claimed here."* This is one place where this document must write
**UNKNOWN — not established from the available evidence** (§9).

The module **skips cleanly** when the shim is absent or disabled: `pytest.importorskip("sitecustomize", ...)`
plus `skipif` markers for the shim helpers and for non-Windows hosts. So on a normal user environment these
tests skip; in the authoring sandbox they reach the guard and 4 of the 7 fail.

**The stale adapter test.** One test asserts `optical_sar` is absent when CROMA is unshipped. CROMA is now
shipped, so the assertion is stale. This is a **stale test**, not a code defect.

**The ordering flake.** One router route test fails only when the full suite runs in a particular order, and
**passes in isolation**. That is the definition of an ordering flake.

**A different environment state recorded earlier.** `docs/PHASE12_115_METRIC_COMPUTED.md` §8 records, for
2026-09-22, a full unit suite result of **2,237 passed, 0 failed (622.68 s)**. That is a real, measured
result in a *different* environment state (the safe-delete shim's guard state and the CROMA shipping status
differ). It is recorded here rather than suppressed, because the two results are both true and the difference
is exactly the environmental/ordering story. The same document records a correction: an earlier revision
claimed `2,179 passed` computed as `2,171 + 8`, *"presented as a check but the total was never measured — it
was inferred from a stale baseline."* The measured figure was 2,237.

**UNKNOWN.** The exact **collected** test count for the current-session full-suite run is
`UNKNOWN — not established from the available evidence`: the record gives the failure classification and the
137-passed re-run, but not a collected total.

### 7.4 Why the failures are not regressions — the 137-passed re-run

Re-running the affected files **together** gives **137 passed**. The logic: if the failures were real
regressions in the code under test, re-running those files together would still fail. They do not — which is
what distinguishes an environmental/ordering failure from a regression. The four shim failures are
*environment* (a sandbox guard), the flake is *ordering* (passes in isolation), and the sixth is a *stale
test* whose premise changed (CROMA shipped).

### 7.5 The evidence-engine purity and determinism tests

`tests/unit/test_evidence_engine.py` (73 test functions) pins the evidence engine's two properties that
matter — **determinism** and **no loss** — plus the confidence honesty rule. The module docstring states the
contract:

> *"determinism — the same claims must produce the same ordered, identified collection even when specialists
> finish in a different order, or when the uuids differ between processes. no loss — no specialist's evidence
> may vanish without being counted, and agreement between specialists must be recorded rather than collapsed
> with one name thrown away. Plus the honesty rule on confidence: an uncalibrated engine must pass the raw
> score through and SAY it is uncalibrated, never manufacture a fitted mapping."*

The core purity claim is `test_aggregate_is_deterministic_across_repeat_calls` (line 132): *"Same input ->
byte-identical output."* It asserts three things — identical `ids()`, identical `evidence_digest()`, and
identical `model_dump()` lists — across two calls with the same input. The determinism family also includes
`test_order_is_independent_of_input_order` (forward vs backward input order give the same digest),
`test_equal_scores_order_deterministically_by_coordinates` (*"Ties must not fall back to insertion order —
that is input-order dependence wearing a disguise"*), `test_ids_are_sequential_and_zero_padded`,
`test_ids_restart_from_one_for_each_aggregation`, and `test_source_results_are_not_mutated` /
`test_result_objects_are_not_mutated`.

On the confidence side, the tests pin that the engine never manufactures a fitted mapping:
`test_no_calibration_passes_raw_through_and_says_so`, `test_uncalibrated_is_never_labelled_as_fitted`,
`test_identity_temperature_is_reported_as_uncalibrated`, `test_engine_without_calibration_does_not_claim_calibration`,
and `test_temperature_scaling_is_deterministic`. The calibration artifact I/O is pinned too:
`test_missing_artifact_degrades_instead_of_failing`, `test_missing_artifact_raises_when_required`,
`test_malformed_artifact_is_not_silently_swallowed`, `test_artifact_without_a_temperature_is_rejected`.

**Environment note carried in the module docstring:** *"The `--basetemp=.pytest_tmp` flag is mandatory (see
the project handoff): the default pytest temp root triggers a sandbox denial on this machine."* The module's
`scratch` fixture is deliberately **not** cleaned up on teardown: *"the sandbox's safe-delete guard rejects
the recursive delete."* See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) §9.

### 7.6 The metric-verification tool

[`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py) is the machine check behind
Rule 1. It is **read-only**, walks each quoted claim to its source artifact, and compares at the **printed
precision**. It resolves **nested** artifact keys, including keys that themselves contain dots — the `recall`
dict is keyed `"0.10"/"0.25"/"0.50"` — by preferring the **longest** matching key at each step, *"so a naive
`split('.')` walk would break."*

**20 numeric claims** are checked. They are:

| # | Claim | Artifact | Key path |
|---|---|---|---|
| 1 | change pooled IoU | `artifacts/change/eval_test/eval_result.json` | `metrics.pooled.iou` |
| 2 | change macro IoU | same | `metrics.macro.miou` |
| 3 | change pooled F1 | same | `metrics.pooled.f1` |
| 4 | grounding canonical `head_threshold` mean_best_IoU | `…/eval_result_canonical.json` | `results.head_threshold.mean_best_iou` |
| 5 | grounding canonical `head_threshold` recall@0.5 | same | `results.head_threshold.recall.0.50` |
| 6 | grounding matched6 `head_threshold` mean_best_IoU | `…/eval_result_matched6.json` | `results.head_threshold.mean_best_iou` |
| 7 | grounding matched6 `head_threshold` recall@0.5 | same | `results.head_threshold.recall.0.50` |
| 8 | grounding `head_argmax` mean_best_IoU (canonical) | `…/eval_result_canonical.json` | `results.head_argmax.mean_best_iou` |
| 9 | grounding zero-shot baseline IoU (canonical) | same | `results.zero_shot_matched.mean_best_iou` |
| 10 | optical-SAR accuracy | `…/pre_registered_115_metric.json` | `accuracy` |
| 11 | optical-SAR macro_F1 | same | `macro_f1` |
| 12 | change_vqa test accuracy | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test_accuracy` |
| 13 | change_vqa test macro_F1 | same | `verification.test_macro_f1` |
| 14 | change_vqa test2 accuracy | same | `verification.test2_accuracy` |
| 15 | change_vqa test2 macro_F1 | same | `verification.test2_macro_f1` |
| 16 | router overall ungated accuracy | `artifacts/router/threshold_sweep_val.json` | `overall_ungated_accuracy` |
| 17 | calibration ECE before scaling | `artifacts/calibration_v001.json` | `metrics.ece_before` |
| 18 | calibration ECE after scaling | same | `metrics.ece_after` |
| 19 | VLM adapter exact_match | `artifacts/vlm/phase6_closure.json` | `why_usable_verified.adapted_test.exact_match` |
| 20 | VLM adapter F1 | same | `why_usable_verified.adapted_test.f1` |

It **also asserts statuses** — that the VLM `headline` contains the literal `ACCEPTANCE-REJECTED`; the
router's `corpus_limited` and `n_val`; the calibration `temperature` and `ece_improvement`. It prints
`ALL CLAIMS VERIFIED` and exits 0 only when everything matches.

**Committed output** (`../tools/readme_metrics_report.txt`): all 20 rows `MATCH`, and the status assertions
print:

```
VLM headline contains ACCEPTANCE-REJECTED : True
VLM status                               : CLOSED
router corpus_limited                    : True
router n_val                             : 86
calibration temperature (temperature_scaling.temperature) : 0.9772731820958189
calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)

RESULT: ALL CLAIMS VERIFIED
```

The comparison is **at the printed precision**, which is why e.g. change-VQA `test_accuracy` reads
`0.697626367` in the artifact and `0.697626` in the README, and both are `MATCH`.

---

## 8. What is NOT evaluated — exhaustive

This is the honest ledger. Every item here is something the evaluation does **not** establish. Nothing in
this table is a claim of success.

### 8.1 Benchmarks that do not exist or were not run

| Item | State | Note |
|---|---|---|
| **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router → specialist → envelope pipeline. **No such number is claimed anywhere, and none is produced by averaging the per-task numbers.** |
| **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored; `n_test_examples_scored: 0`, `test_split_touched: false`. |
| **Benchmark adapters** | **NOT RUN** | The four real-corpus adapters exist and the corpora are on disk, but adapter-based benchmark runs were not executed. The registry is empty at import. |
| **End-to-end latency benchmark** | **NOT RUN** | Per-specialist latency is recorded incidentally (e.g. grounding `latency_ms_per_image` 2.205 ms for the head; change eval 55.359 s for 2,048 tiles). There is no end-to-end latency benchmark. |
| **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. No specialist is evaluated on another family's data. |
| **Human evaluation** | **NOT RUN** | No human rating of any output. |
| **Adversarial / robustness evaluation** | **NOT RUN** | The plan's §60 Adversarial Tests enumerate cases (blank image, all-zero image, extremely bright/dark, noise, unsupported TIFF, 10000×10000, missing metadata, wrong modality labels, same image twice, one temporal image, three images, optical/SAR size mismatch). Those are **test-case intentions**, not a run robustness benchmark; no adversarial *evaluation* was scored. |
| **Caption benchmark** | **NOT RUN** | The caption metrics are implemented and capability-gated, but **no caption score is published**; there is no caption artifact. |
| **Public-test evaluation** | **NOT RUN** | The public-test corpus directory is **empty**; `available=False`; `open_public_test()` raises rather than evaluating nothing. |
| **Multi-label BigEarthNet/reBEN evaluation** | **NOT RUN** | The scored subset is single-label; a multi-label evaluation was not performed. |
| **Calibrated reliability curve** | **NOT RUN (not plotted)** | The plotted reliability diagram is the pre-scaling one; the calibrated curve is not plotted. |
| **Semantic match for Change-VQA** | **NOT RUN** | The plan leaves it optional ("if the benchmark specifies it"); it is not attempted. |
| **mAP for grounding** | **NOT RUN** | The plan says "mAP where applicable"; no mAP was computed. |

### 8.2 Things measured but whose acceptance ruling is OPEN or REJECTED

| Item | State | Why it must not be read as a pass |
|---|---|---|
| Optical-SAR fusion | **OPEN** | `is_deciding_statistic: False`; the tool *"selects no head, ranks nothing and compares no arms"*. The metric is single-label-subset and not comparable to published BigEarthNet numbers. |
| Change-VQA | **OPEN** | The artifact says *"the plan leaves the accuracy/macro-F1 interpretation owner-gated. No official aggregate metric is asserted here."* |
| VLM adapter | **ACCEPTANCE-REJECTED** | V2's per-class guardrail failed on one class (Mixed forest, 4 questions of 33, z = 2.1335) even though the aggregate endpoint improved by +49.5 pp. |
| Calibration | **not an improvement** | ECE worsened (0.013755 → 0.014929). |
| Grounding | **MEASURED, protocol-sensitive** | Two protocols and three decode variants; absolute values are protocol-sensitive; the argmax decode (0.1215) is barely above the zero-shot baseline (0.0972). |
| Router | **TEST NOT RUN** | Val-only, n = 86, corpus-limited, 0 hard negatives in val. |

### 8.3 Distribution and generalisation gaps

| Item | State |
|---|---|
| Hidden ISRO/SAC imagery (Cartosat-2S + RISAT) | **NOT RUN** — VRSBench is overhead optical; the hidden set is a different distribution entirely. `docs/PHASE7_RESOLUTION_DECISION.md` says so explicitly. |
| Sensors other than Sentinel-1/2 in the fusion path | **NOT RUN** — CROMA's pretrained architecture is Sentinel-1/Sentinel-2 oriented; the sensor adapter exists but no non-S1/S2 evaluation was scored. |
| Grounding on a trained head at 448 | **NOT RUN** — the resolution decision was made on the zero-shot method; a learned head's resolution sensitivity is unknown and would require a new pre-registered experiment. |
| Any claim about all 19 optical-SAR classes | **NOT RUN** — 5 of 19 have no test samples. |
| Any claim about rare classes in change-VQA | **NOT RUN** — the macro-F1 is low and the per-class breakdown is not published in the release artifacts. |

### 8.4 Evaluation-infrastructure items that are implemented but not exercised end-to-end

| Item | State |
|---|---|
| The benchmark-adapter runner against a real corpus | `IMPLEMENTED` — not run in this release (`NOT RUN`). |
| The public-test seal/verify cycle on a real corpus | `IMPLEMENTED` — no corpus exists to seal; sealing an empty corpus is refused by design. |
| The hidden-compatible evaluation mode (plan §62) | `IMPLEMENTED` as a contract; no hidden imagery was available, so it was not run. |
| The `RESOURCE_BLOCKED` scorecard state | `IMPLEMENTED` — describes the case where the immutable evaluation corpus does not exist. |

### 8.5 Explicit non-claims

- **No claim of state-of-the-art performance** on any benchmark.
- **No claim of production readiness** for model quality.
- **No claim that the trained heads generalise** beyond their training-family test splits.
- **No claim that calibration improves confidence.**
- **No claim that the VLM adapter is accepted** for production use.
- **No system-level accuracy** is claimed anywhere.

---

## 9. What is NOT RUN / OPEN / BLOCKED for this topic

| Item | Status |
|---|---|
| System-level end-to-end benchmark | **NOT RUN — none exists** |
| Router test split | **NOT RUN** |
| Benchmark-adapter runs | **NOT RUN** |
| End-to-end latency benchmark | **NOT RUN** |
| Cross-dataset generalisation | **NOT RUN** |
| Human evaluation | **NOT RUN** |
| Adversarial / robustness evaluation | **NOT RUN** |
| Caption benchmark score | **NOT RUN** |
| Public-test corpus evaluation | **NOT RUN** (corpus empty; reported `available=False`) |
| Multi-label BigEarthNet/reBEN evaluation | **NOT RUN** |
| Optical-SAR acceptance ruling | **OPEN** |
| Change-VQA acceptance ruling | **OPEN** |
| VLM adapter acceptance | **REJECTED** (`ACCEPTANCE-REJECTED`) |
| Calibration | **MEASURED — worse**; retained only because frozen |
| Grounding | **MEASURED — protocol-sensitive**; two protocols, three decode variants |
| The 5–6 full-suite test failures | **environmental / ordering**, not regressions (137 passed on the affected files re-run together) |
| The precise Windows-internal trigger for the safe-delete shim's intermittent return code | `UNKNOWN — not established from the available evidence` (per the shim test module) |
| The exact collected count for the current-session full-suite run | `UNKNOWN — not established from the available evidence` |

---

## 10. Where the evidence lives

| What | Where |
|---|---|
| Change detection metrics | `artifacts/change/eval_test/eval_result.json` |
| Change threshold sweep | `artifacts/change/threshold_sweep_val.json` |
| Grounding metrics (canonical) | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` |
| Grounding metrics (matched6) | `artifacts/grounding/remoteclip_grounding_v001/eval_result_matched6.json` |
| Grounding resolution experiment | `artifacts/grounding/resolution_experiment.json`, `…_ck.json`, `…_smoke.json`; `docs/PHASE7_RESOLUTION_DECISION.md` |
| Grounding per-sample (smoke only) | `artifacts/grounding/per_sample_224_smoke.jsonl`, `…_448_smoke.jsonl` |
| Optical-SAR pre-registered metric | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
| Optical-SAR computation narrative | `docs/PHASE12_115_METRIC_COMPUTED.md` |
| Optical-SAR label policy | `docs/PHASE12_LABEL_POLICY_DECISION.md` |
| Optical-SAR decisions | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
| Change-VQA promotion | `artifacts/change_vqa/run/PROMOTION.json` |
| VLM closure | `artifacts/vlm/phase6_closure.json`; `docs/PHASE6_CLOSURE.md`, `docs/PHASE6_AUDIT_AND_CONTRACT.md`, `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md` |
| VLM adapter verification | `artifacts/vlm/run1_test_recovery/adapter_verification.json`, `…/run_manifest.json`, `…/test_adjudication.json` |
| Router threshold sweep | `artifacts/router/threshold_sweep_val.json` |
| Calibration | `artifacts/calibration_v001.json` |
| Metric verification tool + output | `../tools/verify_readme_metrics.py`, `../tools/readme_metrics_test_report.txt` (see `../tools/readme_metrics_report.txt`) |
| Evaluation subsystem source | `evaluation/` (runner, leakage, manifests, normalize, run_manifest, benchmark_adapters/, metrics/, public_test/) |
| Leakage / firewall tests | `tests/unit/test_evaluation_runner.py`, `tests/leakage/`, `tests/unit/test_manifest_freeze.py`, `tests/unit/test_prompt_freeze.py` |
| Metric tests | `tests/unit/test_change.py`, `test_grounding_metrics.py`, `test_vqa_metrics.py`, `test_caption_metrics.py`, `test_eval_normalize.py`, `test_eval_fusion_115.py` |
| Evidence-engine purity/determinism tests | `tests/unit/test_evidence_engine.py` |
| Frontend live-wiring suite | `tests/unit/test_frontend_live_wiring.py` |
| Doc/frontend suites | `tests/unit/test_frontend_guide_doc.py`, `test_api_contract_doc.py`, `test_runbook_doc.py`, `test_deploy_config.py` |
| Safe-delete shim tests | `tests/unit/test_safe_delete_shim.py` |
| Live validation (3 passes) | `live_validation/LIVE_VALIDATION_POSTFIX.md`, `results_final.json`, `results_pass3.json`, `run_output.txt`, `run_final2.txt`, `run_final3.txt`, `A1…B2*.png` |
| Live validation recomputation | `live_validation/recompute_verdicts.py` |
| Final delivery report (test results §7) | `docs/FINAL_DELIVERY_REPORT.md` |
| Frozen config | `configs/base.yaml` §`evaluation`, §`grounding`, §`grounding_head`, §`croma`, §`fusion` |

**Reproduce the machine check:**

```bash
python release/tools/verify_readme_metrics.py
```

It prints `ALL CLAIMS VERIFIED` and exits 0 only when all 20 numeric claims match and the status assertions
hold. See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the full reproduction guide, including the
environment traps and the exact commands.