docs/RESEARCH_NOTES.md

Research Notes

Engineering findings, negative results, and design decisions that would otherwise be lost. Every one was learned by probe or execution, not assumption, and every one is recorded so it is not rediscovered. Where a finding changed code, the change is named; where a finding was reported rather than patched, the reason is given.

Status tags: MEASURED · RESOLVED · REJECTED · OPEN · ATTEMPTED.

How to read this document. §2 lists the twelve numbered findings that changed the code (F4-1…F5-3, P7-1, C-1, C-6…C-9), each with what was measured, the consequence, and the file that records it. §3–§9 are the larger case studies: the pre-registered resolution rejection, the router-defect fix, the harness false-positive, the auto-mode fallthrough, the interpret()/chooseTask() asymmetry, the environment findings, and the BigEarthNet contradiction.

Companions. LIMITATIONS.md (the exhaustive catalogue of what does not work), REPRODUCIBILITY.md (the environment traps in operator form), architecture/04-router.md, DEPLOYMENT.md §8.1.

---

1. The convention that matters most here

A finding is only recorded when it was measured, and the measurement is quoted rather than summarised. Two examples of why this convention exists, both recorded in the source documents:

• A per-class guardrail (V2, below) was written with a 1.0 pp threshold that sits below the measurement resolution of its own statistic — it flagged three classes whose drops were statistically indistinguishable from zero. The rule was wrong, not the model.
• A "bounded, not fixed" classification of a path-disclosure finding was wrong in three ways, and the third mattered most: it was not bounded and not measured. The recorded lesson is in the wording: "bounded" is a claim, and a claim is not a measurement.

Where a value could not be established, the honest entry is "not measured" rather than a hedge.

---

2. Findings that changed the code

F4-1 — the MiniLM tokenizer ceiling is 256, not 128 (MEASURED)

What was measured. The config said router.max_length: 128. A probe showed the MiniLM tokenizer's own ceiling is 256. 128 is therefore a deliberate truncation well inside the ceiling, not the model's limit — but the config read as if it were the latter.

Consequence. router.max_length: 128 is retained, but FrozenEncoder now applies it explicitly and rejects any value above 256, because truncating above the ceiling is a silent no-op — "a control that appears to work and does nothing". Satellite queries are short, so halving the sequence halves attention cost for no measurable accuracy loss.

Recorded in. docs/PHASE4_ROUTER_REPORT.md §F4-1; docs/ARCHITECTURE_FREEZE.md; artifacts/router/router_adapter_v001/metadata.json (encoder.max_length: 128).

F4-2 — the router does not need a GPU (MEASURED)

What was measured. The encoder is frozen, so embeddings are a pure function of the query text. Embedding the corpus measured 0.118 s / 64 queries on CPU; training the 51,725-parameter adapter on the cached vectors measured 0.28 s for 20 epochs over 4,096 vectors.

Consequence. The router can be retrained during development at zero GPU quota cost. The plan's budget line — "Router | CPU/T4 | <1 h" — was roughly three orders of magnitude pessimistic; Phase 4 ran to completion locally.

Number discipline. artifacts/router/router_adapter_v001/metadata.json records num_parameters: 51725. The measured figure is 51,725; use it rather than any other recollection of the parameter count.

Recorded in. docs/PHASE4_ROUTER_REPORT.md §F4-2; artifacts/router/router_adapter_v001/metadata.json.

F4-3 — the corpus needs group-level splitting (MEASURED)

What was measured. Template-generated queries are near-duplicates. Splitting by example would put "Show me the water body." in train and "Show me the road." in val — one token apart — and report a fake accuracy.

Consequence. Splits are by group (template id, or hard-negative family), never by example, mirroring evaluation.leakage.assign_splits_by_scene deliberately: "the failure mode is identical, so the guard should look identical". Hard-negative families are placed in the test split so their accuracy measures generalisation rather than memorisation. The corpus metadata records groups: 54 over total: 576 queries.

Recorded in. docs/PHASE4_ROUTER_REPORT.md §F4-3; artifacts/router/router_adapter_v001/metadata.json (corpus.groups).

F5-1 — AutoModelForVision2Seq is absent, not deprecated (MEASURED)

What was measured. In transformers 5.17.0:

Class	Present?
AutoModelForImageTextToText	present
AutoModelForVision2Seq	ABSENT
AutoModelForMultimodalLM	present

The class does not exist (it is not merely deprecated).

Consequence. The loader is resolved by feature detection, never hardcoded to one class name, so a transformers version that renames or removes a class does not break the load path.

Recorded in. docs/PHASE5_VLM_CONTRACT.md §F5-1.

F5-2 — the processor cost overrun is ~17×, not 4× (MEASURED)

What was measured. The processor's default longest_edge is 2048, which upscales a 512-px tile 4× and then splits it (do_image_splitting=True) into sub-images:

INPUT: one 512×512 RGB tile

DEFAULT  (size.longest_edge = 2048, do_image_splitting = True)
    pixel_values  (1, 17, 3, 512, 512)     <- 17 images
    prompt tokens 1142

PINNED   (processor_longest_edge = 512)
    pixel_values  (1, 1, 3, 512, 512)      <- 1 image

The plan estimated a 4× cost overrun; the real figure is ~17×.

Consequence. processor_longest_edge must be set explicitly on the processor at construction time, and core/config.py now enforces processor_longest_edge <= image.tile_size so the pin is a control rather than a comment. Confirmed in the real load path: max_images_seen = 1 — "the F5-2 pin holds in the real load path, not just the probe. Unpinned it would read 17."

Recorded in. docs/PHASE5_VLM_CONTRACT.md §F5-2 (and the max_images_seen confirmation); docs/ARCHITECTURE_FREEZE.md (vlm.processor_longest_edge: 512).

F5-3 — SmolVLM requires <image> tokens in the prompt (MEASURED)

What was measured. Hand-written prompt strings fail:

ValueError: The total number of <image> tokens in the prompts should be the
same as the number of images passed. Found [0] <image> tokens and [1] images

Consequence. Prompts are always built through processor.apply_chat_template(), enforced by the config key vlm.prompt_must_use_chat_template: true.

Recorded in. docs/PHASE5_VLM_CONTRACT.md §F5-3.

P7-1 — the RemoteCLIP projected dimension is 512, not 768 (MEASURED)

What was measured. visual.positional_embedding is 768 wide, but visual.proj is (768, 512) — the embeddings comparable against the text tower are the projected ones, i.e. 512 (dim_match: True). The grounding head's per-cell feature is therefore 4 × 512 = 2048.

Consequence. Using 768 anywhere in the grounding path would be a silent shape error, caught only at the similarity computation — after the patch features have already been computed and cached. grounding.encoder_projected_dim: 512 is declared in config so core/config.py can validate the head without importing torch, and RemoteCLIPEncoder._verify_contract() asserts the same value against the real model at load time.

Recorded in. docs/PHASE7_GROUNDING_CONTRACT.md §P7-1; docs/ARCHITECTURE_FREEZE.md.

C-1 — the availability mask is consumed by the head, not by CROMA (MEASURED)

What was measured. CROMA always sees the canonical channel counts (12 optical, 2 SAR, zero-filled to canonical order). The availability mask is applied by the fusion head, not the encoder. The verified fusion-head input dimensions:

optical_GAP      (B, 768)
SAR_GAP          (B, 768)
joint_GAP        (B, 768)
optical_mask     (B, 12)     <- availability, from the sensor adapter
sar_mask         (B,  2)     <- availability, from the sensor adapter
                 ---------
concat           (B, 2318)

i.e. input_dim = 3 × 768 + 12 + 2 = 2318.

Consequence. Channel/band dropout during fusion-head training is mandatory — "it is what teaches the head to trust the availability mask". The sensor adapter never fabricates a missing band; missing channels are masked/zero-filled per the validated adapter policy.

Recorded in. docs/ARCHITECTURE_FREEZE.md §2.5 (C-1).

C-6 — T4 is SM 7.5, so training uses fp16, not bf16 (MEASURED)

What was measured. The target GPU (Tesla T4) is compute capability 7.5; bf16 tensor cores are unavailable there.

Consequence. training.precision: fp16. The loader validates the value is one of fp16|bf16|fp32. The fp16-vs-planned-bf16 deviation is documented as a finding in its own right, not hidden.

Recorded in. docs/ARCHITECTURE_FREEZE.md (training.precision row, C-6); docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md §6.3.

C-7 — image_resolution % 8 == 0 (MEASURED)

What was measured. CROMA requires image_resolution % 8 == 0. The native value 120 satisfies it and yields 225 patches.

Consequence. Enforced at config load (croma.image_resolution: 120).

Recorded in. docs/ARCHITECTURE_FREEZE.md (croma.image_resolution row, C-7).

C-8 — ZeroGPU does not support torch.compile (MEASURED)

What was measured. torch.compile must never be enabled on the (historical) ZeroGPU target.

Consequence. Enforced: the loader fails startup if deployment.torch_compile is true. This is why setting SATQUERY_TORCH_COMPILE=true fails startup rather than silently taking effect (see DEPLOYMENT.md §6.3).

Recorded in. docs/ARCHITECTURE_FREEZE.md (deployment.torch_compile row, C-8); configs/deploy.yaml header.

C-9 — STANet hyperparameters are upstream-verified (MEASURED)

What was measured. Change detection uses a STANet-style architecture with upstream-verified hyperparameters: ResNet-18 encoder, PAM self-attention mode, tile 256 (non-overlapping), threshold 0.50, loss 0.5·BCE + 0.5·Dice, lr = 1e-3, batch_size = 8. The LEVIR-CD split is 7120 / 1024 / 2048 (256-px patches). Post-processing: threshold → morphological cleanup → connected components → minimum-component filter.

Consequence. The implementation is reimplemented, not vendored — so the upstream hyperparameters are pinned as the contract, and the trained head's embedded metadata carries the same values (sa_mode: "PAM", width: 128, encoder: "resnet18").

Recorded in. docs/ARCHITECTURE_FREEZE.md §2.5 (C-9); artifacts/change/eval_test/eval_result.json (checkpoint_embedded_config).

---

3. The grounding resolution decision — a pre-registered rejection (REJECTED)

Question. Should grounding decode at 448 or 224?

Answer: 224. 448 was rejected — notable because the rejection was pre-registered and then confirmed by a paired test over identical samples (n = 16,159), on a Tesla T4 (--all --device cuda --tag full).

3.1 The rule, fixed before the result was seen

448 WINS  if Recall@0.5 improves by >= 0.05 absolute
          OR mean best IoU improves by >= 0.05 absolute
224 WINS  otherwise
INCONCLUSIVE if fewer than 30 samples were scored

The artifact records rule_changed_since_preregistration: false — the rule was not modified after the result was seen.

3.2 The result

224 WINS
  recall@0.5 gain 448/224 : -0.0022
  bestIoU    gain 448/224 : -0.0147
  latency          ratio  : 1.59x

Neither component came close to the +0.05 margin. Both were negative.

3.3 Measured detail

metric	224	448	delta
token grid	7 × 7 = 49	14 × 14 = 196	4.0× tokens
attention cost (n²)	1×	16×	—
with boxes	16159/16159	16159/16159	—
mean best IoU	0.0972	0.0825	−0.0147
Recall@0.10	0.3298	0.2599	−0.0699
Recall@0.25	0.1187	0.0944	−0.0243
Recall@0.50	0.0234	0.0212	−0.0022
matched IoU	0.0972	0.0825	−0.0147
latency mean	20.0 ms	31.8 ms	1.59×
latency p90	20.9 ms	32.9 ms	1.57×
peak VRAM	592.1 MB	599.8 MB	+7.7 MB
wall time	~8.5 min	~11.2 min	1.32×

448 is worse on every quality metric and slower. There is no axis on which it wins.

Best-IoU distribution — the shift is a whole-distribution move toward the zero-overlap bucket, not a tail effect:

bucket	224	448
0.00–0.10	10,829	11,957
0.10–0.25	3,412	2,677
0.25–0.50	1,540	1,182
0.50–0.75	336	307
0.75–1.01	42	36

The 224 column dominates the top three buckets; 448 has ~1,100 more near-total misses.

3.4 Paired analysis — independent confirmation

Both resolutions scored the same 16,159 samples, so the paired test removes between-object variance:

paired samples     : 16159
mean 224           : 0.0972
mean 448           : 0.0825
mean paired diff   : -0.0147   (95% CI -0.0160 .. -0.0134)
t statistic        : -22.63
CI excludes zero   : True

448 better on      :  1371/16159 ( 8.5%)
448 worse on       :  3372/16159 (20.9%)
identical          : 11416/16159 (70.6%)

The paired test and the pre-registered rule agree. There is no rule-versus-evidence disagreement to escalate: both say 224, and the confidence interval excludes zero by a wide margin. The win/loss split is also informative: 448 wins on only 8.5 % of records and loses on 20.9 % — the finer grid is not merely neutral, it is actively harmful on a fifth of the corpus.

3.5 Recall ladder, paired

threshold	224	448	diff	95% CI
0.10	0.3298	0.2599	−0.0699	excludes zero
0.25	0.1187	0.0944	−0.0243	excludes zero
0.50	0.0234	0.0212	−0.0022	excludes zero

The gap narrows as the threshold rises — the signature of a method that cannot reach high IoU either way. At IoU 0.50 the two are within 0.002 of each other and both are near the floor.

3.6 Why 448 did not help — the honest reading

The zero-shot method selects a patch by text similarity and returns that patch's box. At 224 a box is 1/7 of the image; at 448 it is 1/14. Two things work against the finer grid:

1. The peak is not sharper at 448. Splitting each cell into four gives four chances to pick a wrong sub-cell, and the similarity field on frozen features is smooth, so the argmax moves around. 448 wins on 8.5 % and loses on 20.9 % — losses outnumber wins by 2.5 : 1.
2. Recall@0.10 drops the most (−0.0699). If finer tokens genuinely localised better, the loosest threshold would benefit most. It degrades most, which means the fine grid adds positional noise rather than positional precision.

This is the zero-shot baseline's limitation, not a property of RemoteCLIP. A learned head trained to regress boxes from these features may respond differently.

3.7 What this establishes, and what it does not

Establishes: grounding runs at 224 (frozen in configs/base.yaml: grounding.image_size: 224, grounding.resolution_frozen: true); peak VRAM for the frozen encoder at 224 is 592 MB; encoder latency at 224 on a T4 is 20 ms/image (5× faster than the CPU figure of 97 ms); the 224 localisation floor is 1/7 of image width per token.

Does not establish: whether the zero-shot baseline is good (it is not — mean best IoU 0.0972 and Recall@0.5 0.0234 are weak, and this is an ablation floor for the Phase-8 head, not a product); whether a trained head has the same resolution sensitivity (re-opening the question after Phase 8 is legitimate if the head's validation curve suggests it, and would be a new pre-registered experiment, not a silent retune); anything about hidden ISRO/SAC imagery (VRSBench is overhead optical; the hidden set is Cartosat-2S + RISAT, a different distribution entirely).

Degeneracy note. At n=12, n=40 and n=6 the smoke runs reported Recall@0.5 = 0.0000 at both resolutions and the script emitted a degeneracy warning; at full scale the metric is non-zero (0.0234 / 0.0212), so the note correctly did not fire. The sub-floor runs were never treated as evidence.

Reproduction.

python scripts/exp_grounding_resolution.py \
    --vrsbench  <data-root> \
    --checkpoint <RemoteCLIP-ViT-B-32.pt> \
    --all --device cuda --tag full

python scripts/analyze_grounding_resolution.py --tag _full

Artifacts: per_sample_224_full.jsonl, per_sample_448_full.jsonl, resolution_experiment_full.json — 16,159 lines each. Every aggregate is recomputable from the JSONL without re-running the encoder.

Recorded in. docs/PHASE7_RESOLUTION_DECISION.md (full document).

---

4. The router defect — a real bug, found and fixed (RESOLVED)

4.1 Symptom

The query "Where are the built-up areas in this image?" — with one asset attached — collapsed to vqa and answered "River", instead of routing to grounding. A second query, "Where is the new airport?", behaved the same way.

4.2 Root cause

Two functions with different information:

• interpret() — produces the console's reading; asset-count-blind (text only).
• chooseTask() — performs dispatch; asset-count-aware.

The defect was in the reading/dispatch path's handling of spatial/lexical cues. The pre-fix replay of the shipped functions shows the mechanism exactly:

"Where are the built-up areas in this image?" — one asset.
Before: reading=change, temporal=required → dispatched=vqa (wanted=change_vqa, substituted=true).
        \bbuilt\b matched the temporal regex and `area` matched inside "areas".

So the word "built" was in the temporal (change) set, and "area" matched as a substring of "areas". Region queries fell through to the generic VQA specialist.

4.3 Fix and verification

The fix was deployed to SatQuery-Frontend (mission.js interpret(), ff46eba42b18) and the sibling core.js SQ.policy (2d7ae53b482d). It was validated by three independent live passes:

Pass	Deployed HEAD	Result
1	ff46eba42b18 + d413d3672311	8/8
2	2d7ae53b482d	8/8
3	2d7ae53b482d	8/8

Both defect queries now dispatch to grounding:

Query	Pass-1 run id	Pass-2 run id	Pass-3 run id
Where are the built-up areas in this image?	run_f0d7a90b5aa1	run_2a07dcdbae96	run_467ffa406f22
Where is the new airport?	run_69e38a182a71	run_9134f40a258c	run_46980ba55c62

24 live runs, 24 correct dispatches, 0 mock nodes. No run id is shared between passes. Post-fix the live answers read reading=grounding, temporal=none and [grounding] Located 6 candidate region(s) … Highest objectness 0.82 / 0.83. Screenshots (A1…B2*.png) are in the live-validation scratch directory.

4.4 The residuals the fix left (and did not hide)

• "What is the new runway?" still reads change rather than vqa (the new-as-change heuristic fires on non-where questions). Strictly better than pre-fix, where new was unconditionally temporal. "A lexical router cannot cleanly separate 'the new X' from 'what's new'."
• "How much built-up area was added?" now reads vqa (under-trigger), because built was dropped from the temporal set and area no longer matches inside areas.

Recorded in. docs/FINAL_DELIVERY_TODO.md §5 B-08, §6 E-09/E-11/E-14; .workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md; run_final2.txt, run_final3.txt; results_final.json, results_pass3.json.

4.5 The corpus the fix was validated against (and its limits)

The router is trained and validated on a 576-query corpus in 54 groups (corpus_total: 576, corpus_groups: 54), distributed by task: caption 91, change 115, grounding 128, optical_sar 50, unsupported 105, vqa 87. The validation split used for the reported accuracy is n = 86; the corpus is flagged corpus_limited: true. The adapter has 51,725 parameters over a 384-dim frozen MiniLM encoder (6 task classes, 4 modalities, 3 binary heads).

This is why the router number must be read as indicative only — see LIMITATIONS.md §1 (L-12) and §8 (L-67). The fix was validated by live behaviour (24 runs, 24 correct dispatches), not by a corpus accuracy jump, precisely because the corpus is small.

Recorded in. artifacts/router/router_adapter_v001/metadata.json; artifacts/router/threshold_sweep_val.json.

---

5. The harness false-positive — caught before it could lie (RESOLVED)

5.1 What happened

An earlier live-validation harness typed queries with synthetic CDP key events, which Chrome silently drops when the window lacks OS focus. The harness therefore dispatched the page's default query and still recorded a "result" — a false pass. Measured directly: with Chrome backgrounded, press_key("Z") left #qtext.value unchanged, while type_text("Q") (CDP Input.insertText, not focus-gated) inserted fine. The re-run attempt failed on case 1 with run_id=0002, mock_nodes=9, answer="No answer yet", and only a capabilities call — the mock path.

5.2 Fix

The current harness asserts form state before dispatch (q_ok, obs_ok, t0_ok) and uses deterministic query entry (js() value-set + type_text() via CDP Input.insertText). Two further harness bugs were found and fixed, both causing false failures:

• the answer [task] tag exists only for region tasks (vqa/caption answers are bare); and
• the intent panel is a concatenated string, so task([a-z_]+) must be matched non-greedily up to modality.

The harness now computes the dispatched task as answer_tag when present, else the reading.

5.3 Independent check

The earlier 8/8 run was re-examined and confirmed not infected — its intents were query-specific (e.g. A1 read taskvqa…temporalnone, not the default's taskchange…temporalrequired), its answers embedded the query text, and A6's answer proved two files were uploaded. The failure mode is recorded because it is exactly the silent false-positive an evaluation harness must never have.

Recorded in. .workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md ("Why this pass needed a new harness"); docs/FINAL_DELIVERY_TODO.md §6 E-14.

---

6. The transport_mode: auto fallthrough (OPEN)

SATQUERY_TRANSPORT=auto tries the tunnel, then falls through to the forward path on timeout. The forward path to a private repo returns 302 quickly, but the wake step still consumes SATQUERY_WAKE_TIMEOUT_S (120 s) first — so a worst-case failed request takes ≈ 249 s (150 + 120). This is the root shape of the observed transient tunnel gap.

The recorded root cause is precise: in auto transport mode a tunnel timeout falls through to the forward path (SatQuery-Backend/main.py:546), which then burns wake_timeout_s=120 on a 302 → the observed 504.

A patch (fix-b07-forward-unavailable.patch) was authored and verified (git apply --check clean, py_compile clean, applies to the deployed 89d80eaddec5). It adds forward_unavailable (503, terminal 302/401/403 on the forward path) and upstream_timeout (504, tunnel healthy but slow) codes, plus the codespace_name .strip() fix.

Status: OPEN. The patch is prepared but NOT deployed. The deployed health payload still shows the trailing \n.

Recorded in. release/CURRENT_RELEASE_STATE.md §6; docs/FINAL_DELIVERY_TODO.md §5 B-07; DEPLOYMENT.md §8.1.

---

7. The interpret() / chooseTask() asymmetry — intentional (RESOLVED)

For "What changed between the earlier and later image?" with one asset attached, the console reads change while dispatch correctly falls back to change_vqa. This is not a bug:

• the reading describes the question's intent (asset-count-blind);
• the dispatch respects what can actually be computed with the assets present (asset-count-aware).

It is documented so it is not mistaken for a defect. In the live-validation table this appears as the one case where reading ≠ dispatched (change → change_vqa) — "flagged, not failed".

Recorded in. .workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md ("Discriminator note"); architecture/04-router.md.

---

8. Environment findings (would otherwise cost hours)

Finding	Detail
Dead proxy in the authoring sandbox	outbound calls need --noproxy '*' (curl) or ProxyHandler({}) (Python).
The sandbox proxy is slow for uploads	huggingface_hub uploads stalled at ~51 kB/s through http_proxy=127.0.0.1:58294; the fix is to unset http_proxy/https_proxy and set no_proxy='*'.
hf_hub_download returned an EMPTY file	sha256 e3b0c442… (the empty-content hash), which produced a false FAIL for all six artifacts. The honest check is a direct HTTPS download with ProxyHandler({}).
pytest is only in the repo venv	.venv/Scripts/python.exe; a bare pytest misses it.
The full test suite trips a bulk-delete guard	sandbox-specific; affects test_safe_delete_shim.
Cloudflare _headers concatenate	two matching rules are merged, not overridden; Chromium takes the first max-age.
Cloudflare 308-redirects X.html → /X	reference the extensionless path.
A forwarded Codespace port returns 302	for a private repo — this is why the tunnel exists.
Chrome drops synthetic CDP key events without OS focus	the harness false-positive (§5).
browser-use block-buffers stdout	even when redirected; needs explicit line buffering to stream.
WDAC blocked orjson, so FastAPI could not import (historical — no longer reproduces)	ImportError: DLL load failed while importing orjson: An Application Control policy has blocked this file. Verified as OSError [WinError 4551] from a direct ctypes.CDLL on the binary. FastAPI's own guard catches ModuleNotFoundError, not a policy-blocked ImportError, so it aborted the import. When the block lifted, four live defects became reachable that had been sitting in gateway/ the whole time (G-1…G-4). Do not cite this as a current limitation.
pandas._libs.parsers / sparsefuncs_fast App Control block	56 evaluation-side tests remain uncollectable. Unchanged; unrelated to the backend chain.
CPU-only torch	CUDA autocast is a no-op; no GPU path was exercised in the authoring environment.

8.1 A withdrawn explanation, recorded rather than deleted

The reason that nine backend-chain tests (D–I, Q, R) were NOT RUN was originally recorded as "the environment's egress proxy intercepts outbound HTTP and returns 502". That explanation was re-measured on 2026-09-22 and does not hold: https://huggingface.co returned HTTP 200, /api/models returned 200 with real JSON, and a real weight-file path returned 200. The correct reason is that no upstream exists — no Space and no service has ever been deployed, and the gateway is a pure proxy that holds no second copy of the capability table, so a metadata request is an upstream request. The classification (NOT RUN / ENVIRONMENT-BLOCKED) is unchanged; only the reason changed.

8.2 The transferable lesson from the stale-negative class

Three documents written after the Phase-12 A/B experiment still described it as un-run, even though it had completed 34 hours earlier. A stale negative claim is more dangerous than a stale positive one: a wrong number is eventually contradicted by the artifact it describes, but "this has never been executed" is contradicted by nothing — no test fails, no hash moves, no invariant breaks. The rule this suggests: never assert that an artifact does not exist from memory; assert it from a command, and record the command. No test can check that a documented absence is still absent.

Recorded in. docs/PHASE19_FINAL_HARDENING.md §5 (historical blocker) and §5.3 (CPU-only torch); docs/STEP7_BACKEND_CHAIN_REPORT.md §5, §11, §13, §15; docs/PHASE12_CURRENCY_CORRECTION.md §6; REPRODUCIBILITY.md.

---

9. The BigEarthNet format contradiction (ATTEMPTED, reported not resolved)

The BigEarthNet data format contradicts the original plan. This was reported rather than silently patched, because quietly changing the preprocessing would move the frozen config hash. Two facts matter:

1. The percentile stretch is not upstream. The BigEarthNet documentation — its uses, mentions, or endorsements — does not specify a percentile stretch. This project nevertheless applies percentile normalisation (2/98) for optical inputs to match the CROMA contract. That is a deliberate, documented choice, not an upstream fact.
2. The local subset is single-label. It is 100 % single-label against the official 1–11 multi-label scheme, so metrics computed on it are not comparable to published multi-label numbers.

Consequence. The contradiction is recorded as a limitation and an open item rather than resolved by editing preprocessing (which would move Config.hash off 78f1e3700da15aa1).

Recorded in. LIMITATIONS.md §3 (L-24); docs/PHASE12_LABEL_POLICY_DECISION.md; docs/PHASE14_CROMA_NORMALISATION_CHANGE.md.

---

10. Additional measured defects that shaped the design

These are not headline findings, but each changed a decision or a guard. They are recorded together because they share a shape: a value or a claim that was produced without a measurement, then corrected by one.

10.1 The train/serve skew (the F2 finding) (RESOLVED)

What was measured. scripts/prepare_change_vqa.py builds its change features from the trained STANet (DEFAULT_CHANGE_CHECKPOINT), but serving had no equivalent wiring: change.checkpoint_path is unset in configs/base.yaml, so the registry passed no checkpoint_path and the change-VQA specialist would construct an untrained STANet and answer from a representation the head was never fitted on.

Consequence. app/serving.py::_wired_change_vqa_builder supplies the same checkpoint the change builder uses, so the detector behind a change answer and the detector behind the head's training features are one artifact by construction. The specialist's own feature_spec_mismatch() check stays armed as a second line of defence, not the only one.

Recorded in. app/serving.py (_wired_change_vqa_builder docstring).

10.2 The optical-SAR encoder was unreachable by default (RESOLVED)

What was measured. specialists/optical_sar/specialist.py builds CROMA only when handed a checkpoint_path that exists. croma.checkpoint_path is not in configs/base.yaml — and must not be, or Config.hash moves — so the registry passed no path, the gate was False, and the default serving composition ran with encoder=None while the checkpoint sat on disk the whole time. The registry then correctly reported DEGRADED ("no encoder; running on fallback"): a deployment that could never answer an optical/SAR question.

Consequence. app/serving.py::_wired_optical_sar_builder resolves the checkpoint from the pinned identity through the hash-exempt channel (croma.resolve_checkpoint_path: env → config → pinned Hub cache, offline first) and wires the fusion head the same way. The resolution source is logged so "which checkpoint did this process actually use" is answerable from the trace.

Recorded in. app/serving.py (_wired_optical_sar_builder docstring); specialists/optical_sar/specialist.py.

10.3 G-1 … G-4 — four defects the first real ASGI run found (RESOLVED)

What was measured. The ASGI layer had never executed in the repository, and the reason was recorded in the code as an environment blocker (a WDAC block on orjson.pyd). When the block lifted, within the first hour of actually running the server four real defects surfaced, four of them latent for as long as the blocker was believed:

ID	Defect	Severity
G-1	request: Request never resolved — an in-function import left Request out of __globals__, so FastAPI silently reinterpreted the parameter as a required query parameter named request; every POST body was misread and no handler ran	Critical
G-2	an unsupported force_task enum value was forwarded upstream instead of refused locally → wrong status, wasted round trip	High
G-3	an empty HF_TOKEN produced Authorization: Bearer , which httpx rejects → a crash reported as an upstream failure	High
G-4	a non-JSON upstream error body was relayed verbatim → contract break and internal-text disclosure	High

Consequence. All four fixed, with regression tests that assert the cause rather than the symptom. The transferable lesson: "a documented blocker is a place where evidence stops, and nothing was watching for the blocker to lift."

Recorded in. docs/STEP7_BACKEND_CHAIN_REPORT.md §13, §14.

10.4 F-1 … F-7 — the gateway layer-crossing audit (RESOLVED / recorded)

What was measured. A series of findings, each found by driving the real ASGI stack rather than by reading:

ID	Finding	Disposition
F-1	a vacuous assertion in the gateway's own test suite	fixed
F-2	an upstream CORS header bypassed the allowlist entirely (security)	fixed
F-3	404/405 did not carry the contract's error envelope	fixed (gateway); app/space_app.py does not register the handler (bounded defect F-12b)
F-4	rate_limited was emitted but documented nowhere	fixed
F-5	the per-IP rate limiter is defeated by a client-supplied X-Forwarded-For (measured 5/8 throttled without the header, 0/8 with a fresh value per request)	ruled: fairness only, not a security control
F-6	the body-size cap was declarative, not enforced — an omitted Content-Length buffered the entire body past the cap (measured: 12 MiB body → 13.9 MiB peak; allocation tracked body size with no ceiling)	fixed with a streaming check alongside the header check
F-7	two parsers of one variable (SATQUERY_MAX_FILE_BYTES) diverged in opposite directions	fixed; both layers now refuse a malformed value and name the variable

Consequence. The rate limiter's claim was narrowed to what the measurement supports, and the size cap is now enforced twice — deliberately, because "the header check protects the gateway's memory against honest clients, not against hostile ones".

Recorded in. docs/STEP8_FINAL_CONFORMANCE_AUDIT.md §15–§17; docs/DEPLOYMENT_ARCHITECTURE.md §5.2, §4 (F-6 note).

10.5 F-13 … F-17 — what an unauthenticated client could learn (RESOLVED / recorded)

What was measured. A pass-by-pass audit of every value written to a client-visible field found that trace.inputs (F-13), trace.steps[PARSE].detail["inputs"] (F-14), and the construction-failure exception string (F-15) published server-side filesystem paths to an unauthenticated client — F-15's case was "strictly worse" because the path was purely server-side and reachable on the first request. F-16 found result.change_map / evidence[].artifact_ref carrying a real filesystem path where the contract promised an artifact:// URI. F-17 found change_vqa.artifact_dir configured but never read.

Consequence. The owner ruled: sanitize client-facing exception messages to a basename (not a replacement — the reason survives) and log the raw detail server-side; set an unavailable artifact_ref to null with an explicit non-retrievable warning; do not fabricate artifact:// URIs. The F-15 ruling needed four carriers, not three, because result.execution_trace = trace serializes the trace object twice. F-17 is documented, not patched.

Recorded in. docs/DEPLOYMENT_ARCHITECTURE.md §5.3–§5.6; SECURITY.md.

10.6 The single capability authority (RESOLVED)

What was measured. Two independent producers answered "what can this deployment do?": AnalysisController.health() reported 6 capabilities from the registry; space_app.describe_deployment() reported 2 from two Path.exists() calls — and only the latter was served. Also measured: HealthStatus(**controller.health()) fails with 3 extra_forbidden errors.

Consequence. Owner ruling: the registry is authoritative, and app/deployment.py is the single adapter that derives contract vocabulary from the registry's spec table and the filesystem. This is the "three-copy problem": any table that exists in two places will drift, and this one already had.

Recorded in. docs/DEPLOYMENT_ARCHITECTURE.md §3.3.1; docs/STEP7_BACKEND_CHAIN_REPORT.md §16.

10.7 T-1 / T-2 — two test-side findings worth carrying (RESOLVED)

What was measured. (T-1) An earlier note recorded "22 documented error codes"; the real taxonomy and the contract both have 23. The test now parses the contract's §5.2 table and compares it against core/errors.py, so the two must agree. (T-2) Three documentation tests initially failed by matching the prohibition itself (e.g. a search for gr.Blocks matched the docstring that forbids it); the tests now parse the module with ast and strip docstrings before searching — "a search for a forbidden token must run over executable code".

Consequence. Both corrections became bidirectional checks rather than one-off fixes.

Recorded in. docs/STEP7_BACKEND_CHAIN_REPORT.md §3, §13; docs/PHASE19_FINAL_HARDENING.md §3.7.

10.8 The calibration measurement, stated exactly (MEASURED)

What was measured. Temperature scaling was fitted on the Val split: temperature = 0.9772731820958189, n_samples = 16441, ece_before = 0.013755, ece_after = 0.014929, ece_improvement = −0.001174 (worse), nll_before = 0.6897411, nll_after = 0.6896308, nll_improvement = +0.0001104, hit_bound = false, effective = true. The raw softmax was already near-calibrated, and temperature scaling made ECE very slightly worse while improving NLL marginally.

Consequence. The path is live (core/controller.py → EvidenceEngine), so a deployed result carries a calibrated value; but this is a measurement, not a quality judgment, and must not be described as scaling being "more accurate".

Recorded in. artifacts/calibration_v001.json; docs/STEP7_BACKEND_CHAIN_REPORT.md §7.

---

11. Where the evidence lives

Topic	Evidence
Findings F4-1…F4-3	docs/PHASE4_ROUTER_REPORT.md
Findings F5-1…F5-3	docs/PHASE5_VLM_CONTRACT.md
Finding P7-1	docs/PHASE7_GROUNDING_CONTRACT.md
Findings C-1, C-6…C-9	docs/ARCHITECTURE_FREEZE.md
CROMA normalisation upstream evidence	docs/CROMA_NORMALISATION_UPSTREAM_EVIDENCE.md
The 448-vs-224 paired test	docs/PHASE7_RESOLUTION_DECISION.md
The VLM rejection	docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md, artifacts/vlm/phase6_closure.json
Router defect + 3 live passes	.workbuddy-ai/scratch/live_validation/ (run_output.txt, run_final2.txt, run_final3.txt)
The undeployed B-07 patch	session scratch fix-b07-forward-unavailable.patch
Live validation harness	.workbuddy-ai/scratch/run_all_postfix2.harness, recompute_verdicts.py
The withdrawn egress explanation	docs/STEP7_BACKEND_CHAIN_REPORT.md §5
The stale-negative lesson	docs/PHASE12_CURRENCY_CORRECTION.md §6
The BigEarthNet contradiction	docs/PHASE12_LABEL_POLICY_DECISION.md, docs/PHASE14_CROMA_NORMALISATION_CHANGE.md
The gateway layer-crossing findings (F-1…F-7)	docs/STEP8_FINAL_CONFORMANCE_AUDIT.md §15–§17
The ASGI defects (G-1…G-4)	docs/STEP7_BACKEND_CHAIN_REPORT.md §13
The path-disclosure rulings (F-13…F-17)	docs/DEPLOYMENT_ARCHITECTURE.md §5.3–§5.6
The train/serve skew and optical-SAR wiring	app/serving.py
The calibration measurement	artifacts/calibration_v001.json, docs/STEP7_BACKEND_CHAIN_REPORT.md §7