docs/REPRODUCIBILITY.md

Reproducibility

Status tags: IMPLEMENTED · VERIFIED · MEASURED · ATTEMPTED · NOT RUN · BLOCKED · DEFERRED · REJECTED · OPEN · RESOLVED · CLOSED.

This document states exactly what a third party can reproduce, with what, and — just as importantly — what they cannot. It is written to the same depth as the rest of this documentation set: real commands, real file paths, real hashes, real expected output, and an explicit status for every claim.

It is deliberately not a "quickstart". A quickstart tells you the happy path and hides the edges. This document records the edges, because in this project the edges are where the real debugging time went — a dead sandbox proxy, a pytest that only exists in one virtualenv, a Chrome that silently drops synthetic key events, a Hugging Face helper that returned an empty file and would have produced a false failure. Those are documented in §10 so that a third party does not rediscover them.

The one rule this document obeys, like every document in docs/: do not fabricate. Every claim below is grounded in a file that was read (core/config.py, configs/base.yaml, release/tools/*.py, artifacts/**, docs/**). Where the evidence does not establish something, the text says UNKNOWN — not established from the available evidence.

---

Table of contents

1. The reproducibility contract
2. The frozen configuration
3. Reproduce the metric verification (cheap, no GPU)
4. Reproduce the environment
5. Reproduce the tests
6. Reproduce a live run
7. Reproduce and verify the released artifacts
8. What "reproduce" means for the two externally-trained artifacts
9. What is NOT reproducible from this release — exhaustive
10. Environment traps recorded for reproducibility
11. What is NOT RUN / OPEN / BLOCKED for this topic
12. Where the evidence lives

---

1. The reproducibility contract

Reproducibility here is not a slogan; it is a set of enforced guarantees. Each row below states a guarantee, how it is enforced in code or in a tool, and how a third party can independently check it.

Guarantee	How it is enforced	How to check it
Frozen configuration	All tunables live in configs/base.yaml; there are no magic numbers in Python. The loader validates invariants and computes a hash.	Read configs/base.yaml; run the loader; compare the hash.
Frozen config hash	Config.hash = sha256(json.dumps(data, sort_keys=True, default=str))[:16] (core/config.py:76-80). Every evaluation run and every artifact records the hash it was produced against.	Config.hash returns 78f1e3700da15aa1; the same string appears in models/manifest.json and each artifact.
Pinned backbones	Every backbone is pinned by repo_id and revision (a commit prefix, not a floating tag) in configs/base.yaml.	The four pins are listed in §4.3; each resolves to an exact Hub commit.
Seed	project.seed: 42 (core/config.py:82-84).	Read the key; it is in the registry.
Immutable public test	evaluation.immutable_public_test: true and evaluation.hidden_data_access: false (configs/base.yaml:273-277).	Read the two keys; the corpus module refuses to mutate a sealed corpus.
Leakage isolation	evaluation.leakage_split_key: scene_id — splits are assigned by scene, never by tile.	Read the key; evaluation/leakage.py audits manifests.
No official aggregate	evaluation.official_aggregate_weights: null. There is no weighted composite score.	Read the key; normalize.py raises on aggregate requests.
Byte-verified artifacts	Every released artifact ships with a sha256 in models/checksums.sha256, generated from the actual files.	sha256sum -c models/checksums.sha256.
Verified metrics	Every number quoted in the README is checked against its source artifact by release/tools/verify_readme_metrics.py.	Run the tool; it prints ALL CLAIMS VERIFIED.
Independent Hub verification	release/tools/hf_verify.py re-downloads every uploaded file over direct HTTPS and re-hashes the bytes it received.	Run the tool; it prints artifacts failed : 0.
Negative results preserved	Rejected and open rulings stay in the record (PHASE7_RESOLUTION_DECISION.md, phase6_closure.json, threshold_sweep_val.json).	Read the artifacts; they contain the rejections, not scrubbed versions.

1.1 What the contract is not

The contract does not promise:

• a single command that retrains every artifact from scratch (§8);
• an end-to-end system-level accuracy number (§9 — no such benchmark exists);
• that the two externally-trained artifacts can be regenerated inside this repository (§8);
• bit-identical training on arbitrary hardware (only bit-identical released artifacts are guaranteed, by hash).

1.2 The reproducibility ladder

Reproduction here comes in four tiers, from cheapest to most expensive. A reviewer who wants to spend five minutes gets tier 1; a reviewer who wants to audit the whole thing walks all four.

Tier	What it proves	Cost	Section
1. Metric verification	Every quoted number matches its artifact	seconds, CPU	§3
2. Environment + tests	The code installs and the suites behave as recorded	minutes, CPU	§4–§5
3. Live run	The deployed stack answers correctly on unseen imagery	minutes (plus cold start), network	§6
4. Artifact + Hub verification	The released weights are byte-identical to what was uploaded	minutes, network	§7

---

2. The frozen configuration

2.1 What "frozen" means here

configs/base.yaml is the single registry for every tunable in the system. The design rule is: no magic numbers in Python. If a value can change behaviour — an image size, a channel count, a threshold, a precision, a split key — it lives in the registry, not in a function body.

This matters for reproducibility because it means the entire behavioural surface of the system is captured by one file, and that file has a hash.

2.2 The hash and how it is computed

@property
def hash(self) -> str:
    """Stable hash of the whole registry. Recorded in every evaluation run."""
    blob = json.dumps(self._data, sort_keys=True, default=str).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

— core/config.py:76-80.

Three properties make this hash stable:

1. sort_keys=True — key order in the YAML file does not change the hash.
2. default=str — non-JSON-native values (dates, paths) are stringified deterministically.
3. [:16] — the first 16 hex characters are used as the human-readable identifier.

The frozen value is:

78f1e3700da15aa1

It is recorded in models/manifest.json ("config_hash": "78f1e3700da15aa1"), in each artifact's metadata, and in the live evidence (each live run's panel shows the hash). This is the single identifier that ties "the thing that produced this number" to "the configuration a reviewer is reading".

2.3 The enforced invariants, with exact arithmetic

Editing configs/base.yaml moves the hash and invalidates every artifact keyed to it. To make that failure loud rather than silent, the loader rejects a config that violates a set of recorded invariants. Two of them exist specifically because their violation is a silent error that torch only surfaces much later.

Invariant 1 — fusion input dimension (finding C-1)

fusion.input_dim == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels

With the frozen values encoder_dim = 768, optical_channels = 12, sar_channels = 2:

fusion.input_dim == 3 * 768 + 12 + 2
                 == 2304      + 12 + 2
                 == 2318

This is the concatenation of CROMA's three 768-d GAP vectors (optical_GAP, SAR_GAP, joint_GAP) plus the 12 optical channels and 2 SAR channels of the sensor adapter. The loader recomputes expected from the config and raises ConfigError if fusion.input_dim disagrees (core/config.py:154-163). The docstring is explicit about why: "CROMA emits optical/SAR/joint GAP vectors; the [declared value must match]". The upstream contract was independently reproduced in docs/PHASE14_OPTICAL_SAR_DECISIONS.md §1, where the fusion input is recomputed at runtime (fusion_head.py:74, :311) rather than hardcoded.

Two related guards fire alongside it (core/config.py:165-168):

croma.optical_channels == 12   # CROMA s2_channels is fixed
croma.sar_channels     == 2    # CROMA s1_channels is fixed

Invariant 2 — grounding head feature dimension (finding P7-1)

grounding_head.feature_dim == 4 * grounding.encoder_projected_dim

With encoder_projected_dim = 512:

grounding_head.feature_dim == 4 * 512 == 2048

The loader's own comment explains why this is load-bearing:

A mismatch here is a SILENT shape error. torch only raises at the [similarity step], after patch features are already cached. It is therefore rejected at load time.

— core/config.py:176-195.

The "4×" arises because the grounding head consumes four projected feature streams; the projected dimension is declared in config precisely so this guard can exist without hardcoding 512.

Further load-time guards

The _validate() method (core/config.py:94) also enforces:

Guard	Rule	Finding
CROMA resolution	croma.image_resolution is an int and a multiple of 8 (CROMA asserts % 8 == 0)	C-7
Training precision	training.precision ∈ {fp16, bf16, fp32}	C-6
ZeroGPU compile ban	deployment.torch_compile must not be True — ZeroGPU does not support torch.compile	C-8
VLM prompt template	vlm.prompt_must_use_chat_template must be true — SmolVLM raises otherwise	—

These are the invariants a reviewer can verify by reading one method. They are the reason a plausible-looking config edit fails at load instead of producing a wrong number three hours later.

2.4 The evaluation invariants

evaluation:
  immutable_public_test: true
  hidden_data_access: false
  official_aggregate_weights: null
  leakage_split_key: scene_id

— configs/base.yaml:273-277.

Key	Value	Meaning
immutable_public_test	true	The public-test corpus is sealed; it cannot be mutated by an evaluation.
hidden_data_access	false	No code path may read hidden data during a scored run.
official_aggregate_weights	null	There is no composite score. A single "overall accuracy" is forbidden by construction.
leakage_split_key	scene_id	Splits are assigned by scene, so tiles from one scene never straddle train/test.

The evaluation-honesty rules that these keys serve are stated in full in EVALUATION.md §2.

2.5 The deployment section is frozen paperwork

deployment:
  platform: huggingface-spaces
  sdk: gradio
  zerogpu: true
  torch_compile: false
  ...

— configs/base.yaml:282-295.

The deployment target described here is the superseded HF-Space/ZeroGPU design. The active topology is Cloudflare Pages → Render → Codespace (docs/DEPLOYMENT.md §1). This section is retained as frozen paperwork: editing it would move the config hash, and the hash is what every artifact is keyed to. No Gradio runtime exists in code. The torch_compile: false line is the C-8 guard and remains meaningful regardless of host.

2.6 The hash-exempt path

One decision (DEV-2, the CROMA input-normalisation question) needed to become inspectable without adding a base.yaml key — because adding a key would have moved the hash away from 78f1e3700da15aa1 and invalidated every artifact. The resolution was a hash-exempt path (docs/PHASE14_OPTICAL_SAR_DECISIONS.md, item 7):

SATQUERY_CROMA_USE_8_BIT (env)  →  croma.use_8_bit (config)  →  default True

It is inspectable at runtime via CROMAEncoder.describe()["input_normalisation"]. This is recorded because it is the one place where "frozen" and "configurable" were reconciled deliberately, and a reviewer should know the escape hatch exists and is documented rather than hidden.

2.7 What invalidates the hash

Anything that changes self._data when serialised with sort_keys=True. Concretely:

• editing any value in configs/base.yaml;
• adding or removing a key;
• changing a value's type such that its default=str rendering changes.

The consequence is not merely a different hash; it is that every artifact keyed to 78f1e3700da15aa1 no longer describes the running configuration, and the honest action is to re-run the evaluation rather than reinterpret the old numbers.

---

3. Reproduce the metric verification (cheap, no GPU)

This is the cheapest, highest-value reproduction in the whole release. It takes seconds, needs no GPU and no network, and it is the check that a skeptical reviewer should run first.

3.1 The verifier

python release/tools/verify_readme_metrics.py

release/tools/verify_readme_metrics.py is read-only. Its own docstring states its contract:

Verify every metric quoted in the release README against its source artifact. Read-only. Each claim is compared at the precision at which the README states it.

The tool is not a re-computation of the metrics. It is a provenance check: it reads the artifact that the README cites, resolves a dotted key into it, and compares the value against the literal string the README prints — at the README's own precision.

3.2 The 20 claims and their exact key paths

The claim list is not assumed; the key names were discovered by walking the artifacts. Several live under nested paths, and the tool's docstring says so:

Key names were discovered by walking the artifacts, NOT assumed: several live under nested paths (e.g. change metrics are metrics.pooled.iou, grounding is results.head_threshold.mean_best_iou, VLM is why_usable_verified.adapted_test.*).

#	Claim	Artifact	Dotted key	README value
1	change pooled IoU	artifacts/change/eval_test/eval_result.json	metrics.pooled.iou	0.8122
2	change macro IoU	artifacts/change/eval_test/eval_result.json	metrics.macro.miou	0.8457
3	change pooled F1	artifacts/change/eval_test/eval_result.json	metrics.pooled.f1	0.8964
4	grounding canonical head_threshold mean_best_IoU	artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json	results.head_threshold.mean_best_iou	0.2838
5	grounding canonical head_threshold recall@0.5	…/eval_result_canonical.json	results.head_threshold.recall.0.50	0.2198
6	grounding matched6 head_threshold mean_best_IoU	…/eval_result_matched6.json	results.head_threshold.mean_best_iou	0.2566
7	grounding matched6 head_threshold recall@0.5	…/eval_result_matched6.json	results.head_threshold.recall.0.50	0.1938
8	grounding head_argmax mean_best_IoU (canonical)	…/eval_result_canonical.json	results.head_argmax.mean_best_iou	0.1215
9	grounding zero-shot baseline IoU (canonical)	…/eval_result_canonical.json	results.zero_shot_matched.mean_best_iou	0.0972
10	optical-SAR fusion accuracy	artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json	accuracy	0.931
11	optical-SAR fusion macro_F1	…/pre_registered_115_metric.json	macro_f1	0.434161
12	change_vqa test accuracy	artifacts/change_vqa/run/PROMOTION.json	verification.test_accuracy	0.697626
13	change_vqa test macro_F1	artifacts/change_vqa/run/PROMOTION.json	verification.test_macro_f1	0.378373
14	change_vqa test2 accuracy	artifacts/change_vqa/run/PROMOTION.json	verification.test2_accuracy	0.651469
15	change_vqa test2 macro_F1	artifacts/change_vqa/run/PROMOTION.json	verification.test2_macro_f1	0.372309
16	router overall ungated accuracy	artifacts/router/threshold_sweep_val.json	overall_ungated_accuracy	0.965116
17	calibration ECE before scaling	artifacts/calibration_v001.json	metrics.ece_before	0.013755
18	calibration ECE after scaling	artifacts/calibration_v001.json	metrics.ece_after	0.014929
19	VLM adapter exact_match	artifacts/vlm/phase6_closure.json	why_usable_verified.adapted_test.exact_match	0.963
20	VLM adapter F1	artifacts/vlm/phase6_closure.json	why_usable_verified.adapted_test.f1	0.96432

Note rows 1–3, 6–7, 12–15, and 16: these are the rows that keep the honesty rules honest. The change numbers are quoted as pooled and macro together (rule 4); grounding is quoted under both protocols (rule 2); change-VQA is quoted under both test sets (rule 3); the router number is quoted with its validation-only caveat (rule 5). A verifier that checked only one number per task would be reproducible and wrong.

3.3 The status assertions

Metrics are not the only claims. After the 20 numeric comparisons, the tool asserts four statuses that the README also states (verify_readme_metrics.py:117-131):

VLM headline contains ACCEPTANCE-REJECTED : True
VLM status                               : CLOSED
router corpus_limited                    : True
router n_val                             : 86
calibration temperature (temperature_scaling.temperature) : 0.9772731820958189
calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)

These are the checks that prevent the documentation from quietly upgrading a status. In particular:

• the VLM metric is usable but the artifact's own headline contains ACCEPTANCE-REJECTED — so USABLE ≠ ACCEPTED (rule 7);
• the router number is flagged corpus_limited: true with n_val: 86 — so it can never be presented as a test number (rule 5);
• the calibration ece_improvement is negative, i.e. calibration made ECE worse, and the tool prints that interpretation inline (rule 6).

3.4 The nested-key resolution trick

Two subtleties in the resolver are worth recording, because they are the kind of thing that turns a verifier into a false-pass machine if done naively (verify_readme_metrics.py:69-90):

1. Keys that themselves contain dots. The grounding recall dictionary is keyed by the string "0.10", "0.25", "0.50". A naive dotted.split(".") walk would break results.head_threshold.recall.0.50 into …recall → 0 → 50 and fail. The resolver therefore tries the longest matching key at each step first, so 0.50 stays intact.
2. Precision matching. The comparison is round(float(val), decimals(claimed)) == float(claimed) — i.e. the artifact value is rounded to the number of decimals the README prints, then compared. This is why row 15 shows the artifact value 0.372308516 matching the README's 0.372309: the README states six decimals and the artifact rounds to it.

A verifier that compared raw floats would report false differences on every rounded value. A verifier that split dots naively would report false differences on the recall rows. Both failure modes are silent false failures — the same class of bug discussed in §7.3 and §10.

3.5 The committed output

STATUS   claim                                                  artifact       readme  source
----------------------------------------------------------------------------------------------------------------------
MATCH    change pooled IoU                                        0.8122       0.8122  artifacts/change/eval_test/eval_result.json#metrics.pooled.iou
MATCH    change macro IoU                                         0.8457       0.8457  artifacts/change/eval_test/eval_result.json#metrics.macro.miou
MATCH    change pooled F1                                         0.8964       0.8964  artifacts/change/eval_test/eval_result.json#metrics.pooled.f1
MATCH    grounding canonical head_threshold mean_best_IoU         0.2838       0.2838  artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json#results.head_threshold.mean_best_iou
...
MATCH    VLM adapter exact_match                                   0.963        0.963  artifacts/vlm/phase6_closure.json#why_usable_verified.adapted_test.exact_match
MATCH    VLM adapter F1                                          0.96432      0.96432  artifacts/vlm/phase6_closure.json#why_usable_verified.adapted_test.f1

=== status assertions ===
  VLM headline contains ACCEPTANCE-REJECTED : True
  VLM status                               : CLOSED
  router corpus_limited                    : True
  router n_val                             : 86
  calibration temperature (temperature_scaling.temperature) : 0.9772731820958189
  calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)

RESULT: ALL CLAIMS VERIFIED

— release/tools/readme_metrics_report.txt (committed).

The tool exits 0 and prints ALL CLAIMS VERIFIED only when every numeric comparison is MATCH and no artifact is missing or lacks the key. Any NOFILE, NOKEY, or DIFFER line increments the failure counter, the final line reads N CLAIM(S) FAILED, and the process exits 1. This makes the check usable in CI: a non-zero exit is a hard failure.

3.6 Why this is a real check and not a tautology

A skeptic might object: "you wrote the README from the artifacts, so of course they match." The answer is that the check is not that the README was derived from the artifacts, but that the README's published claims remain mechanically traceable to files a third party also has. If someone edits the README to state a nicer number, the tool fails. If someone edits an artifact to a different number, the tool fails. The check binds the published prose to the recorded evidence, and it does so without trusting either.

---

4. Reproduce the environment

4.1 What you need

• Python 3.11+ (README.md §Installation: "Python 3.11+ and a CPU are sufficient").
• A CPU is enough. There is no CUDA requirement.
• Network access on first use, to fetch the pinned backbones from the Hugging Face Hub.

4.2 The two install profiles

requirements.txt documents two profiles explicitly:

#   CPU (local dev / schema / geo / unit tests):
#       pip install -r requirements.txt
#   GPU (Kaggle T4x2 / HF ZeroGPU): torch is preinstalled on both.
#       Do NOT pin torch here — Kaggle and HF ship their own builds.

— requirements.txt (header comment).

The pins, with the contract each one carries (requirements.txt):

Package	Pin	Why
numpy	>=1.26,<3	core
pyyaml	>=6.0	config loading
pydantic	>=2.6,<3	schema
rasterio	>=1.3	geospatial IO
pyproj	>=3.6	CRS handling
opencv-python-headless	>=4.9	image ops
transformers	>=4.52	SmolVLM via AutoModelForImageTextToText; AutoModelForVision2Seq is deprecated (finding C-2)
open-clip-torch	>=2.24	RemoteCLIP loaded via pretrained=<path> so load_checkpoint() runs its state-dict fixups (C-4)
sentence-transformers	>=2.7	router embedding
peft	>=0.10	LoRA adapter
huggingface_hub	>=0.23	pin RemoteCLIP / SmolVLM / CROMA revisions
safetensors	>=0.4	adapter serialisation
einops	>=0.7	required by the vendored use_croma.py (from einops import rearrange)
gradio	>=4.44	UI (frozen deployment target)
reportlab	>=4.1	reporting
pytest	>=8.0	tests
pytest-cov	>=5.0	coverage

The comment block closes with an instruction a reviewer should heed: "do not 'fix' by changing these pins" — the pins encode verified contracts, not preferences.

4.3 The einops lesson — a dependency no document declared

requirements.txt carries an unusually explicit note about einops:

REQUIRED by the vendored specialists/optical_sar/vendor/use_croma.py (from einops import rearrange). Found missing on 2026-09-18 by executing the vendored module: it raised ModuleNotFoundError, so the CROMA path was blocked on a dependency that no document declared. Not optional: without it the encoder cannot be imported at all.

This is recorded verbatim because it is a reproducibility fact: a missing dependency that no document mentioned was found by executing the code, not by reading a spec. A third party building the environment will hit the same wall if they install from a partial list.

4.4 Create the environment

git clone https://github.com/Anish-lab-blip/SatQuery-AI
cd SatQuery-AI
python -m venv .venv
source .venv/Scripts/activate      # Windows git-bash; use .venv/bin/activate on Linux/macOS
pip install -r requirements.txt

— README.md §Installation.

Trap (Windows, recorded in §10.2). In the authoring environment, pytest exists only in the repository virtualenv. Invoking the system pytest fails or resolves to a different interpreter. Always invoke the venv interpreter explicitly: .venv/Scripts/python.exe -m pytest ….

4.5 Backbones are fetched, pinned by revision, and never redistributed

The four third-party backbones are pinned by repo_id and revision (README.md §Reproducibility item 2; models/manifest.json per-artifact backbone fields):

Model	Revision	Role
HuggingFaceTB/SmolVLM-500M-Instruct	a7da5b986cb5	VQA + captioning backbone
chendelong/RemoteCLIP	bf1d8a3ccf2d	remote-sensing grounding encoder
sentence-transformers/all-MiniLM-L6-v2	1110a243fdf4	router embedding
antofuller/CROMA	0dd28e3d633b	optical/SAR fusion encoder

No backbone weights are redistributed in this release (release/HF_RELEASE_VERIFICATION.md §4). The released artifacts are small trained modules over these frozen backbones; anyone using them must fetch the backbones too. Pinning by commit prefix (not by a floating tag like main) is what makes "the same backbone" reproducible months later.

4.6 Device selection

Device is selected via SATQUERY_DEVICE, and all placement is .to(device), never .cuda() (README.md §Installation; core/config.py:86-91):

@property
def device_preference(self) -> str:
    override = os.environ.get("SATQUERY_DEVICE")
    if override:
        return override
    return "cuda" if _torch_cuda_available() else "cpu"

This is why a CPU-only reviewer can run everything except the training paths. The env override is read without importing torch in the deployment context (docs/DEPLOYMENT.md §3.2), so a misconfigured device cannot crash the service at import time.

---

5. Reproduce the tests

5.1 The suites and their expected results

Suite	Command	Expected	Status
Frontend live-wiring	python -m pytest tests/unit/test_frontend_live_wiring.py -q	106 passed	VERIFIED
Doc/frontend suite	pytest on the 5 doc/frontend files	183 passed	VERIFIED
Full unit suite	python -m pytest tests/unit	5–6 environmental/ordering failures, rest pass; a re-run passes 137	MEASURED

The three rows tell a story a single number would hide: the two targeted suites are clean, and the full suite has 5–6 failures that are environmental or ordering-related, not regressions. The next subsections reproduce each.

5.2 Frontend live-wiring regression suite — 106 passed

python -m pytest tests/unit/test_frontend_live_wiring.py -q

— README.md §Local development.

This suite is the regression net for the frontend↔backend wiring: the query box, the intent panel, the task tags, the run identifiers, and the evidence rendering. It is the suite that would catch a re-introduction of the router defect described in §6.4.

Expected: 106 passed. Note the historical figure, because a reviewer may see both numbers: docs/FINAL_DELIVERY_REPORT.md §7 records 94 passed for this file, while CURRENT_RELEASE_STATE.md:121 and README.md §The test suites record 106 passed ("re-run this session"). The difference is the suite growing after the delivery report was written — the regression tests added for the router fix took this file from 100 → 106 tests (DELIVERY_REPORT_2026-09-25.md, regression-tests note). This document quotes 106, the re-run-this-session figure, and cites both. (The exact count of tests collected in a single full-suite invocation is UNKNOWN — not established from the available evidence; the per-suite counts are what the evidence establishes.)

5.3 Doc/frontend suite — 183 passed

The documentation-and-frontend suite is five files, named explicitly in docs/FINAL_DELIVERY_REPORT.md §7:

• test_frontend_guide_doc
• test_frontend_live_wiring
• test_api_contract_doc
• test_runbook_doc
• test_deploy_config

Together they report 183 passed. These are the checks that the docs' internal links resolve, the frontend assets are present, the API contract document matches the code, the runbook is consistent, and the deploy config is coherent.

The same three-row test table is published in this release at README.md §The test suites (README.md:1063-1074), so a public reader can see the counts without the project-internal report. The underlying record is docs/FINAL_DELIVERY_REPORT.md §7 (project-internal); the frontend count was re-run this session to 106 (release/CURRENT_RELEASE_STATE.md:121). See also EVALUATION.md §7.2.

5.4 Full tests/unit — the 5–6 failures, reported honestly

Running the entire unit tree trips 5–6 failures. They are reported here rather than hidden, and they are classified by cause exactly as docs/FINAL_DELIVERY_REPORT.md §7 records them:

#	Failure	Cause	Regression?
1–4	test_safe_delete_shim (×4)	the sandbox's bulk-delete guard (Windows verbatim-path behaviour)	No
5	one ordering flake in the router route test	passes in isolation; order/collection-dependent	No
6	one stale adapter test	asserts optical_sar absent when CROMA is unshipped — CROMA is now shipped	No

Why these are not regressions. They were isolated by a re-run of the affected files together, which passes 137 tests (docs/FINAL_DELIVERY_REPORT.md §7; EVALUATION.md §7.3–7.4). The test_safe_delete_shim failures are Windows-specific verbatim-path behaviour that does not occur on the deployment host; the ordering flake disappears when the router route test runs in isolation; the stale adapter test encodes an assumption (CROMA not yet shipped) that the codebase has since outgrown. None of them is a behavioural defect in shipped code.

Honesty note. The precise full-suite collected count is UNKNOWN — not established from the available evidence. What the evidence establishes is: the two targeted suites pass (106, 183), the full suite has 5–6 environmental/ordering failures, and a re-run of the affected files passes 137.

5.5 The evidence-engine determinism suite

The evidence engine (the component that turns specialist output into a proof) has its own suite — tests/unit/test_evidence_engine.py (73 tests) — covering determinism and purity: the same input produces the same evidence, and computing evidence does not mutate inputs. This is what makes the "evidence is independent of the verdict" property in §6.5 hold.

5.6 Invocation traps

• Use the venv interpreter. .venv/Scripts/python.exe -m pytest … on Windows.
• The full-suite run trips a bulk-delete guard in the authoring sandbox (§10.3). Run the targeted suites to avoid it.
• Do not pipe pytest through grep in the authoring sandbox — output is block-buffered and a killed pipeline swallows it (§10.6). Redirect to a file instead.

---

6. Reproduce a live run

6.1 The deployed stack

The live topology is Cloudflare Pages → Render → outbound tunnel → GitHub Codespace (docs/DEPLOYMENT.md §1; README.md §Deployment):

Layer	Role	Host
Cloudflare Pages	static frontend	https://satquery.pages.dev
Render	orchestrator / API gateway (/api/*, CORS, wake flow)	https://<backend-host>
GitHub Codespace	FastAPI inference host, CPU, port 8000	potential-space-trout-r4ppw969w45j2pvvw

Deployed revisions (VERIFIED):

Component	Revision
Frontend	2d7ae53b482d
Backend / orchestrator	89d80eaddec5
Inference	5a0936ace491

Trap. The deployment sources are private repositories, separate from this release. The deploy/ directory inside the monorepo working copy is stale and untracked — it is not the deployed source (docs/DEPLOYMENT.md §1). Reproducing a live run means driving the deployed endpoints, not rebuilding from deploy/.

6.2 Health and capabilities

curl --noproxy '*' https://<backend-host>/api/health
curl --noproxy '*' https://<backend-host>/api/capabilities

The /api/health payload (recorded live, docs/DEPLOYMENT.md §2):

{"status":"ok","service":"satquery-orchestrator",
 "tunnel":{"agent_connected":true,"agent_id":"codespaces-fd1038","pending":0,"completed":97},
 "config":{"codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n","codespace_port":8000,
           "transport_mode":"auto","tunnel_timeout_s":150.0,"wake_timeout_s":120.0,
           "upstream_timeout_s":90.0,"device":"cpu","has_github_token":true}}

/api/capabilities returns six tasks, all available: true. A live run requires tunnel.agent_connected: true; if the Codespace is stopped, the request parks until the tunnel timeout (docs/DEPLOYMENT.md §5).

--noproxy '*' is not optional in the authoring sandbox (§10.1). The sandbox proxy is dead; without the flag, the request fails before reaching Render. In a normal environment the flag is harmless.

6.3 What was run, and how to re-run it

The behavioural validation is 3 passes × 8 cases = 24 live runs, each pass 8/8, with 0 mock nodes and a live trace bar at 94.4444 % (.workbuddy-ai/scratch/live_validation/ LIVE_VALIDATION_POSTFIX.md). The eight cases exercise all six specialist tasks plus the two router-defect queries (B1, B2).

pass	HEAD	harness	result
1	ff46eba42b18+d413d3672311	v1 (fill_input)	8/8
2	2d7ae53b482d	v2 asserting	8/8
3	2d7ae53b482d	v2 asserting (pre-discriminator-fix)	8/8 (recomputed)

To re-run: drive https://satquery.pages.dev in a headed browser (the headed requirement is explained in §6.4), upload one asset per case for single-image tasks and a pair for the temporal tasks, submit the query, and read the run identifier, the mock_nodes count, the trace fill, and the answer's [task] tag. The eight expected cases are tabulated in EVALUATION.md §6.2.

6.4 The Chrome focus trap — why the harness asserts

Pass 1's harness drove the query box with fill_input(), which types with real CDP key events. A re-run attempt failed on case 1 with run_id=0002, mock_nodes=9, answer="No answer yet", and only a capabilities call — the mock path. Root cause:

Chrome drops synthesized key events when the browser window does not hold OS focus; the harness had no assertion, so it clicked Run with the page's default query still in the box.

Measured directly: with Chrome backgrounded, press_key("Z") left #qtext.value unchanged, while type_text("Q") (CDP Input.insertText, not focus-gated) inserted fine (LIVE_VALIDATION_POSTFIX.md, "Why this pass needed a new harness").

This is the single most important trap for anyone reproducing a live run, because it produces a silent false pass — the pipeline "works", it just answered a different question. The fix was to assert the input state before dispatching, with three pre-dispatch assertions:

• q_ok — the query box really held the query;
• obs_ok — #obsTail == 'ready';
• t0_ok — both frames present where required.

The full account, including the check that pass 1 was not infected, is in EVALUATION.md §6.4–6.5. In short: pass 1's intents are query-specific (A1 reads taskvqa…temporalnone, not the default's taskchange…temporalrequired), their answers embed the query text, and A6's answer proves two files were uploaded — so pass 1's results are clean.

6.5 The recompute-verdicts safety property

Pass 3 was launched with a harness build that still carried two discriminator bugs (the answer [task] tag exists only for region tasks; the intent panel is a concatenated string needing a non-greedy match). Its raw output says SUMMARY 0/8; the verdicts are recomputed from the recorded evidence by recompute_verdicts.py, giving 8/8. This is not a workaround — it is the intended safety property:

The recorded evidence (run id, mock_nodes, intent, answer, assertions) is independent of the verdict computation, so a harness bug can never silently turn a real failure into a pass, and never costs a re-run to correct.

— LIVE_VALIDATION_POSTFIX.md, Pass 3.

Two further harness bugs were found and fixed, both causing false failures (not false passes): the answer [task] tag exists only for region tasks, and the intent panel needs a non-greedy match. The harness now computes the dispatched task as answer_tag when present, else the reading. Both are recorded because a false failure and a false pass are different risks and the project tracks them separately.

6.6 Cold start and tunnel caveats

• Cold start. Render's free tier sleeps and the Codespace may be stopped. The first request can exceed the client timeout while weights are fetched; a retry a few seconds later normally succeeds. Warm the stack before any demonstration and confirm agent_connected: true (README.md §Deployment caveats).
• Tunnel gaps (B-07, OPEN). The tunnel agent can be briefly absent; a request during a gap may hang or return 504. This is not fixed in production. Root cause: in auto mode a tunnel timeout falls through to the forwarded-port path, spending the 120 s wake timeout on a 302 — the observed ~249 s failure (docs/DEPLOYMENT.md §6). A patch exists and was deliberately not deployed.
• Never retry POST /api/infer at the gateway — a retry consumes inference twice (docs/DEPLOYMENT.md §7).

---

7. Reproduce and verify the released artifacts

7.1 The manifest and checksums are generated, not typed

models/manifest.json and models/checksums.sha256 are produced by release/tools/generate_model_manifest.py, which reads the actual files:

Generated by reading the files. No byte count or hash is typed by hand. Backbones are NOT redistributed; they are fetched from the Hugging Face Hub, pinned by revision.

— models/manifest.json ("note" field).

Verify the local weights:

sha256sum -c models/checksums.sha256

models/checksums.sha256 (committed):

c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa  change/head.pt
cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a  change_vqa/head.pt
785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab  optical_sar/head.pt
93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb  grounding/head.pt
8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c  router/adapter.pt
07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e  vlm/adapter_model.safetensors

The manifest records, per artifact: id, task, kind, path (in the project), hf_path (on the Hub), backbone, architecture, source_metric_artifact, config_hash, bytes, sha256, and status. The manifest's own header records config_hash: 78f1e3700da15aa1 and artifact_count: 6, generated 2026-09-25T18:15:38+00:00.

7.2 Independent Hub verification

export HF_TOKEN=...            # token with repo.content.read
python release/tools/hf_verify.py

release/tools/hf_verify.py is an independent check: it does not trust the upload step. It reads the manifest, re-downloads each artifact from the Hub over direct HTTPS, hashes the bytes it received, and compares against the locally-computed sha256. Its docstring is explicit that it uses direct HTTPS deliberately:

Note: hf_hub_download is deliberately NOT used here — in this environment it returned an empty file (sha256 e3b0c442…), which would have produced a false FAIL. Direct HTTPS is the honest check.

Recorded result (release/tools/hf_verify_report.txt):

repo            : thundercode/SatQuery
private         : False
sha (HEAD)      : 55681e0cddb91a4a5655da98a49bc025e537b657
lastModified    : 2026-09-25T18:21:52.000Z
files on Hub    : 22

STATUS  hf_path                                    remote bytes   local bytes
----------------------------------------------------------------------------------
MATCH   change/head.pt                               63,231,009    63,231,009
MATCH   change_vqa/head.pt                            5,822,809     5,822,809
MATCH   optical_sar/head.pt                          14,427,457    14,427,457
MATCH   grounding/head.pt                            12,639,041    12,639,041
MATCH   router/adapter.pt                               211,961       211,961
MATCH   vlm/adapter_model.safetensors                34,798,048    34,798,048

artifacts verified : 6
artifacts failed   : 0

  OK  README.md
  OK  MODEL_CARD.md
  OK  models/manifest.json
  OK  models/checksums.sha256

Total released weight payload: 131,130,325 bytes (~125 MiB) across the six artifacts (release/HF_RELEASE_VERIFICATION.md §4). Exit code 0 with artifacts failed : 0 means the release is intact.

7.3 A verification method that was itself wrong — the empty-file false FAIL

This is the most instructive reproducibility incident in the release, and it is recorded because a verifier that silently hashes an empty file is a verifier that can lie in both directions.

The first verification attempt reported all six artifacts DIFFER, with every remote hash equal to:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

— the sha256 of empty content. The cause was not the upload: hf_hub_download returned an empty file in the authoring environment (a download-path problem), so the verifier hashed nothing.

It was caught by a second, independent method — a direct curl download — which produced the correct hash 8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c for router/adapter.pt, byte-identical to the local file and confirmed to be a real PyTorch zip (PK\x03\x04, containing adapter/data.pkl). The verifier was then rewritten to use direct HTTPS with proxies disabled (ProxyHandler({})), and now reports 6/6 MATCH.

release/HF_RELEASE_VERIFICATION.md §5.1 states the lesson directly:

The failed first attempt is recorded because a verifier that silently hashes an empty file would have produced a false failure — and, with a different bug, could just as easily have produced a false pass.

What a third party should take from this. Do not use hf_hub_download as the sole verification path. The honest check is a direct HTTPS download with proxies disabled, hashing the bytes received, and comparing sizes as well as hashes. hf_verify.py does all three (size_ok = remote_bytes == local_bytes == len(content); hash_ok = remote_hash == local_hash).

7.4 Cross-checks against independently-recorded hashes

Two of the six hashes can be checked against values recorded elsewhere in the project, independent of this release (release/HF_RELEASE_VERIFICATION.md §6):

Artifact	Recorded elsewhere	Computed here	Agree
change_vqa_head	cfae5e43…d63a82a (artifacts/change_vqa/run/PROMOTION.json)	cfae5e43…d63a82a	yes
vlm_lora_adapter	07c76a75…a5adf5e (adapter provenance)	07c76a75…a5adf5e	yes

This is a stronger form of verification than self-consistency: it means the released bytes match hashes that were recorded by the training/export process, not merely by the release step.

7.5 The evidence archive

release/tools/build_archive.py builds the Phase-7 evidence archive. Per owner decision G3, everything under artifacts/ is included verbatim — including duplicates and caches — so the archive is ~4 GB and requires ZIP64. The layout:

SatQuery_AI_Final_Archive_2026-09-25/
  README_ARCHIVE.md              (what is inside, what was excluded and why)
  release/                       (the curated public release tree, verbatim)
  artifacts/                     (VERBATIM, per owner decision G3)
  evidence/
    live_validation/             (3 validation passes + screenshots)
    delivery/                    (delivery report + handoff)
  verification/                  (state, manifests, verification reports)

Weights and caches are stored with ZIP_STORED (no recompression) to avoid wasting hours on already-compressed data. The archive excludes secret/token files, OS junk, virtualenvs, node_modules, the HF hub cache, and temp browser profiles — documented in README_ARCHIVE.md, never silent. It is verified by release/tools/verify_archive.py, which extracts to a separate temp directory and checks the CRC of every member plus the sha256 of every artifact against the live files.

---

8. What "reproduce" means for the two externally-trained artifacts

Six trained artifacts are released. Four were trained inside the project's own pipeline; two were trained outside this repository on external GPU. This distinction is the heart of the reproducibility contract, so it is stated plainly rather than implied.

Artifact	Where it trains	Guide
router/adapter.pt	local CPU	configs/base.yaml §router
grounding/head.pt	local	configs/base.yaml §grounding_training
change/head.pt	local	configs/base.yaml §change
optical_sar/head.pt	local, seed sweep	docs/PHASE14_OPTICAL_SAR_DECISIONS.md
change_vqa/head.pt	external GPU (Kaggle)	docs/R02_KAGGLE_TRAINING_GUIDE.md
vlm/adapter_model.safetensors	external GPU	configs/base.yaml §training

8.1 What the repository reproduces for the external artifacts

For the two externally-trained artifacts, this repository reproduces:

1. The promotion gate — the checks the returned checkpoint must pass before it is accepted (byte-identity, sha256 match against the recorded digest, zero non-finite tensors).
2. The evaluation — the scoring path that turns a checkpoint into the recorded metrics.
3. The serving wiring — the code path that loads the artifact and routes to it.

It does not ship a one-command retrain for those two artifacts. That is stated, not implied.

8.2 The change-VQA path (Kaggle)

docs/R02_KAGGLE_TRAINING_GUIDE.md is the authoritative guide; it states its own scope:

Audience: the person running the external GPU job. Scope: everything needed to go from this repository to a returned, reviewable checkpoint. Not in scope: deciding whether R-02 is done — that happens after the artifact comes back.

Its status is IMPLEMENTATION_READY_FOR_EXTERNAL_TRAINING, and it is explicit that "training produces an artifact, not a verified capability, and the run record says TRAINED_UNVERIFIED."

The reproduction is byte-scoped: the upload set is defined by a digest so the external run can be shown to have started from exactly this code.

Item	Value
Upload set	249 files, at the root of the archive
Digest-defining subset	248 files, 67,225,247 bytes
Manifest hash	e98d3854db5f3edec76c1748c0c1bc3041d006ae5858debee31701539af7d0d6
Archive	satquery-ai.zip — 250 entries, 60.0 MB compressed
Inside the archive	UPLOAD_MANIFEST.txt — every file with size and SHA256, plus the digest

The digest covers 248 files, not 249, for a principled reason the guide states: the guide itself lives in docs/, so it is part of the upload set — but it also records the digest, and a document cannot contain the hash of itself. The digest is therefore defined over the upload set excluding this one file; that is the only definition that can be reproduced. For the same reason the byte total for all 249 files is not quoted in the guide — it changes whenever the guide is edited; the authoritative value is in UPLOAD_MANIFEST.txt.

The frozen STANet checkpoint the head depends on is verified by size and SHA256 in the notebook:

checkpoint : <CODE_ROOT>/artifacts/change/levir_change_v001/head.pt
bytes      : 63,231,009
sha256     : c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa
checkpoint verified against the frozen digest

Training knobs are fixed, not tuned-to-pass: HARD_STOP_SECONDS = 3 * 3600, EPOCHS = 40, BATCH_SIZE = 256, DEVICE = "cuda", SEED = 42 (the trainer's own defaults are epochs=40, batch_size=128, seed=42, patience=6, time_limit=10800s). The guide instructs: "Do not change these to get past an error."

Split integrity is measured, not assumed:

measured corpus ground truth (unique scenes / resolved questions):
  Train    1600 scenes   65967 questions
  Val       400 scenes   16441 questions
  Test      968 scenes   39686 questions
  Test2     968 scenes   31036 questions
integrity clean : True
scenes/split    : {'Train': 1600, 'Val': 400}
overlap         : {'Train|Val': 0}

The two test sets (Test 39686, Test2 31036) are why change-VQA is quoted under two test sets (rule 3) — and why the promotion artifact carries both.

8.3 The VLM path

The VLM artifact is a PEFT LoRA adapter (r=16, alpha=32, dropout=0.05) on the text-model projections of HuggingFaceTB/SmolVLM-500M-Instruct (models/manifest.json). Its acceptance status is ACCEPTANCE-REJECTED — the metrics are usable (exact_match 0.963, f1 0.96432) but the adapter was not promoted. USABLE ≠ ACCEPTED (rule 7). Reproducing this artifact's training requires the external GPU environment described in configs/base.yaml §training; reproducing its evaluation and its serving wiring is done by this repository. The rejection is preserved in the record (artifacts/vlm/phase6_closure.json), not scrubbed.

8.4 The optical-SAR seed sweep

The optical-SAR head was trained locally with a seed sweep; the selected production artifact is fusion_head_production_v001, and its pre-registered metric is the 11.5 target recorded in docs/PHASE12_115_METRIC_COMPUTED.md and artifacts/optical_sar/fusion_head_production_v001/ pre_registered_115_metric.json. The input dimension is derived at runtime (not hardcoded) — 2318 = 3×768 + 12 + 2 (fusion_head.py:74, :311), independently confirmed in docs/PHASE14_OPTICAL_SAR_DECISIONS.md §1. The ruling is OPEN, and the accuracy (0.931) is never quoted without the macro-F1 (0.434161) — rare classes are poorly handled (rule 4).

8.5 What is not one-command

There is no script that takes this repository and produces all six artifacts in one invocation. The four locally-trained artifacts have local training paths, and the two external ones require an external GPU environment. This is a deliberate boundary: the release ships frozen artifacts with provenance, not a retraining harness.

---

9. What is NOT reproducible from this release — exhaustive

This section is the honest counterweight to §1. Every item below is something a reader might reasonably expect to reproduce and cannot from this release alone.

9.1 Not reproducible because the source is not released

Item	Reason
The three private deployment repositories (SatQuery-Frontend, SatQuery-Backend, SatQuery-Inference)	Private by design; their links 404 for an outside audience (docs/DEPLOYMENT.md §6).
The live deployment itself	Requires the private sources plus the Render/Codespace/Cloudflare accounts.
The monorepo working copy	Local only, no remote; 334 dirty entries (docs/DEPLOYMENT.md §1).

9.2 Not reproducible because it does not exist or was not run

Item	Reason
System-level end-to-end benchmark	No such benchmark exists. Per-specialist metrics are real; a single end-to-end number is NOT RUN (README.md §Known limitations 1).
Router test-split number	The test split was never run. The only router number is validation, ungated, n = 86 (rule 5).
Captioning benchmark	IMPLEMENTED but the benchmark is NOT RUN (EVALUATION.md §4.8).
A composite / "overall accuracy"	Forbidden by construction: official_aggregate_weights: null; normalize.py raises on aggregate requests (rule 8).

9.3 Not reproducible because the data is large or external

Item	Reason
CDVQA / SECOND imagery	Public but large. The release documents the acquisition + name-verification procedure, not the data.
BigEarthNet full corpus	Not downloaded — only a 28k S2 subset was used.
LEVIR-CD-256 / VRSBench raw data	Not redistributed; the splits (7120/1024/2048, and VRSBench n=16159) are documented, not shipped.
Fusion held-out test (n=4000)	Documented, not shipped.
Feature caches	Reproducible but included only in the verbatim archive, not in the release.

9.4 Not reproducible because it was external-GPU training

Item	Reason
change-VQA retrain	External GPU (Kaggle); the repo reproduces the promotion gate, evaluation, and serving wiring, not a one-command retrain (§8.2).
VLM LoRA retrain	External GPU; and the adapter is ACCEPTANCE-REJECTED anyway (§8.3).

9.5 Not reproducible because it is frozen paperwork

Item	Reason
The historical ZeroGPU / Gradio deploy target	configs/deploy.yaml describes the old HF-Space/ZeroGPU target and is left undisturbed as frozen paperwork; no Gradio runtime exists in code (docs/DEPLOYMENT.md §9). Editing it would move the config hash.

9.6 Not reproducible as a bit-identical training run

The released artifacts are byte-identical (guaranteed by hash). The training runs that produced them are not claimed to be bit-reproducible on arbitrary hardware: floating-point non-determinism, different accelerators, and library-version drift all affect a training run. What is guaranteed is: the config is frozen, the seed is recorded (42), and the artifact is hashable.

9.7 Explicit non-claims

To remove any ambiguity, this release does not claim:

• that the live system achieves any single accuracy number;
• that the router's 0.965116 is a test result (it is validation-only, n = 86);
• that the VLM adapter is accepted (it is rejected);
• that calibration improves ECE (it worsens it, 0.013755 → 0.014929);
• that optical-SAR fusion is production-ready (ruling OPEN, macro-F1 0.434161);
• that change-VQA is settled (ruling OPEN, two test sets);
• that grounding is solved (0.2838 canonical IoU — useful, not solved);
• that B-07 tunnel gaps are fixed (patch prepared, NOT deployed).

---

10. Environment traps recorded for reproducibility

These are the edges that cost real debugging time in the authoring environment. They are recorded so a third party does not rediscover them — and, more importantly, so that a reviewer can distinguish "this is a documented environment quirk" from "this is a defect in the released code".

10.1 The dead sandbox proxy needs --noproxy '*' / ProxyHandler({})

Symptom. Outbound HTTP calls fail or hang in the authoring sandbox. Root cause. The sandbox proxy is dead; requests are routed to it and never reach the target. Fix. Disable proxies for the call:

curl --noproxy '*' https://<backend-host>/api/health

opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))

release/tools/hf_verify.py:37-38 does exactly this (opener_no_proxy()). In a normal environment the flag is harmless; in the sandbox it is mandatory.

10.2 pytest exists only in the repository virtualenv

Symptom. Invoking the system pytest fails or resolves to a different interpreter. Root cause. pytest is installed only in .venv. Fix. Always invoke the venv interpreter explicitly:

.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q

10.3 The full-suite pytest trips a bulk-delete guard

Symptom. Running the whole tests/unit tree trips 4× test_safe_delete_shim failures. Root cause. Windows verbatim-path defects in the sandbox's bulk-delete guard. The precise condition under which the shim intermittently triggers on Windows is UNKNOWN — not established from the available evidence. Fix / workaround. Run the targeted suites (106 and 183 pass cleanly); treat the 4 shim failures as environmental, not regressions (§5.4).

10.4 Cloudflare 308-redirects X.html → /X

Symptom. A request to something.html returns a 308 redirect. Root cause. Cloudflare Pages 308-redirects X.html → /X. Fix. Reference the extensionless path (docs/DEPLOYMENT.md §7).

10.5 Chrome drops synthetic CDP key events without OS focus

Symptom. A browser-driven run silently answers the default query; the query box looks untouched; mock_nodes is non-zero. Root cause. Chrome drops synthesized key events when the browser window does not hold OS focus. press_key/fill_input (real CDP key events) are focus-gated; Input.insertText (type_text) is not. Fix. Use type_text (not fill_input), and assert the input state before dispatch (q_ok/obs_ok/t0_ok). This is the single most dangerous trap because it produces a silent false pass (§6.4; EVALUATION.md §6.4).

10.6 Buffering hides liveness (two separate causes)

Symptom 1. A redirected browser-harness log stays at 0 bytes until the process exits — indistinguishable from a stall. Root cause 1. browser-use block-buffers stdout even when redirected. Fix 1. Force line-buffering and print a per-case marker (CASE_START <id>).

Symptom 2. Piping the harness through grep swallows all output if the pipeline is killed. Root cause 2. grep block-buffers when piped. Fix 2. Redirect to a file instead of piping through grep.

10.7 hf_hub_download returned an EMPTY file → a false FAIL

Symptom. Every artifact hashes to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 (empty content) and the verification reports all DIFFER. Root cause. hf_hub_download returned an empty file in the authoring environment (a download-path problem), so the verifier hashed nothing. The upload was fine. Fix. Use a direct HTTPS download with proxies disabled, hash the bytes received, and compare size as well as hash. release/tools/hf_verify.py is the honest check (§7.3).

10.8 codespace_name carries a trailing \n

Symptom. /api/health reports "codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n". Root cause. B-02; the wake path strips it, so it is cosmetic. State. OPEN (cosmetic); the fix is not deployed (docs/DEPLOYMENT.md §2, §6).

10.9 A forwarded Codespace port returns 302 for a private repo

Symptom. A forwarded Codespace port returns HTTP 302 instead of serving. Root cause. Private-repo port forwarding returns 302 — which is why the outbound tunnel exists (docs/DEPLOYMENT.md §7). Fix. Use the tunnel, not the forwarded port.

10.10 The tunnel agent must be started by the devcontainer

Symptom. A restarted Codespace comes up with agent_connected: false. Root cause. The tunnel agent is started by the devcontainer postStartCommand; if it does not run, the agent is absent. Fix. Ensure .devcontainer/ starts the agent on start (docs/DEPLOYMENT.md §7).

---

11. What is NOT RUN / OPEN / BLOCKED for this topic

Per the documentation standard, this section lists, for the topic of reproducibility itself, what is not settled.

Item	Status	Note
End-to-end system benchmark	NOT RUN	No such benchmark exists; nothing to reproduce.
Router test-split number	NOT RUN	Only validation (n = 86, ungated) exists.
Captioning benchmark	NOT RUN	Implemented; not benchmarked.
VLM adapter acceptance	REJECTED	Metrics usable; not promoted. USABLE ≠ ACCEPTED.
Optical-SAR fusion ruling	OPEN	Accuracy without macro-F1 is never quoted.
Change-VQA ruling	OPEN	Two test sets.
Grounding acceptance	OPEN	Two protocols; modest IoU.
Calibration improvement	REJECTED (retained in frozen config)	ECE worsened 0.013755 → 0.014929.
B-07 tunnel gaps	OPEN	Patch prepared, NOT deployed.
B-02 trailing \n	OPEN (cosmetic)	Not deployed.
Licence	OPEN	No LICENSE file exists; owner decision.
Exact full-suite collected count	UNKNOWN	Per-suite counts are known (106, 183, 137 re-run); the single collected total is not established.
Exact Windows trigger for the delete-shim flake	UNKNOWN	The intermittent condition is not established from the available evidence.
Bit-identical training on arbitrary hardware	NOT CLAIMED	Only artifacts are guaranteed byte-identical, by hash.

---

12. Where the evidence lives

Evidence	Location
The 20-claim metric verifier	release/tools/verify_readme_metrics.py
Its committed output	release/tools/readme_metrics_report.txt (ALL CLAIMS VERIFIED)
The Hub verifier	release/tools/hf_verify.py
Its committed output	release/tools/hf_verify_report.txt (6/6 MATCH)
Hub release verification write-up	release/HF_RELEASE_VERIFICATION.md
Model manifest (generated)	models/manifest.json
Checksums (generated)	models/checksums.sha256
Manifest generator	release/tools/generate_model_manifest.py
Archive builder / verifier	release/tools/build_archive.py, release/tools/verify_archive.py
Config loader + invariants	core/config.py (hash :76-80; _validate :94)
Frozen config registry	configs/base.yaml (§evaluation :273, §deployment :282)
Dependency manifest + profiles	requirements.txt
Live topology, env vars, traps	docs/DEPLOYMENT.md
Live validation (3 passes, 24 runs)	.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md, results_final.json, results_pass3.json, run_final2.txt, run_final3.txt
Verdict recomputation	recompute_verdicts.py
Kaggle training guide (change-VQA)	docs/R02_KAGGLE_TRAINING_GUIDE.md
Optical-SAR decisions	docs/PHASE14_OPTICAL_SAR_DECISIONS.md
Pre-registered 11.5 metric	docs/PHASE12_115_METRIC_COMPUTED.md
Grounding resolution decision	docs/PHASE7_RESOLUTION_DECISION.md
Delivery test results (94/183/5–6/137)	docs/FINAL_DELIVERY_REPORT.md §7
Frontend suite re-run count (106)	release/CURRENT_RELEASE_STATE.md:121, README.md §The test suites
Evaluation protocols and honesty rules	EVALUATION.md
Benchmarks and their statuses	BENCHMARKS.md
Model cards and artifact details	MODELS.md
Datasets and splits	DATASETS.md
Training paths	TRAINING.md
Known limitations	LIMITATIONS.md