punyajain/epidebug-mm-v4

EpiDebug MM v4 (CPU, evidence-grounded)

Small CPU-trained heads on a frozen sentence-transformers/all-MiniLM-L6-v2 encoder. They rank competing failure hypotheses, classify the failure category and rank follow-up checks. It is a research seed, not a frontier reasoner, and is not for safety-critical decisions. Confidence is not calibrated.

Why v4

An audit showed that v3 (punyajain/epidebug-mm-v3) barely used the evidence. On the test split, v3's hypothesis accuracy was 0.641 with title, evidence, priors and stats all replaced by x, against 0.632 on clean input. Its ranking came from how the hypotheses are worded.

Changes in v4:

• The input is ordered TITLE, then EVIDENCE, then shortened PRIORS last. v3 ignored the title and put the priors first, and about 10% of test inputs went over 256 tokens. The case budget is now 512 tokens, and the evidence is also embedded as sentence chunks, so nothing is lost to truncation.
• The scorer uses interactions only: no score term depends on the hypothesis text alone.
• Evidence-shuffled negatives are added with two losses: a margin loss (the gold hypothesis must score higher on its own evidence than on another case's) and a uniform-KL loss under mismatched evidence.
• Training ran for 120 epochs (cosine schedule) over 4 hypothesis-order views, with the best epoch picked on validation (4).

Test (v3 split, 337 rows, 10 held-out families)

metric	v3	v4
hypothesis acc	0.632	0.602
cause acc	0.496	0.501
cause macro-F1	0.501	0.493
follow-up acc	0.362	0.386

The v3 numbers are from its own metrics file. The ablations below re-run v3 with a fixed hypothesis order, which gives a clean hypothesis accuracy of 0.632.

Evidence-sensitivity ablations (test)

ablation	v3 hyp acc	v4 hyp acc	v3 preds changed	v4 preds changed	v3 Δ gold prob	v4 Δ gold prob
clean	0.632	0.602	0.000	0.000	+0.000	+0.000
gold_first	0.638	0.703	0.009	0.136	-0.001	+0.072
evidence_x	0.638	0.644	0.009	0.214	-0.021	-0.018
all_x	0.641	0.131	0.412	0.795	-0.085	-0.291
zero_stats	0.546	0.596	0.187	0.178	-0.075	-0.044
evidence_shuffled	0.591	0.383	0.270	0.602	-0.085	-0.245

How to read this:

• A model that uses the evidence should drop when the evidence is shuffled or removed (all_x), and gain under gold_first.
• v4 does all of these. v3 is flat.

Honest limits

• Clean hypothesis accuracy is slightly lower than v3's, because v4 can no longer exploit wording priors.
• evidence_x (only the evidence body replaced, with the title and priors kept) barely hurts v4. The per-case physics priors and titles carry much of the signal, which is a dataset leak to fix in the next benchmark revision.
• The frozen 22M encoder has limited reasoning capacity. The scale-up path is GPU LoRA on a larger LM.
• Data is mostly synthetic, templated vignettes.
• No banned or private frames were used.

Files

• model.safetensors: the heads only. The MiniLM encoder is loaded from the Hub.
• v4.py, common.py: training, featurization and ablations. Run python v4.py; load with safetensors.torch.load_file.
• metrics.json, v3_ablations_test.json.