epidebug-mm-v6

This is v5, an interaction-only option scorer over frozen all-MiniLM-L6-v2 that is invariant to option order, with evidence-swap negatives and a permutation-consistency loss, plus:

  • Option-only debiasing: a debiased product-of-experts with a train-time bias head that sees only the option texts (embedding plus row-relative style features: length rank, commas, digits, hedges). The main scorer is trained on main + log p_bias.detach(), so it only has to explain what option style can't. A gradient-reversal adversary (Ξ»=0.3) predicts the gold from the scorer's option projections. Both heads are dropped at inference.
  • Temperature scaling fitted on validation (hyp 0.99, follow-up 0.94, cause 4.32). The temperatures are stored in the T buffer and already applied to the outputs.

The selected run is seed 11, epoch 22, chosen by validation score. Load it with ckpt/v4.py.

Bench v2.4 test (n=337; internal benchmark, not public)

model hyp follow-up cause ECE hyp / fu / cause
v5 .401 .472 .549 .121 / .151 / .325
v6 .478 .665 .585 .091 / .037 / .139
best surface baseline .211 .377 β€”
option-only style_gb (data probe) .389 .579 β€”

Memorization caveat: 75 of the 337 test hyp golds also appear as train golds. On those rows v6 scores .933 hyp / .987 fu (v5: .840 / .773). On the other 262 rows v6 scores .347 / .573 (v5: .275 / .385). Most of the headline hyp number comes from the overlap rows.

Probes (change in hyp acc): blank evidence βˆ’.240, evidence swap βˆ’.318, gold cause inserted into the evidence +.300. The permutation probe gives identical predictions under every ordering.

Style reliance (v6 accuracy when style_gb is right vs wrong): hyp .534 vs .442 (v5: .481 vs .350); follow-up .667 vs .662. The hyp gap is smaller than v5's but not gone.

Caption pairs (n=24, private): hyp .667 vs v5's .917, a regression on this small set. There were 0 flips among the 8 clean-correct pairs.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
1.22M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support