grammar-v7b (run #2 do-no-harm calibration) -- FAILED the rule; NOT shipped

Educational / provenance release. This checkpoint is not used in production. The live site keeps shipping mydeskmate/grammar-v5.

What it was

Run #2: v7 (run #1) calibrated on its own mined hard negatives -- 478 clean wikitext-2-train sentences it wrongly edited (7.97% of 6000 scanned), upsampled x3, mixed with jfleg x2 / wikitext x0.1 / coedit x0.05 / agentlans x0.01, --identity_weight 3.0, lr 8e-5, 2 epochs, 250 steps on a Google Colab Tesla T4. Checkpoint selected by best validation loss.

Measured (official scorers, GPU run; local CPU re-verification of the same

pipeline on run #1 matched all four metrics exactly)

benchmark v5 (live) v7 (run #1) this run rule verdict
CoNLL-2014 test F0.5 47.35 52.10 40.25 (P 52.70 / R 20.69) > 47.35 FAIL
BEA-2019 dev F0.5 33.90 35.48 29.65 (P 40.39 / R 14.37) > 33.90 FAIL
JFLEG test GLEU 62.78 63.25 59.64 >= 62.30 FAIL
Do-No-Harm (clean touched) 2.39% 5.00% 1.40% (7/500) < 5% PASS

Verdict

Over-calibration: the do-no-harm gate was fixed (1.40%, better than the live v5) but correction recall collapsed (CoNLL recall 34.87 -> 20.69, BEA recall 22.24 -> 14.37, JFLEG below the 62.30 hold-line). The pre-registered rule (CoNLL/BEA up AND GLEU >= 62.30 AND Do-No-Harm < 5%) fails on three of four conditions. Reported as measured; nothing hidden. The lesson: 3x identity weight on an identity-heavy mix overshoots; a future run needs a much lighter calibration nudge (or checkpoint selection by JFLEG GLEU instead of validation loss, which is biased toward copying on an identity-heavy mix).

Downloads last month
16
Safetensors
Model size
0.2B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support