ntlfi/opd-exopd-lambda1.0-klref-step10

ExOPD λ=1.0 + KL-to-reference trust region, step 10/15

From a smoke test of a fix for a reasoning-shortcut reward hack found in On-Policy Distillation with Reward Extrapolation (ExOPD) training. Full context: ntlfi/opd_training (see SESSION_LOG.md and algos/exopd_incident_2026-08-07.md).

GSM8K test accuracy: 79.53% (1049/1319) — essentially matches the untrained base model (79.61%), and a large improvement over the same training run without this fix (74.15% at a comparable stage). Training-time metrics: loss=0.218, kl_penalty=0.0024.

Teacher: meta-llama/Meta-Llama-3-70B-Instruct (frozen). Student/base: meta-llama/Meta-Llama-3-8B-Instruct. λ=1.0 (standard on-policy distillation reduction, no reward-extrapolation reference term active), plus an added KL(π_θ‖π_ref) trust-region penalty (kl_ref_coef=0.1, not part of the published method — added to counter a reward-hacking failure mode this repo found independently) using Schulman's k3 estimator.

For comparison, plain OPD (this repo's closed-form top-K reverse-KL loss, same teacher/student) reaches 81.35% at its best checkpoint.