ntlfi/opd-exopd-lambda1.0-klref-step15

ExOPD λ=1.0 + KL-to-reference trust region, step 15/15 (final)

From a smoke test of a fix for a reasoning-shortcut reward hack found in On-Policy Distillation with Reward Extrapolation (ExOPD) training. Full context: ntlfi/opd_training (see SESSION_LOG.md and algos/exopd_incident_2026-08-07.md).

GSM8K test accuracy: 76.12% (1004/1319) — better than the same run without this fix (74.15%), but degraded from this same run's own step10 checkpoint (79.53%, see ntlfi/opd-exopd-lambda1.0-klref-step10) as the KL penalty grew from 0.0024 (step10) to 0.222 (this step) over the last 5 steps. The fix works directionally but kl_ref_coef=0.1 is not strong enough to hold for the full run — a stronger or scheduled coefficient is the natural next thing to try, not yet done.

Teacher: meta-llama/Meta-Llama-3-70B-Instruct (frozen). Student/base: meta-llama/Meta-Llama-3-8B-Instruct. λ=1.0 (standard on-policy distillation reduction, no reward-extrapolation reference term active), plus an added KL(π_θ‖π_ref) trust-region penalty (kl_ref_coef=0.1, not part of the published method — added to counter a reward-hacking failure mode this repo found independently) using Schulman's k3 estimator.

For comparison: base model 79.61%, plain OPD's best checkpoint 81.35%.