logan7000/llama32-3b-self-certainty-math345

Llama-3.2-3B-Instruct · Self-Certainty (Intuitor) · MATH-Level345

Self-supervised RL baseline (Intuitor self-certainty intrinsic reward) for the un-grpo-maj paper comparison. best_model/ checkpoint = step 10, the best eval point before collapse.

Result

metric	value
eval set	MATH-500 (held-out, pass@1, temp 0.6)
best eval_reward	0.4603 (step 10)
base model (≈step 0)	Llama-3.2-3B-Instruct

Training

• Method: GRPO with self-certainty intrinsic reward (Intuitor, arXiv 2505.19590, Eq.2: r = mean_t KL(U ‖ p_t), online, computed from the current policy).
• Data: q1716523669/MATH-Level345, 2 epochs, lr 3e-6, num_generations 12, EB 128, loss_type bnpo, beta 0 (no KL penalty), full-parameter ZeRO-3.

⚠️ Collapse note (important for interpretation)

This run collapsed at ~step 20: completions shrank to ~7 tokens, the intrinsic reward exploded to ~12.2, and eval accuracy fell 0.46 (step 10) → 0.002 (step 20). The model length-hacked the self-certainty signal (shorter output → higher mean per-token certainty).

Root cause: beta = 0 — no KL regularization to the reference model. The Intuitor paper explicitly states the KL penalty is critical to stability and that mis-calibrating it causes exactly this reward-gaming collapse. A faithful re-run should use beta > 0. This checkpoint is therefore the pre-collapse best (≈ near-base), not a converged self-certainty model.