paper-zero-rl-format-reward-4b

Recipe: recipes/papers/zero-rl-format-reward · Collection: Papers, replicated

Zero RL on a base model: GRPO from Qwen3.5-4B-Base on MATH levels 3 to 5, strict boxed reward against correctness only. A rigid format reward costs accuracy. Both arms are read leniently at eval so the target is one number.

Result

Run 2026-09-18, both arms, one H100, 46.2 GPU minutes.

Arm	pass@1	95% CI	pass@4	Steps	GPU min
Base, no training	0.51	[0.45, 0.57]	0.74	0	0
Baseline (strict boxed reward, -1 without a box)	0.63	[0.57, 0.68]	0.80	30	26.4
Recipe (correctness only)	0.72	[0.66, 0.77]	0.88	30	19.8

Recipe vs baseline: +0.094 [+0.052, +0.139] over 160 paired tasks. Both arms beat the base; the gap between them is the paper's finding and it clears the noise band the three base re-runs set. On the baseline's own strict reward the recipe arm scores lower (0.53 vs 0.58): it boxes less and is right more.

The six seed folders are the 2026-09-21 follow-up the README's Learned section asks for: three training seeds per arm at the same settings. Their paired read-out is not in the recipe README yet; the adapters are here so the numbers can be recomputed.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	recipe arm: correctness-only reward, 2026-09-18 run (the README's table)
baseline	baseline arm: strict boxed reward, 2026-09-18 run
recipe-seed17	recipe arm, seed 17, 2026-09-21
recipe-seed18	recipe arm, seed 18, 2026-09-21
recipe-seed19	recipe arm, seed 19, 2026-09-21
baseline-seed17	baseline arm, seed 17, 2026-09-21
baseline-seed18	baseline arm, seed 18, 2026-09-21
baseline-seed19	baseline arm, seed 19, 2026-09-21

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B-Base")
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b", subfolder="baseline")  # another arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/zero-rl-format-reward
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.