paper-flash-reinforce-1.5b

Recipe: recipes/papers/flash-reinforce · Collection: Papers, replicated

Single-rollout REINFORCE on GSM8K from a stale sampler, with FlashREINFORCE's importance ratio, trust gate and length normalization. Rollouts come from a frozen copy of the adapter refreshed every 4 optimizer steps, the way an asynchronous trainer overlaps generation with training.

Result

Arm	pass@1	95% CI	pass@4	Steps	GPU min
Base, no training	0.40	[0.33, 0.47]	0.58	0	0
Baseline (uncorrected stale REINFORCE)	0.47	[0.41, 0.55]	0.68	40	22.9
Recipe (wai.FlashReinforce)	0.41	[0.34, 0.48]	0.63	40	21.3

Recipe vs baseline: -0.069 [-0.115, -0.025] over 120 paired tasks. Verdict: unresolved, one seed per arm. The rollouts were barely stale at this step size: mean sampler-to-learner KL was 3e-4 and the trust gate masked one of 64 trajectories on 5 of 40 steps. The correction had almost nothing to correct. Only the recipe arm's adapter was kept; history.json is its per-step log.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	recipe arm: wai.FlashReinforce update, 2026-09-21 run

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "while-ai/paper-flash-reinforce-1.5b")  # the headline arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/flash-reinforce
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.