course-refunds-grpo-1.5b

Recipe: recipes/04-train/grpo · Collection: Course and community runs

The course's GRPO run: a 1.5B refunds agent trained on Modal with the dashboard watching. The reward is the recipe's executable refund environment; the holdout is split by scenario. holdout_before.jsonl and holdout_after.jsonl are every sampled row on both arms, so the before/after is recomputable from this repo alone.

What this is

The adapter the 40-step run usersim-grpo-40 left on the whileai-grpo-runs volume, with both holdout row files beside it. The recipe README explains the environment, why the reward is hackable and how the --balance flag closed the split it was hiding. Numbers for this exact run are in the two row files; the README's tables are from the runs it names.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	GRPO, 40 steps, run usersim-grpo-40

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "while-ai/course-refunds-grpo-1.5b")  # the headline arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/04-train/grpo
modal run train_modal.py --steps 40

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.