smollm3-rl — GRPO RL checkpoint

GRPO (TRL) LoRA adapter (r=64, alpha=128, all attention and MLP projections) on HuggingFaceTB/SmolLM3-3B-Base (revision d78a42f79198603e614095753484a04c10c2b940). Trained on MATH train (7,393 problems) for 300 steps with the RL-Zero prompt and no forced opener: 32 prompts x 16 samples per step, temperature 1.0, 4,096-token completion cap, lr 1e-5, no KL, clip 0.20/0.28, truncation penalty (a rollout that hits the cap scores 0). Reward is 1 if the committed answer is correct: the last Answer: line, falling back to the last \boxed{}.

Usage

Merge the adapter into the base weights and save them. vLLM has no native SmolLM3 implementation (it runs the model through its Transformers backend), so we evaluated the merged weights rather than serving the LoRA adapter.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE, REV = "HuggingFaceTB/SmolLM3-3B-Base", "d78a42f79198603e614095753484a04c10c2b940"
base = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "reasoning-cues/smollm3-rl").merge_and_unload()
tok = AutoTokenizer.from_pretrained(BASE, revision=REV)   # the base tokenizer; see note below
model.save_pretrained("smollm3-rl-merged")
tok.save_pretrained("smollm3-rl-merged")

Then point vLLM at the merged directory, e.g. LLM(model="smollm3-rl-merged", dtype="bfloat16").

Tokenizer: load it from the base model, as above. The tokenizer_config.json in this repo was written by a recent transformers (tokenizer_class: TokenizersBackend, plus pad_token = <|end_of_text|> added for training) and may not load in older versions; the vocabulary is unchanged from the base.

Prompt

Raw text: no chat template, and no BOS token (SmolLM3-3B-Base defines none). Replace {q} with the problem. The prompt ends immediately after "Answer:" with no trailing newline or space; generation continues from there.

Solve the following math problem step by step.
The last line of your response should be the answer to the problem in form Answer: $Answer (without quotes) where $Answer is the answer to the problem.
{q}
Remember to put your answer on its own line after "Answer:"

As a Python string:

PROMPT = ("Solve the following math problem step by step.\n"
          "The last line of your response should be the answer to the problem in form "
          "Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n"
          "{q}\n"
          'Remember to put your answer on its own line after "Answer:"')

A forced opener (cue) is appended directly to this string, e.g. PROMPT.format(q=q) + ".\n\nOkay".

Stop generation on <|end_of_text|>, <|im_end|>, or the string "\nSolve the following math problem".

Answer format and grading

The model commits with \boxed{}, not with an Answer: line, even though the RL-Zero prompt asks for Answer:. Share of traces by how they commit (T=0.6):

eval	\boxed{} only	Answer: line	neither	mean / median tokens
MATH-500, no cue (4,000 traces)	93.4%	0.1%	6.5%	1,036 / 358
MATH-500, .\n\nOkay cue (4,000)	80.9%	10.6%	8.5%	1,535 / 523
AMC 23, no cue (1,280)	83.4%	0.1%	16.6%	4,545 / 923

Bold **Answer:** lines never occur. The "neither" traces are mostly runs that hit the token cap. The usual ending is The answer is: followed by \[ \boxed{...} \].

What this means for graders:

• The grader must read \boxed{}. A grader that reads only the Answer: line would score this model near zero.
• Both of the paper harness's graders work here. data/bank reads \boxed{}; data/bank_rlzero takes the last Answer: line when one exists and falls back to the last \boxed{} otherwise.
• Training used the same rule as bank_rlzero. Our training grader also accepts a bold **Answer:**, but since the model never writes one, that difference doesn't matter.
• Stop strings differ. Training and our evals stop on "\nSolve the following math problem", while the paper harness stops on "\nProblem:". We haven't measured how often either one fires on this model.

Reference scores (for checking a reproduction)

Our harness: vLLM on merged bf16 weights, T=0.6, top-p 0.95, seed 20260814, RL-Zero prompt, the grader described above. "Commit" is the share of traces with an extractable answer; "cond" is accuracy among those.

benchmark	protocol	pass@1 (± SE)	commit	cond	capped
MATH-500	500 × 8, 8,192-token cap	0.619 ± 0.018	0.924	0.670	5.0%
MATH-500 + .\n\nOkay	500 × 8, 8,192-token cap	0.609 ± 0.017	0.904	0.673	8.1%
AMC 23	40 × 32, 31,744-token cap	0.310 ± 0.059	0.834	0.372	7.8%

The AMC 23 run uses the paper harness's protocol (32 samples, 31,744-token budget). The MATH-500 runs use 8 samples and an 8,192 cap. On the base model the two harnesses agreed within a point on MATH-500: .\n\nOkay 0.465 (ours) vs 0.468 (paper harness), and .\n\nThe 0.499 vs 0.506. At a 31,744 budget, expect MATH-500 to come out equal or slightly higher, since 5% of our traces hit the 8,192 cap.

Behavior with a cue

• The model opens with To. In MATH-500 eval traces without a cue, about 62% of responses open with To and 6% with The. Okay never appears.
• Forcing .\n\nOkay is roughly neutral: 0.609 with the cue vs 0.619 without, a gap inside one SE.
• The cue makes traces longer: mean length goes from 1,036 to 1,535 tokens and the cap rate from 5.0% to 8.1%. Accuracy on committed answers is unchanged (0.673 vs 0.670). Under the cue, 10.6% of traces write an Answer: line (vs 0.1% without), so the cue arm depends more on the Answer: path of the grader.
• Different comparison from the base model. On the base model the cue adds 28 points (0.186 → 0.465). On this checkpoint the no-cue score is already above base-plus-cue, so a cue arm here asks whether the cue still adds anything after RL. So far it doesn't.