paper-endpoint-sft-1.5b

Recipe: recipes/papers/endpoint-sft · Collection: Papers, replicated

SFT on 600 R1 math traces, full trace against first-and-last 21 steps only. The cut drops 19.2% of trace tokens by the paper's 20% rule. Holdout is 64 MATH-500 problems, a different corpus, 4 samples each.

Result

Arm	pass@1	95% CI	pass@k	Steps	GPU min
Base, no training	0.46	[0.36, 0.56]	0.69	0	0
Baseline (full trace)	0.29	[0.21, 0.37]	0.53	75	47.7
Recipe (endpoints only)	0.28	[0.20, 0.36]	0.50	75	21.7

Recipe vs baseline: -0.012 [-0.074, +0.047] over 64 paired problems. Verdict: unresolved. The number the paper's table does not have: both SFT arms land below the untrained base. One epoch of 600 traces teaches a 1.5B instruct model to write like R1 without teaching it to answer like R1. The proxy check reads over-optimized: trace shape moved +0.137 while pass@1 did not follow.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	recipe arm: endpoints only (n = 21), 2026-09-17 run
baseline	baseline arm: full trace, 2026-09-17 run

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "while-ai/paper-endpoint-sft-1.5b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-endpoint-sft-1.5b", subfolder="baseline")  # another arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/endpoint-sft
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.