ssurface/cot-dialect-math-olmo3-7b-think-sft-l5-scale500

Olmo-3-7B-Think — MATH L5, distilled from 500 traces

One point on a data-scaling curve: the L5 (extreme) MATH dialect trained on 500 distilled traces instead of the full set. The question is how much data the most compressed dialect actually needs.

The curve

Traces	MATH-500 accuracy
250	51.0%
500 (this model)	51.2%
1000	52.8%
1396	45.6%

Accuracy is flat from 250 to 1000 traces and drops at 1396. On this evidence the extreme dialect is not data-hungry — a few hundred traces buy essentially all of it. Single seed per point, and 500 eval examples gives a 95% half-width of about ±4.4 pp, so treat the ordering within 250–1000 as noise; the 1396 point is the one that sits outside it.

Scored with the project's LaTeX-aware grader. MATH answers are \boxed{}, and a #### n extractor silently scores these models near 0%.

Training

Stage	SFT (distillation), MATH L5 dialect
Traces	500
Engine	HuggingFace transformers + peft
LoRA	r=16, alpha=32, dropout=0.05
Epochs / LR	3 / 2e-4 cosine, warmup 0.03
Batch	16 x 4 = 64 effective
Hardware	1x NVIDIA A100 80GB

Usage

Solve this using Level 5 (Extreme).
Problem: {your problem}

Replies <think>...</think> then \boxed{answer}.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Think", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-math-olmo3-7b-think-sft-l5-scale500")

Limitations

• One point in an ablation, not a model to deploy. The full-data MATH L5 adapter in this collection is the one to use.
• Single seed; n=500.