ssurface/cot-dialect-math-olmo3-7b-think-sft-l5-scale1396

Olmo-3-7B-Think — MATH L5, distilled from 1396 traces

One point on a data-scaling curve: the L5 (extreme) MATH dialect trained on 1396 distilled traces instead of the full set. The question is how much data the most compressed dialect actually needs.

The curve

Traces	MATH-500 accuracy
250	51.0%
500	51.2%
1000	52.8%
1396 (this model)	45.6%

Accuracy is flat from 250 to 1000 traces and drops at 1396. On this evidence the extreme dialect is not data-hungry — a few hundred traces buy essentially all of it. Single seed per point, and 500 eval examples gives a 95% half-width of about ±4.4 pp, so treat the ordering within 250–1000 as noise; the 1396 point is the one that sits outside it.

Scored with the project's LaTeX-aware grader. MATH answers are \boxed{}, and a #### n extractor silently scores these models near 0%.

Training

Stage	SFT (distillation), MATH L5 dialect
Traces	1396
Engine	HuggingFace transformers + peft
LoRA	r=16, alpha=32, dropout=0.05
Epochs / LR	3 / 2e-4 cosine, warmup 0.03
Batch	16 x 4 = 64 effective
Hardware	1x NVIDIA A100 80GB

Usage

Solve this using Level 5 (Extreme).
Problem: {your problem}

Replies <think>...</think> then \boxed{answer}.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Think", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-math-olmo3-7b-think-sft-l5-scale1396")

Limitations

• One point in an ablation, not a model to deploy. The full-data MATH L5 adapter in this collection is the one to use.
• Single seed; n=500.