control-pretrain-30b-baseline-sft

The final model of the control-pretraining 30B baseline campaign — the unfiltered control arm of Geodesic's pretraining-data-filtering study. A Nemotron 3 Nano 30B-A3B architecture (hybrid Mamba2 + attention + MoE, ~30B total / ~3B active) trained entirely from scratch (random init; no NVIDIA weights) through a three-stage curriculum totalling 603.8B tokens, finishing 2026-08-27. Zero NaN iterations across the entire campaign.

Training curriculum

Stage	Tokens	Seq	Schedule	Final loss
1. Pretrain	501.3B	8,192	constant 1e-3	1.84
2. Midtrain (anneal)	52.4B	32,768	cosine 7.5e-4 → 1e-5	1.32
3. SFT (this model)	50.1B (2 epochs)	32,768 packed	cosine 5e-6 → 0	0.717*

*SFT loss is computed on assistant/trainable tokens only (81.8% of tokens under answer-only masking).

• Stage-1 mix: ClimbMix (0.698 aggregate) / Zyda-2 / AI-safety discourse per the campaign sheet (2026-08-20 revision).
• Stage-2 mix: 10-corpus annealing blend at 32k context.
• Stage-3 data: geodesic-research/pa-warm-start-sft-heavy-25b-mix (default/train), 25.0B tokens of English STEM/reasoning SFT (competitive programming, science QA, math, agentic tool use, chat), packed at seq 32768 and run for two epochs. Rendered with the think-HISTORY chat template: prior assistant turns keep their full reasoning traces (94.8% of assistant turns carry reasoning_content).
• A 200-sample ground-truth render of the SFT data is browsable at geodesic-research/pa-warm-start-sft-25b-rendered-review.

Usage

A reasoning ("think") chat model. Use the bundled chat template (the think-history variant, truncate_history_thinking=false): generations open with a reasoning block and close it with </think> before the final answer.

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("geodesic-research/control-pretrain-30b-baseline-sft",
                                         trust_remote_code=True, torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("geodesic-research/control-pretrain-30b-baseline-sft")
msgs = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(m.device)
print(tok.decode(m.generate(ids, max_new_tokens=1024)[0]))

Evaluation status

Coherence-verified at release (8/8 structured generations, zero degeneracy signals, well-formed think blocks; W&B sft-final-coherence-iter2988). Full capability and alignment eval suites are running; this is a research artifact — the unfiltered baseline for filtering comparisons — not a production assistant. It has no safety post-training beyond what its SFT mix carries.

Provenance

• Training run (loss/throughput curves): https://wandb.ai/geodesic/megatron_training/runs/5wnjeoi1

• Megatron-format checkpoints (all 20 across the curriculum, optimizer-bearing): geodesic-research/control-pretrain-30b-baseline-ckpts

• Converted from Megatron iter_0002988 via megatron-bridge (TP1/EP4, --not-strict; MTP weights randomly initialized and unused).

• Sibling releases: control-pretrain-30b-baseline-pretrain (stage-1 final), control-pretrain-30b-baseline-midtrain (stage-2 final), and the mid-SFT 10B-token checkpoint retained internally.

• Trained on Isambard-AI (GH200) at 512 GPUs by the Geodesic control-pretraining campaign, 2026-08-22 → 2026-08-27.