control-pretrain-30b-baseline-midtrain

Stage-2 (midtraining/anneal) final checkpoint of the control-pretraining 30B baseline campaign — the unfiltered control arm of Geodesic's pretraining-data-filtering study. Continues the stage-1 base (501.3B tokens from scratch) with 52.4B tokens of a 10-corpus annealing blend at seq 32768 (weights-only warm start, fresh optimizer, cosine 7.5e-4 → 1e-5, adam_beta2 0.95). Loss 1.97 → 1.32 across the anneal; zero NaN. Completed 2026-08-27.

This is the annealed base — the natural checkpoint for completion-style use, CPT, or custom post-training, and the direct parent of the sft release. No instruction tuning: use plain completion prompting.

Tokenizer note: bundled tokenizer_config declares eos <|im_end|> (id 11), but training used </s> (id 2) as the document separator — for CPT append EOD id 2 (e.g. geodesic-research/nemotron-base-tokenizer). No zero-embedding trap: all vocab rows trained from random init in stage 1.

Provenance: converted from Megatron iter_0003126 (megatron-bridge, TP1/EP4). Megatron

• Training run (loss/throughput curves): https://wandb.ai/geodesic/megatron_training/runs/qeslzwcc checkpoints (both stage-2 saves): geodesic-research/control-pretrain-30b-baseline-ckpts.