control-pretrain-30b-baseline-pretrain

Stage-1 final checkpoint of the control-pretraining 30B baseline campaign — the unfiltered control arm of Geodesic's pretraining-data-filtering study. Nemotron 3 Nano 30B-A3B architecture (hybrid Mamba2 + attention + MoE) trained from scratch (random init) for 501.3B tokens at seq 8192, constant LR 1e-3 (deliberately un-annealed: the anneal is stage 2), GBS 2048 on 512 GH200 GPUs. Final training loss 1.84; zero NaN iterations. Completed 2026-08-27.

Data: ClimbMix (0.698 aggregate, split token-proportionally over 8 shards) / Zyda-2 / AI-safety discourse, per the campaign mix sheet (2026-08-20 revision).

This is a BASE model — no instruction tuning, no chat behavior, and (being un-annealed) intentionally mid-curriculum: for most purposes prefer the midtrain (annealed) or sft (chat/reasoning) siblings. Use plain completion prompting.

Tokenizer note: the bundled tokenizer_config declares eos <|im_end|> (id 11), but pretraining used </s> (id 2) as the document separator — for CPT on this checkpoint, append EOD id 2 (e.g. geodesic-research/nemotron-base-tokenizer). Unlike NVIDIA's released Base checkpoints there is no zero-embedding trap: every vocab row trained from random init.

Provenance: converted from Megatron iter_0029881 (megatron-bridge, TP1/EP4). All 14

• Training run (loss/throughput curves): https://wandb.ai/geodesic/megatron_training/runs/tz0kuy6q optimizer-bearing stage-1 checkpoints: geodesic-research/control-pretrain-30b-baseline-ckpts.