mrinaal-124m-base-v3-mathmix

124M-parameter decoder-only causal language model continued-pretrained from mrinaalarora/mrinaal-124m-base-v2.

This v3 mathmix checkpoint adds 1.5B more GPT-2-tokenized next-token-prediction tokens on top of the v2 base checkpoint, with a heavier math/data-reasoning recipe.

data recipe

share	component	dataset	train tokens	validation tokens
41.67%	finemath-4plus	`HuggingFaceTB/finemath`, subset `finemath-4plus`	625M	12.5M
23.33%	infiwebmath-4plus	`HuggingFaceTB/finemath`, subset `infiwebmath-4plus`	350M	7M
20%	fineweb-edu-dedup	`HuggingFaceTB/smollm-corpus`, subset `fineweb-edu-dedup`	300M	6M
10%	dclm-baseline-1.0	`mlfoundations/dclm-baseline-1.0`	150M	3M
5%	cosmopedia-v2	`HuggingFaceTB/smollm-corpus`, subset `cosmopedia-v2`	75M	1.5M

model config

param	value
parameters	123,551,232
layers	12
hidden size	768
attention heads	12
context length	1024 tokens
vocab size	50257
positional encoding	RoPE
norm	RMSNorm
activation	SwiGLU
tokenizer	GPT-2 tokenizer

training

base checkpoint: mrinaalarora/mrinaal-124m-base-v2/model.safetensors
loaded checkpoint: /root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v2/snapshots/d94b2e9f829173b824705eca148febac28a4198c/model.safetensors
continued pretraining data: 1,500,000,000 GPT-2 tokens
validation data: 30M GPT-2 tokens
dataset dir: /vol/datasets/cpt_mathmix_gpt2_1p5b_train
validation dir: /vol/datasets/cpt_mathmix_gpt2_30m_val
optimizer steps at saved checkpoint: 175000
configured max steps: 183106
train loss at saved checkpoint: 2.208630084991455
validation loss at saved checkpoint: 2.6333396577835084
best validation loss during run: 2.6333396577835084
final validation loss during run: 2.7023850011825563
hardware: NVIDIA H100

files

model.safetensors — best checkpoint converted from best.pt
run_summary.json — full training run metadata
last.pt was not uploaded; this repo intentionally publishes the best checkpoint only.

loading

from safetensors.torch import load_file

state_dict = load_file("model.safetensors")

To use with the original model class, clone the training repo and:

from safetensors.torch import load_file
from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig

config = ModelConfig(
    vocab_size=50257,
    block_size=1024,
    n_layer=12,
    n_head=12,
    n_embd=768,
)
model = DecoderOnlyTransformer(config)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()

strict=False is used because the safetensors conversion removes the duplicate lm_head.weight tensor and keeps token_embedding.weight; the original model class ties those weights.