mrinaal-124m-instruct-v3-mathmix-smoltalk-150k

124M-parameter decoder-only causal language model instruction-tuned from mrinaalarora/mrinaal-124m-base-v3-mathmix.

This checkpoint is supervised fine-tuned on the first 150000 valid examples from HuggingFaceTB/smol-smoltalk, using the dataset's messages column and assistant-token-only loss masking.

model config

param	value
parameters	~124M
layers	12
hidden size	768
attention heads	12
context length	1024 tokens
vocab size	50257
positional encoding	RoPE
norm	RMSNorm
activation	SwiGLU
tokenizer	GPT-2 tokenizer

training

• base checkpoint: mrinaalarora/mrinaal-124m-base-v3-mathmix/model.safetensors
• loaded checkpoint: /root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v3-mathmix/snapshots/e060aaee7cb502b9fd74820ea68659826a6cce09/model.safetensors
• dataset: HuggingFaceTB/smol-smoltalk
• train split examples loaded: 150000
• validation split examples loaded: 3000
• skipped train rows over context length: 104986
• context length: 1024 tokens
• epochs: 1.0
• optimizer steps at saved checkpoint: 16500
• total tokens seen: 72970859
• assistant-label tokens seen: 52170383
• train loss at saved checkpoint: 1.8399193286895752
• validation loss at saved checkpoint: 1.658129612604777
• best validation loss during run: 1.658129612604777
• final validation loss during run: 1.6978322048981984
• hardware: NVIDIA H100

SFT format

Training examples use a simple GPT-2-tokenizer-compatible chat format:

<|endoftext|>user: ...
assistant: ...
user: ...
assistant: ...
<|endoftext|>

Labels are masked so that user/prompt/separator tokens use -100, while assistant answer tokens and the final <|endoftext|> target contribute to loss.

train source mix

• everyday-conversations: 1254
• explore-instruct-rewrite: 1644
• openhermes-50k: 25287
• self-oss-instruct: 26402
• smol-contraints: 19086
• smol-magpie-ultra-short: 51258
• smol-summarize-20k: 9117
• smol-summarize-5k: 1062
• smollm-rewrite-30k: 14890

files

• model.safetensors — best checkpoint converted from best.pt
• run_summary.json — full training run metadata
• last.pt was not uploaded; this repo intentionally publishes the best checkpoint only.

loading

from safetensors.torch import load_file

state_dict = load_file("model.safetensors")

To use with the original model class, clone the training repo and:

from safetensors.torch import load_file
from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig

config = ModelConfig(
    vocab_size=50257,
    block_size=1024,
    n_layer=12,
    n_head=12,
    n_embd=768,
)
model = DecoderOnlyTransformer(config)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()

strict=False is used because the safetensors conversion removes the duplicate lm_head.weight tensor and keeps token_embedding.weight; the original model class ties those weights.

generation note

For instruction-style inference, prompt with:

<|endoftext|>user: your instruction here
assistant:

Stop generation when the model emits <|endoftext|>.