mrinaal-124m-instruct-smoltalk-50k

124M-parameter decoder-only causal language model instruction-tuned from mrinaalarora/mrinaal-124m-base-v2.

This checkpoint is supervised fine-tuned on the first 50000 valid examples from HuggingFaceTB/smol-smoltalk, using the dataset's messages column and assistant-token-only loss masking.

model config

param	value
parameters	~124M
layers	12
hidden size	768
attention heads	12
context length	1024 tokens
vocab size	50257
positional encoding	RoPE
norm	RMSNorm
activation	SwiGLU
tokenizer	GPT-2 tokenizer

training

• base checkpoint: mrinaalarora/mrinaal-124m-base-v2/model.safetensors
• loaded checkpoint: /root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v2/snapshots/d94b2e9f829173b824705eca148febac28a4198c/model.safetensors
• dataset: HuggingFaceTB/smol-smoltalk
• train split examples loaded: 50000
• validation split examples loaded: 1000
• skipped train rows over context length: 34741
• context length: 1024 tokens
• epochs: 1.0
• optimizer steps at saved checkpoint: 5500
• total tokens seen: 24343600
• assistant-label tokens seen: 17409411
• train loss at saved checkpoint: 1.8008244037628174
• validation loss at saved checkpoint: 1.7162978529930115
• best validation loss during run: 1.7162978529930115
• final validation loss during run: 1.9427031993865966
• hardware: NVIDIA H100

SFT format

Training examples use a simple GPT-2-tokenizer-compatible chat format:

<|endoftext|>user: ...
assistant: ...
user: ...
assistant: ...
<|endoftext|>

Labels are masked so that user/prompt/separator tokens use -100, while assistant answer tokens and the final <|endoftext|> target contribute to loss.

train source mix

• everyday-conversations: 406
• explore-instruct-rewrite: 572
• openhermes-50k: 8609
• self-oss-instruct: 8716
• smol-contraints: 6316
• smol-magpie-ultra-short: 17067
• smol-summarize-20k: 2997
• smol-summarize-5k: 339
• smollm-rewrite-30k: 4978

files

• model.safetensors — best checkpoint converted from best.pt
• run_summary.json — full training run metadata
• last.pt was not uploaded; this repo intentionally publishes the best checkpoint only.

loading

from safetensors.torch import load_file

state_dict = load_file("model.safetensors")

To use with the original model class, clone the training repo and:

from safetensors.torch import load_file
from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig

config = ModelConfig(
    vocab_size=50257,
    block_size=1024,
    n_layer=12,
    n_head=12,
    n_embd=768,
)
model = DecoderOnlyTransformer(config)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()

strict=False is used because the safetensors conversion removes the duplicate lm_head.weight tensor and keeps token_embedding.weight; the original model class ties those weights.

generation note

For instruction-style inference, prompt with:

<|endoftext|>user: your instruction here
assistant:

Stop generation when the model emits <|endoftext|>.