sfanm/d24-v6-midtrain

D24 v6 — midtrain

The loadable model at this repository's root is the terminal replay-free midtraining checkpoint in the public D24 v6 lineage. Starting from sfanm/d24-v6-pretrain, it consumed 100,000,595,968 OLMo-3/Dolmino midtraining tokens with an explicit pretraining replay fraction of zero and ended at optimizer iteration 23,842.

Architecture

Field	Value
Parameters	756,819,456
Layers / hidden size	24 / 1,536
Attention heads	12 (MHA)
FFN size	4,096 (SwiGLU/SiLU)
Position encoding	RoPE, theta 10,000
Normalization	RMSNorm
Embeddings	Tied
Tokenizer	GPT-2 BPE, vocabulary padded to 50,304
Context	2,048 tokens
Published weights	BF16

The Transformers class is LlamaForCausalLM, but this model uses the D24 architecture and GPT-2 tokenizer.

Training

Midtraining used sequence length 2,048, global batch 2,048, microbatch 4, AdamW at peak LR 4e-4, weight decay 0.1, gradient clip 1.0, and a WSD schedule with 1,000 warmup steps and a final 4,768-step cosine decay to 4e-5. Final held-out language-model loss was 1.365992 (perplexity 3.920).

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d24-v6-midtrain"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

This remains a base language model and has no chat template. All 12 retained, resumable Megatron distributed checkpoints are published under megatron/, from iter_0002000 through the terminal iter_0023842.

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.