MOME-NanoChat-D24-100B

Raw NanoChat checkpoints for the three d24 models trained for 100,000 optimizer steps with a 1,048,576-token global batch (104,857,600,000 training tokens). These files are native PyTorch checkpoints, not Transformers-format models.

Models

Directory	Architecture	Network params	Memory params	Validation bpb
dense/	Dense d24	0.780B	0	0.6640
vembedding/	Dense value embedding on 12 layers	0.780B	0.604B	0.6484
mome-a2-m12/	MoME A2/12 on 12 layers	0.780B	0.604B	0.6531

All runs use the shared 32k tokenizer and the post-NeurIPS Recipe C settings on ClimbMix. The effective arguments are retained in each meta_100000.json and in configs/d24_100b.json.

Reproducibility

• Training code: commit c85060f297840f29c470028fa2c886110c70a3e1 (recipe-upgrade-denseve-d24).
• Data: ClimbMix training shards 0--2499; validation shard 6542.
• Hardware/runtime: eight H100 GPUs and Flash Attention 3.
• Seed: 42.
• Context length: 2,048.

The organized reproduction branch provides:

bash experiments/reproduce/d24_100b.sh --plan
bash experiments/reproduce/d24_100b.sh

Loading

Use the pinned NanoChat code and the matching metadata:

import torch

state_dict = torch.load("model_100000.pt", map_location="cpu", weights_only=True)

Model construction is defined by model_config in the adjacent metadata file. See SHA256SUMS before using a downloaded checkpoint.

Checkpoint hashes

Checkpoint	SHA-256
dense/model_100000.pt	03f7f4d2f0307d0005afe96f1c148e1c654ea892eb6cb98993464faa35b606e3
vembedding/model_100000.pt	a67bc226d96402af8b9085e81ab09aaad000f71ec9ba53d4adbf39b27d6bc565
mome-a2-m12/model_100000.pt	057df263ae1bc947a55b7e732c1d64558b63a98ee73b1c2dd55756d955bfb434

Intended use

These checkpoints are research artifacts for reproducing the reported pretraining and evaluation results. They have not been instruction-tuned or safety-tuned for deployment.