michaelchenkj/datadecide-dclm-530m-53b

DataDecide 530M × 53.0B (DCLM-baseline)

Autoregressive OLMo-1 530M trained for one pass over 53B DCLM-baseline tokens (DataDecide 530M rung, token:param = 100). Checkpoints, EasyBench evals (metrics + per-example generations), logs, and training scripts are in this repo.

Companion run: michaelchenkj/mdlm-dclm-530m-53b.

Recipe

Architecture	OLMo-1, d=1344, 16L, 16H, SwiGLU, RoPE, RMSNorm, seq 2048, vocab 50280
Tokens	53.0B (token:param = 100)
Steps	57,786
Global batch	448 sequences (917,504 tokens/step)
LR	2.77e-3 cosine, 578-step warmup, α=0.01, AdamW 0.9/0.95, wd 0.05
Hardware	8×H100, DDP, microbatch 8 (7 accum)
Data	allenai/DataDecide-data-recipes fastdclm/.../allenai uint16 npy
Seed	6198

Final FineWeb val NLL (OLMo tokenizer): 2.845. Headline EasyBench table is in RESULTS.md.

Layout

• weights/stepN/ — bf16 safetensors + tokenizer, every 1,000 steps and the last step (step57786)
• step57786-unsharded/ — full resume (model.pt, optim.pt, train.pt)
• evals/ — metrics.jsonl, per-step metrics.json, and per-example .jsonl generations
• logs/ — training and launch logs
• scripts/ — train_530m.py, launch.sh, download/pack, and eval sweeps
• config.yaml — OLMo trainer config for this run
• train_data/global_indices.npy — shuffled index used for the 1-epoch pass

Load a checkpoint

from huggingface_hub import snapshot_download
from pathlib import Path

root = Path(snapshot_download("michaelchenkj/datadecide-dclm-530m-53b", allow_patterns=["weights/step57786/**"]))
# weights/step57786/{model.safetensors,config.json,tokenizer.json,...}

Training data is not stored here (~100GB). Download the official DataDecide DCLM shards with scripts/download_dclm.py.

Train

python scripts/download_dclm.py
python scripts/pack_val.py
bash scripts/launch.sh