submarat/gpt2-small-fineweb-edu-10b-checkpoints

GPT-2 Small (124M) — raw training checkpoints (10B FineWeb-Edu)

Intermediate + final training checkpoints from the reproduction described at submarat/gpt2-small-repro (write-up: https://submarat.github.io/reproducing-gpt2-small/).

The ready-to-use model is at submarat/gpt2-small-fineweb-edu-10b; these are the raw checkpoints kept for reproducibility and training-dynamics / interpretability work (e.g. the capability-emergence and induction-head sweeps).

• ckpt_0002000.pt … ckpt_0018000.pt: every 2000 optimizer steps
• ckpt_final.pt: step 19073 (10B tokens)
• Tokens per step = 524,288, so step N ≈ N × 0.524B tokens.

Format

Each file is a torch.save dict in this repo's custom Transformer format (not a HuggingFace model):

import torch
from gpt2_small import Transformer, TransformerConfig  # from the code repo

ckpt = torch.load("ckpt_final.pt", map_location="cpu", weights_only=False)
cfg = TransformerConfig(**ckpt["config"])
model = Transformer(cfg)
model.load_state_dict(ckpt["model"])
# ckpt also holds: "optimizer" (AdamW state), "step"

To use the final model with transformers instead, load the converted GPT2LMHeadModel from the model repo linked above.