gpt2-124m-ckpt

A from-scratch reproduction of OpenAI's GPT-2 (124M) in PyTorch — architecture, distributed training, and evaluation all written by hand. Trained on 10B tokens of DCLM-baseline for 4 epochs on 8×RTX 4090D (~13 h, ~$27).

📦 Full code, training pipeline, and analysis: https://github.com/zzkai098/reproduce-gpt2

Results

Matches official GPT-2 124M on downstream benchmarks (same eval both models, acc_norm) — on par overall, nominally ahead on 3 of 5 multiple-choice tasks:

Benchmark	Ours	Official GPT-2
ARC-Easy	40.8	39.2
Winogrande	51.0	48.4
ARC-Challenge	22.7	22.5
HellaSwag	32.9	34.3
OpenBookQA	26.8	27.4
WikiText-103 (ppl ↓)	34.4	25.4

At 124M, both models are only clearly above chance (25% / 50%) on ARC-Easy and HellaSwag. WikiText perplexity trails official GPT-2 as a training-distribution effect (DCLM general web vs GPT-2's WebText), not a capability gap.

Architecture

12-layer decoder-only Transformer · 768 hidden · 12 heads · 1024 context · 124M params · 50,257 vocab. Faithful enough that the official Hugging Face GPT-2 weights load into it unchanged (logit parity 6.9e-5).

Training

76,292 steps (4 epochs × 10B tokens) · 0.5M-token batch · AdamW (β 0.9/0.95, wd 0.1) · cosine LR 6e-4 → 6e-5 · bf16 · Flash Attention · torch.compile · 8-GPU DDP.

Usage

This is a raw PyTorch checkpoint using a custom GPT class, so it needs the model code from the repo:

git clone https://github.com/zzkai098/reproduce-gpt2
pip install -e reproduce-gpt2

import torch
from huggingface_hub import hf_hub_download
from gpt2.model import GPT

path = hf_hub_download("zzkai098/gpt2-124m-ckpt", "model_final.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
model = GPT(ckpt["config"]).eval()
model.load_state_dict({k.replace("_orig_mod.", ""): v for k, v in ckpt["model"].items()})

Or generate with the repo's script (see the repo README Quick start).

Files

model_final.pt — final checkpoint (step 76,291)
snapshots/ — per-epoch and early-training snapshots (for eval-over-training curves)
log.txt, run.log — training logs
eval_results.txt — head-to-head benchmark output

Limitations

A faithful reproduction of a 2019-era 124M model — small by today's standards:

Reasoning is near chance — clearly above random only on ARC-Easy / HellaSwag.
Coherence degrades over long generations (grammatical but rambly).
English-only, trained on filtered web text (DCLM) — inherits its biases.
Base model — not instruction-tuned, chat-formatted, or safety-aligned.

License

Downloads last month: -; Downloads are not tracked for this model. How to track

zzkai098
/

gpt2-124m-ckpt