nanochat-d20-chat — a chat model trained from scratch

Every stage was trained from random initialization with karpathy/nanochat (commit 92d63d4): no pretrained weights or tokenizer were used. Training run, scripts, logs and serving code: github.com/rsumit123/private-llm. Try it in the Space.

Architecture	nanochat GPT, 20 layers, width 1280, 10 heads, RoPE + QK-norm, 2,048-token context
Parameters	897M total: ~393M transformer weights + ~503M embedding tables (token + per-layer value embeddings)
Tokenizer	32,768-token BPE trained from scratch
Pretraining	3.48B tokens (NVIDIA ClimbMix), 3,320 steps, FP8, ~5.2 h on 1× H100, final val 0.755 bits/byte
Chat SFT	467 steps on SmolTalk + MMLU (×3) + GSM8K (×4), 47 min, val 0.296 bits/byte

Benchmarks

Measured with nanochat's evaluation code; the reference is the original nanochat d20 run (Oct 2025, ~3× more tokens).

Benchmark	This model	nanochat d20 reference
CORE (base model, 22 tasks, 500/task)	0.236	0.222
ARC-Easy	55.2%	38.8%
ARC-Challenge	42.9%	—
MMLU	35.2%	31.5%
HumanEval (pass@1)	11.0%	8.5%
GSM8K (first 400)	1.5%	4.6%
GSM8K after RL (first 400)	10.0%	7.6%

CORE with the same harness: GPT-2 small (124M) 0.123 · GPT-2 medium (355M) 0.192 · this model 0.236 · GPT-2 XL (1.5B, published) 0.257.

RL checkpoint (chatrl_checkpoints/d20): 75 steps of nanochat's GRPO-style REINFORCE on GSM8K (42 min on one H100). GSM8K 1.5% → 10.0% with ARC-Easy 55.4%, ARC-Challenge 43.9%, MMLU 35.2% (no regression). Load it with load_model("rl", ...); the chat (SFT) model in chatsft_checkpoints/d20 remains the default for general conversation.

Expect fluent English, multi-turn chat, simple explanations, short poems and stories. Do not expect reliable facts: it confidently hallucinates, and it is weak at math and long conversations.

Use

git clone https://github.com/karpathy/nanochat && cd nanochat && git checkout 92d63d4 && uv sync --extra cpu
uvx --from huggingface_hub hf download sumitkClasses/nanochat-d20-chat --local-dir ~/.cache/nanochat
NANOCHAT_BASE_DIR=~/.cache/nanochat .venv/bin/python -m scripts.chat_cli

An OpenAI-compatible server (Gradio UI + /v1/chat/completions) is in nanochat-d20/serve/app.py.