jack03/nanochat-1

nanochat d20 Model

This is a nanochat d20 language model trained from scratch following the nanochat pipeline.

Model Details

• Model Size: d20 (20 layers)
• Parameters: 560,988,160 (~561M parameters)
• Training Tokens: 11.2B tokens
• Architecture: GPT-style transformer
• Sequence Length: 2048 tokens
• Vocab Size: 65,536 tokens (custom BPE tokenizer)

Training Pipeline

The model was trained through three stages:

1. Base Training (Pretraining): Trained on web text data for 21,400 iterations
2. Midtraining: Domain adaptation for 773 iterations
3. Supervised Fine-Tuning (SFT): Chat fine-tuning for 682 iterations

Total Training Time: 38 hours 28 minutes on single H100 GPU

Performance

Metric	BASE	MID	SFT
CORE	0.2084	-	-
ARC-Challenge	-	0.3063	0.3166
ARC-Easy	-	0.4335	0.4226
GSM8K	-	0.0417	0.0531
HumanEval	-	0.0793	0.0793
MMLU	-	0.3258	0.3167
ChatCORE	-	0.1084	0.1080

Files Included

• model_000681.pt - PyTorch model checkpoint (SFT stage, step 681)
• meta_000681.json - Model metadata and configuration
• tokenizer/ - Custom BPE tokenizer files
• report.md - Full training report with detailed metrics

Usage

This model was trained using the nanochat codebase. To use it:

1. Clone the nanochat repository: git clone https://github.com/karpathy/nanochat
2. Download these model files
3. Place the model checkpoint in ~/.cache/nanochat/chatsft_checkpoints/d20/
4. Place the tokenizer files in ~/.cache/nanochat/tokenizer/
5. Run: python -m scripts.chat_cli -i sft

Notes

This is a micro-scale language model trained for educational purposes. As mentioned in the nanochat README, these smaller models "make a lot of mistakes, they are a little bit naive and silly and they hallucinate a ton, a bit like children."

License

MIT (following nanochat's license)

Citation

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}