nanoGPT 107M (FineWeb-Edu)
106.8M-parameter GPT-2-style decoder, trained from scratch with nanoGPT.
- Architecture: 12 layers, 11 heads, d_model 704, context 1024, GPT-2 BPE (vocab 50304), tied embeddings
- Data: ~2.0B tokens of FineWeb-Edu (sample-10BT), 1 epoch
- Training: 3750 iters x 524k tokens, AdamW, lr 6e-4 cosine to 6e-5, bf16, 1x RTX 5080 (~5h)
- Validation loss: 3.41
Limits
Base model: it continues text, it does not follow instructions or answer questions. Output is fluent but often factually wrong, and repetitive at low temperature. No safety tuning.
Use
pip install torch tiktoken safetensors
python chat.py # type a prompt; /t 0.9 temperature, /n 200 length, /k 50 top_k
model.py is nanoGPT's. Weights are in model.safetensors (lm_head is tied to transformer.wte).
- Downloads last month
- 23