Tiny Storyteller
A 5.9M-parameter GPT-style language model trained from scratch (random init, no pretrained weights)
on 19,800 TinyStories plus 3,000 short synthetic stories about two characters, Momo the monkey and
Ella the elephant (1,000 unique stories, each repeated 3 times, generated by momo_stories.py), as a learning project.
Use case: a bedtime story starter for young kids.
- Architecture: 6 transformer blocks, 4 heads, 256-dim embeddings, 256-token context
- Tokenizer: byte-level BPE, 4,096 tokens, trained on the same data
- Training: 1500 steps, batch 32, AdamW, cosine schedule
- Final validation loss 2.202 (perplexity 9.0) on 200 held-out TinyStories (no Momo stories in the validation set)
The full build, explained step by step, is in tiny_llm_from_scratch.ipynb in this repo.
Load with plain PyTorch: GPT(cfg).load_state_dict(torch.load("model.pt")).
- Downloads last month
- 16