TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Paper • 2305.07759 • Published • 50
TinyGPT Visualizer: bpe-small
Checkpoints for TinyGPT Visualizer, an app that steps through a forward and backward pass of a small GPT and shows every number at every stage.
- 6 layers × 8 heads, d_model 256, tied embeddings, 5.8M parameters
- 4,096-token BPE vocabulary, 100-token context
- Trained on TinyStories for 20,000 steps;
runs/bpe-small/holds checkpoints from across training, for replay data/bpe4096/val.bin: tokenized validation text, used by the app's neuron search
The app downloads these files on first run. You don't need to fetch them by hand.
Data
The model was trained on TinyStories by Ronen Eldan and Yuanzhi Li (TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, 2023), which they published openly for others to use.
License
The model weights and the app are MIT licensed. The tokenized validation text (data/*/val.bin) is
derived from TinyStories and remains under the dataset's license,
CDLA-Sharing-1.0.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support