tinygpt-shakespeare (kv-cache-throughput-bench)
A 3.307M-parameter char-level decoder-only transformer trained from scratch on Tiny Shakespeare. It exists to drive the benchmarks in the kv-cache-throughput-bench repo: a from-scratch KV cache, a cached-vs-uncached throughput sweep, and an eviction-policy study.
Architecture
- 4 layers, 4 heads, 256 embedding dim, block size 512, vocab 65 (characters)
- weight-tied token embeddings, pre-norm blocks, learned absolute positions
Training
- 2000 steps, batch 48, block 512, AdamW lr 3e-4
- 134 s on a single RTX 5090 Laptop GPU (CUDA 12.8)
- final validation loss 1.543, perplexity 4.68
Files
tinygpt.pt: state dict plus the config and char->id map
Usage
Clone the repo and load with the provided code:
from src.train import load_model
model, stoi, itos = load_model("tinygpt.pt")
Intended use and limits
Educational. A char model this small produces locally-coherent but repetitive text under greedy decoding. It is not meant for downstream language tasks.