mini-LLM — a 30M-parameter transformer trained overnight on a laptop

A decoder-only GPT written from scratch in plain PyTorch. No transformers, no accelerate, no fastai — the model, the tokenizer training and the training loop are all in the repository.

Code: https://github.com/vous99/mini-llm

Trained on one MacBook Pro (M5 Pro, 24 GB, MPS) in a single overnight run.

Results

Metric	Value
Best val loss	1.1527 (perplexity 3.17)
Loss at initialisation	9.01 = ln(8192), the uniform-over-vocabulary baseline
Steps	7,561
Tokens seen	0.74B (1.4 epochs of TinyStories V2)
Training time	~9 hours of compute at 23–25K tokens/s
Parameters	29,893,120 (25.7M non-embedding)

Architecture

Modern rather than 2017: what separates this from the original transformer paper is where most of the interest lies.


Layers	8
Model width	512
Heads	8 (head_dim 64)
Context	512 tokens
Vocabulary	8,192, byte-level BPE trained on the corpus itself
Positional encoding	RoPE (rotary), not learned embeddings
Normalisation	RMSNorm, pre-norm
Feed-forward	SwiGLU, hidden 1408 (8/3 × width, rounded to a multiple of 64)
Attention	`F.scaled_dot_product_attention`, causal
Embeddings	tied — one matrix serves both input and output
Dropout	0.0

The 8,192-token vocabulary is a deliberate choice: GPT-2's 50,257 would put a 25M-parameter embedding table inside a 30M-parameter model.

Usage

git clone https://github.com/vous99/mini-llm && cd mini-llm
pip install -r requirements.txt

python -c "
from huggingface_hub import hf_hub_download
import shutil, os
os.makedirs('ckpt', exist_ok=True); os.makedirs('data', exist_ok=True)
shutil.copy(hf_hub_download('vous99/mini-llm', 'best.pt'), 'ckpt/best.pt')
shutil.copy(hf_hub_download('vous99/mini-llm', 'tokenizer.json'), 'data/tokenizer.json')
"

python sample.py
python sample.py --prompt "Once upon a time, a little robot" --n 3
python chat.py                # terminal REPL
python serve.py               # browser playground on 127.0.0.1:8890

tokenizer.json is required. It is the BPE trained alongside the model; without it the weights emit token ids, not text.

Training details


Optimizer	AdamW, lr 6e-4, betas (0.9, 0.95), grad clip 1.0
Weight decay	0.1 on matrices, 0 on 1-D parameters (norms)
LR schedule	linear warmup 200 steps → cosine decay to 10% of peak
Batch	24 × 8 gradient accumulation × 512 tokens = 98,304 tokens per step
Precision	bfloat16 autocast

Train and validation loss stayed within 0.01–0.03 of each other for the whole run — no overfitting, which is why dropout is 0.

Limitations

TinyStories only. The corpus is synthetic children's stories using a deliberately small vocabulary. The model writes fluent, coherent short stories in that register and nothing else. It has no world knowledge, cannot answer questions and cannot follow instructions.
512-token context.
No instruction tuning, no RLHF. This is a base model in the most literal sense.
Not safe for production. It is a study artifact for understanding how a transformer trains.

Checkpoint contents

best.pt is a torch.save dict with model, cfg (the GPTConfig), iter, best_val and val_loss. Load it with sample.py from the GitHub repository.

Downloads last month: -; Downloads are not tracked for this model. How to track

vous99
/

mini-llm