letterpress - trained character-level GPTs

The trained checkpoints of letterpress, a character-level GPT built from scratch in PyTorch (following Karpathy's "Let's build GPT" lecture) and trained on nested tiers of a public-domain book corpus. These are base models: they continue text in the style of the books they read, one character at a time - they are not chatbots and do not follow instructions.

(The checkpoint files keep the model family's historical name byob-lm - the letterpress repo was formerly byob_llm - so paths, benchmark reports, and the companion dataset stay stable.)

Models

tier	params	layers x width	context	vocab	best val loss	held-out bpc	wikitext bpc	file
nano	0.82M	4 x 128	128	65	-	2.68	4.09	nano/shakespeare-nano.pt
medium	25.4M	8 x 512	256	97	1.2278	1.69	2.91	medium/byob-lm.best.pt
large	49.7M	10 x 640	384	198	1.1111	1.57	2.30	large/byob-lm.best.pt
xlarge	99.9M	14 x 768	512	199	0.9411	1.31	1.88	xlarge/byob-lm.best.pt
2xlarge	202M	16 x 1024	512	204	0.8725	1.17	1.71	2xlarge/byob-lm.best.pt

Held-out bpc = bits-per-char on public-domain books absent from every training tier; wikitext = out-of-domain. Both measured by the repo's bundled lm_bench harness (seed 1337, byte-normalized); full scorecards live in the repo under lm_bench/benchmarks/. Lower is better; the corpus tiers are nested, so every gain down the table is scale, not data luck.

How to use

The checkpoints are self-describing ({model, config, stoi, itos}, optimizer state stripped) but unpickle the letterpress GPTConfig, so you need the repo's source next to them:

git clone https://github.com/Novotarskyi/letterpress.git && cd letterpress
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
hf download disco-jack-basement/letterpress 2xlarge/byob-lm.best.pt --local-dir models

.venv/bin/python -m inference.sample   --ckpt models/2xlarge/byob-lm.best.pt --prompt "ROMEO:"
.venv/bin/python -m inference.interact --ckpt models/2xlarge/byob-lm.best.pt
cd lm_bench && ../.venv/bin/python -m lm_bench run --model byob:../models/2xlarge/byob-lm.best.pt --tasks core

Training provenance

• nano - Tiny Shakespeare (1.1M chars); the lecture demo, trained locally
• medium - medium tier (523M chars); Apple-Silicon MPS, ~1 h
• large - large tier (1.08B chars); Apple-Silicon MPS, ~3.5 h
• xlarge - xlarge tier (2.05B chars); rented H100 SXM, ~2.5 h
• 2xlarge - 2xlarge tier (4.26B chars); rented H200, ~12.6 h

Each <tier>/ folder here also carries the archive's provenance files (context.md with the full hyperparameters and val curve, corpus_stats.txt, corpus.lock.json, corpus_index.md) where the archive has them.

Data

Trained exclusively on the companion public-domain corpus disco-jack-basement/byob-pd-book-corpus (CC0; Project Gutenberg, Standard Ebooks, Internet Archive, Wikisource), curated under a strict, code-enforced no-Russian-content rule.

License

Code and weights: MIT. The training corpus is CC0 and published separately.