omarkilbi/dante-gpt-scratch

DanteGPT (from scratch)

A 10.65M-parameter character-level GPT trained from scratch on Dante Alighieri's Divina Commedia, using nanoGPT. It starts knowing nothing, not even letters, and learns tercets, dialogue and medieval Italian spelling from a single book.

Companion model: omarkilbi/dante-qwen2.5-0.5b-lora (Qwen2.5-0.5B fine-tuned with LoRA on the same text).

Training

Data	La Divina Commedia di Dante: Complete, Project Gutenberg eBook #1000 (public domain), Gutenberg header/footer and table of contents removed
Split	first 90% train, last 10% validation (end of Paradiso)
Tokenizer	character-level, ~80 characters
Architecture	GPT, 6 layers, 6 heads, 384 embedding dim, 256-character context
Hardware	free Google Colab T4, ~25 minutes
Best checkpoint	step 1250 (overfitting starts afterwards)

Results

Model	Val bits per character
Qwen2.5-0.5B (base, no training)	2.220
DanteGPT from scratch (this model)	2.005 (val loss 1.369 nats/char)
Qwen2.5-0.5B + LoRA	1.791

Usage

git clone https://github.com/karpathy/nanoGPT && cd nanoGPT
mkdir -p out-dante-char data/dante_char
# download ckpt.pt into out-dante-char/ and meta.pkl into data/dante_char/
python sample.py --out_dir=out-dante-char --start="Nel mezzo del cammin" --num_samples=3

Sample

e tre volte fiammeggiava la benza, del ciel, che son l’arco avviso, al suo vestito fia di gran che si parte.

Limitations

A learning project. The model imitates the form of the Commedia (tercets, elisions, «dialogue») but often invents words and has no real meaning. ckpt.pt is a PyTorch pickle that also contains the optimizer state; only load files you trust.