notyost/claudius-purus-110m

Claudius Purus 110M

A generative language model with a knowledge cutoff of 0 BCE. Trained from scratch — architecture, tokenizer, and weights — exclusively on text composed before the Common Era. It has never seen a word written after antiquity. Ask it who rules, and the only possible answer is ancient.

To our knowledge (verified against HF, arXiv, and the vintage-LLM community as of mid-2026), this is the first generative model with a genuine pre-CE training cutoff; the earliest prior from-scratch cutoff was 1875 CE.

The five facts

• 25.85M training tokens — the entire defensibly-pre-CE, license-clean corpus we could assemble: Ancient Greek, Latin, Akkadian, Sumerian, Biblical Hebrew (consonantal), and Classical Chinese. 953 works spanning ~21 centuries of composition (22nd c. BCE – 1st c. BCE).
• Every source dated and auditable: inclusion is governed by a public dating manifest (per-author dates, confidence, and rationale — including the exclusions: Strabo, Ovid, the Periochae, pseudo-guwen chapters…). "Pre-CE" is a config file, not a vibe.
• 110.9M parameters, GPT-2-small architecture (12L/768/12H, 1024 ctx, 32k byte-level BPE trained on this corpus only), fp16, 16 epochs, seed 1453. Exported with exact (0.00e+00) logit parity to HF format.
• Best val loss 3.783 (ppl 44.0) on held-out ancient works (by-work split). Per-language numbers below — read the caveat first.
• Total training cost: about $2 of rented H100 time. The corpus, not compute, is the binding constraint: there is no more pre-CE text to scale onto.

What it is — and is not

It is a rhapsode, not an oracle: BabyLM-scale results apply — strong grammatical and stylistic competence, near-zero world-model. It writes period-true Ciceronian prose, hexameter-shaped Greek with genuine Homeric formulae, psalm-register consonantal Hebrew, annalistic Classical Chinese, and cuneiform-transliteration Akkadian — and it hallucinates freely within those registers. It answers questions the way a base model does: by continuing text. (Short Latin questions tend to summon Roman comedy; short Greek questions summon Platonic dialogue. This is a feature: the model is a detector for the register-distribution of surviving ancient literature.)

Known biases, stated plainly: the corpus is what antiquity's elites wrote and later ages preserved — literate, male, canonized. The model simulates surviving literature, not ancient populations. No benchmark can validate "thinks like 50 BCE"; our evals measure linguistic fidelity only. Hebrew is the unpointed consonantal text (the vowel points are medieval additions).

Evaluation

lang	val tokens	NLL/token	per-token ppl
Akkadian	110,579	2.537	12.6
Ancient Greek	88,155	3.781	43.9
Latin	36,470	4.739	114.3
Classical Chinese	45,397	6.209	497.2
ALL	280,601	3.808	45.1

⚠ Per-token perplexity is not comparable across languages: Akkadian transliteration splinters into ~4.4 predictable syllable-tokens per word, while each Chinese character is one whole-morpheme token. Per-word NLL inverts the ranking. Both views ship in the eval notes. The validation split contains no Hebrew or Sumerian works (hash-split coverage gap, noted for v2).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("notyost/claudius-purus-110m")
model = AutoModelForCausalLM.from_pretrained("notyost/claudius-purus-110m")
ids = tok("Non ad usum. Non ad probandum. Ludendi causa.",
          return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

Few-shot works and is period-appropriate: prime it with a master–pupil exchange (Platonic dialogue, catechism form) and it will hold the format for a beat or two. Expect drift; enjoy the drift.

Family

• claudius-purus-110m (this model): 6 languages, all training sources CC BY-SA / CC0 / public domain → released CC BY-SA 4.0.
• claudius-purus-110m-g: adds Vedic Sanskrit (GRETIL) and the Epic of Gilgamesh (eBL) — both CC BY-NC-SA sources — and is therefore released CC BY-NC-SA 4.0, honoring the licensors' one request. Research and play: use -g. Anything commercial: use this one.

Provenance & thanks

See ATTRIBUTION.md. Built on the shoulders of Perseus, Open Greek & Latin, ORACC, Tanach.us, and Kanripo — decades of scholarly digitization made a model like this possible for the price of a sandwich. First words, spoken at release through this very export, given its own founding incantation ("Not for use. Not for proof. For the joy of the making."):

Quid faciam? … Nunc ad omnia. Ridiculum exoritur: nam hic in te est. "What shall I do? … Now — to everything. The ridiculous arises: for it is here, in you."