genaforvena/english-char257

English char257 — the control expert

The same architecture, hyperparameters and tokenizer as finnegans-fake-char257, trained on ordinary 19th-century English. It exists to be the only controlled comparison in the project.

What it is

architecture	GPT-2 (6 layers, 6 heads, 384 embd)
parameters	10,942,848
vocabulary	257
context	512 tokens
trained	6,000 iters, batch 24, lr 0.0006, dropout 0.2
best val loss	1.2004
end-of-run val loss	1.2065

This repo carries the best-val checkpoint, not the end of the run.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("english-char257")
model = AutoModelForCausalLM.from_pretrained("english-char257")

ids = tok("riverrun, past Eve and Adam's,", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True,
                     temperature=0.9, top_k=100, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Sampled at pack time, seed 0, so this is reproducible rather than curated:

riverrun, past Eve and Adam's, and Rather de Bourgh, but I am not careless to inquire to do what they are not

One thing to know about the tokenizer

It is a byte-level BPE with add_prefix_space: true, so decoding inserts a single leading space that was not in your prompt:

tok.decode(tok("riverrun").input_ids)   # -> ' riverrun'

Round-trip is otherwise exact — verified on 20k characters of the training corpus, byte for byte, once that one space is accounted for. Prompt with a leading space, or strip one from the output; do not go looking for a lossy character.

Why a second model exists at all

finnegans-fake-char257 reaches val 1.8840. This one reaches 1.2004 on the same architecture, the same hyperparameters, the same 1.31MB of text (matched to the Wake's character count within 0.2%) and a byte-identical tokenizer file. It is the one loss comparison in this project where nothing else differs, so for once the difference means what it looks like: ordinary English is more predictable than Finnegans Wake.

The corpus is eight public-domain novels from Project Gutenberg (wake/prepare_english.py --fetch), which is why this repo — alone among the set — is CC0 and public.

It is also one half of a failed experiment worth reading: multiplying this model's distribution with the Wake model's was supposed to force a portmanteau, a string readable as two lexicons at once. It does the exact opposite, monotonically, and the reason is definitional rather than a tuning failure — a geometric mean is a veto, and ordinary English vetoes precisely the characters that would have made a coinage. See the project README.

A bug worth keeping in the card

Every loss figure this project first published was produced by a broken objective. batch() returned nanoGPT-style pre-shifted labels while transformers shifts labels itself, so the shift happened twice and every run learned to predict token t+2 from position t. It never raised: training ran, loss fell, the curves looked plausible, and the inflated perplexity supported a confident story about the Wake being statistically incompressible. That story was a property of the bug. Only the generated text exposed it, by coming out looking like every second character had been deleted. After the fix, 400 steps beat the 6000 broken ones.

The figures in this card are post-fix.

What it is not

It is not good, and it is not trying to be. The aim was to find out what a language model does when the only language it has ever seen is one book — whether anything resembling English survives, and whether the machine can coin words the way Joyce did rather than quote the ones he already coined. Do not use it for anything.

Code: https://github.com/genaforvena/finnegans-fake — CC0.