etherfuse/timelocked-1830-125m

Time-locked English 1780-1830 (125M pilot)

A GPT-style language model trained from scratch only on English printed 1780-1830, with the Book of Mormon and everything after 1830 left out. It is the pilot model of an open, pre-registered study of the authorship, borrowing and dating of the earliest Book of Mormon text (https://github.com/lifehug/book-of-mormon-authorship; pre-registration https://osf.io/6ugxc/). The project does not argue for or against divine inspiration; the model exists to measure how ordinary a passage was for its time.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("etherfuse/timelocked-1830-125m")
model = AutoModelForCausalLM.from_pretrained("etherfuse/timelocked-1830-125m")
ids = tok("And it came to pass that", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.9)[0]))

Model

Architecture	pre-norm decoder (GPT-2 class): 14 layers, width 768, 12 heads, learned positions, tied embeddings, exact GELU, no biases (zero in this export)
Parameters	124,475,136
Context	1,024 tokens
Tokenizer	byte-level BPE, 32,000 entries, fit on a 1 GB seeded sample of the training split only
Training	one pass over 25.7B tokens (48,931 steps of 524,288 tokens); AdamW (0.9, 0.95), weight decay 0.1, peak learning rate 3e-4, 2% warmup, cosine to 10%; bf16 on one H100
Validation loss	2.5148 at step 48,931 (20M tokens from 989 held-out volumes)

Loss curves: loss_curves.csv. The exported weights were checked against the native model: conversion_check.json (largest logit difference on held-out tokens).

Data

202,816 volumes printed 1780-1830 (corrected dates), newspapers excluded:

Source	Volumes	Terms
Internet Archive	172,264	public-domain scans
Harvard Institutional Books 1.0	17,681	Institutional Data Initiative terms (noncommercial; derivatives only if significantly transformed, with attribution)
HathiTrust (public-domain, open access)	10,056	HathiTrust researcher agreement
Text Creation Partnership (EEBO, ECCO, Evans)	2,815	CC0

Removed before training: every catalogued copy of the Book of Mormon and every volume holding 20 or more of its original 12-token strings (the most any kept volume shares is 12); every volume by an author in the study's validation benchmark (1,489), by a named candidate author (19), and the known-answer texts with their copies (225). 0.5% of volumes (989) held out for validation.

Attribution. Contains derivatives of Harvard Institutional Books 1.0, Institutional Data Initiative, Harvard Law School Library. Released under CC BY-NC 4.0 to respect that source's noncommercial terms. No training text or token files are distributed.

Limits

• 1780-1830 English as digitised: OCR errors, long s, and the period's spelling and typography are in the training data, and the model reproduces them.
• Small by current standards and trained once; not instruction-tuned; it will produce confident period prose on any topic, including false and offensive content typical of its sources.
• The training set is weighted to what libraries kept and digitised (religion, law, history, sermons), not to how people spoke.