Time-locked English 1780-1830 (590M, QK-norm)
A GPT-style model trained from scratch only on English printed 1780-1830: 28 layers, width 1280, 20 heads, QK-norm (each head's queries and keys layer-normalised, Amendment 39), 592,849,664 parameters, 48,931 steps of 524,288 tokens (one pass over 25.7B tokens), peak learning rate 2.5e-4, bf16 on two H100s. Final validation loss 2.1917 at step 48,931 (the 297M reached 2.3248).
QK-norm is not in the GPT-2 class, so the weights are native PyTorch with their own loader:
from modeling_timelocked import load # this repo's file
from tokenizers import Tokenizer
model, tok = load(), Tokenizer.from_file("tokenizer.json")
model.pt is the training run's own state dict; model.safetensors the same weights;
milestones/stepNNNNNN_model.pt the weights every 5,000 steps; loss_curves.csv every step.
Benchmark (110 held-out tests, data/benchmark/alm_mid.json in the GitHub repository): the
primary method picks the right author 0.982 of the time with 0.218 false matches.
About
Part of an open, pre-registered study of the authorship, borrowing and dating of the
earliest Book of Mormon text (https://github.com/lifehug/book-of-mormon-authorship; pre-registration https://osf.io/6ugxc/). The project does not argue for or
against divine inspiration. Everything is published so it can be checked and redone; the corpus
texts we agreed to keep private are not, and the repository's docs/replication.md gives the
steps to obtain the same access.
Licence: CC BY-NC 4.0. Training data: English printed 1780-1830 (Internet Archive, Harvard
Institutional Books 1.0, HathiTrust public-domain volumes, Text Creation Partnership), with the
Book of Mormon and everything after 1830 removed; see the base model cards
(etherfuse/timelocked-1830-125m). Contains derivatives of Harvard Institutional Books 1.0,
Institutional Data Initiative, Harvard Law School Library; used for noncommercial research.
Checksums: every file's SHA-256 is in the GitHub repository, data/manifests/hf_<repo>.jsonl.
- Downloads last month
- -