openpd-gpt2 v1
I trained a language model from scratch! It's called GPT-2.
Well, technically it just has the architecture of GPT-2, with different training data and some minor tweaks. Consider it a demonstration of what is possible when training with the strictest limitations (though I should note there are some much bigger models that pursued similar goals, with varying levels of strictness).
The code is based on Karpathy's well-known nanoGPT tutorials, with the twist that the training data is entirely public domain and CC0 text (and therefore, so is the model). Otherwise, I stuck relatively closely to the original architecture of GPT-2. Which does mean that it is less optimized than it could be; model architectures have come along way since the distant past of 2019. The nature of the particular public-domain data means that books from 1929 and before make up a large part of the dataset, so it's also not as versed in modern topics as the original internet-poisoned GPT-2 is. On the other hand no Reddit posts were consulted in training this model.
It's replicating GPT-2-small (124M parameters), which let me train it from scratch in a semi-reasonable amount of time, which a semi-reasonable amount of data: 8.8B tokens of strictly public-domain and CC0 text: one full epoch of openpd-strict-v1, which I've also uploaded, though the files are relatively accessible, being public domain and all.
The earlier version, v0, was trained on a 2.5B token subset of the data, plus Ubuntu IRC. I dropped Ubuntu IRC for two reasons: the point was to do the absolute maximalist definition of "public domain" and because the formatting wasn't really helping much (as the evals demonstrate). I trained v0 on my personal 3090. I could have done the same for v1 if I wanted to wait a couple of days, but I needed the machine for other things (and wanted to test my setup in the cloud) so I trained v1 on 4 rented H100s in 85 minutes. Both approaches used roughly the same amount of power, as it turned out.
On held-out prose from 1930 its perplexity is 40% lower than the original GPT-2's (17.4 vs 28.8); on modern Wikipedia text it is about twice GPT-2's. It writes period prose well and knows little about the world after about 1930. A better eval would be to find Project Gutenberg books scanned after the datasets' cutoff, rather than just 1930, but it works well enough for a model that is just barely aware there was a recent stock crash.
It's a base model with no safety tuning, guardrails, or instruction tuning; use at your own risk. It doesn't have much in the way of code, so the risk is more likely to be in the realm of racism, sexism, telling bad stories, etc.
Try it
In your browser: openpd-gpt2 demo (runs locally
via transformers.js and the 8-bit ONNX weights in onnx/; about 170 MB download).
With transformers:
from transformers import pipeline
generator = pipeline("text-generation", model="ikarth/openpd-gpt2-v1")
print(generator("The old captain looked out across the harbor and")[0]["generated_text"])
Sampling defaults (generation_config.json): temperature 0.9, top-k 50, 120 new tokens.
Model
| Architecture | GPT-2 small: 12 layers, 12 heads, 768 dims, 1,024-token context, tied embeddings (vocab padded to 50,304 for training; exported with 50,257) |
| Tokenizer | GPT-2 BPE (tiktoken gpt2). Not trained on open data (see Known issues) |
| Parameters | ~124M |
| Code | same model code as v0 (build-nanogpt lineage). Commit 8202e0a, clean |
| Weights | model.safetensors (fp32); onnx/model_quantized.onnx (8-bit, for transformers.js) |
Training data
Tier pd only: every document is public domain (by law, mostly US) or CC0, checked per document
where the source records licenses (metadata.license, peS2o's oa_license) and per source
otherwise. Dataset openpd-strict-v1
(tokenized with GPT-2 BPE), fingerprint e777bfedc277d972β¦ (sha256 of the token stream). It is
openpd-v1 (v0's data) minus Ubuntu IRC, with the same sources, sampling, and cleaning.
| Source | License basis | Trained tokens | Share | Cleaning |
|---|---|---|---|---|
| Project Gutenberg (Common Pile) | PD (US) | 2.168 B | 24.7% | prose unwrapped, verse kept |
| Pre-1929 books, Internet Archive (Common Pile) | PD (US) | 2.104 B | 24.0% | de-hyphenated |
| Library of Congress Selected Digitized Books (storytracer) | CC0 / PD (US) | 1.152 B | 13.1% | de-hyphenated, prose unwrapped |
| Caselaw Access Project (Common Pile) | PD (US) | 0.721 B | 8.2% | none |
| arXiv abstracts (Common Pile) | CC0 | 0.548 B | 6.2% | none |
| US Government Publishing Office (Common Pile) | PD (US gov works) | 0.461 B | 5.2% | de-hyphenated |
| Biodiversity Heritage Library (Common Pile) | PD | 0.432 B | 4.9% | de-hyphenated |
| PubMed Central, CC0 subset (Common Pile) | CC0 | 0.408 B | 4.6% | none |
| peS2o, CC0/PD subset (Common Pile) | CC0 / PD | 0.300 B | 3.4% | none |
| arXiv papers, PD/CC0 subset (Common Pile) | PD / CC0 | 0.282 B | 3.2% | de-hyphenated |
| Regulations.gov (Common Pile) | PD (US gov works) | 0.202 B | 2.3% | de-hyphenated |
| Python Enhancement Proposals (Common Pile) | PD | 0.003 B | 0.0% | none |
| Total | 8.782 B |
Trained tokens are exact (from per-shard source counts, provenance_step_016749.json in this
repo): 16,750 steps Γ 524,288 tokens, 0.996 epochs, no document seen twice.
Model involvement in the data
No synthetic text. Upstream processing involved models only for text extraction and filtering: OCR for the book, BHL, and case-law sources; Grobid PDF parsing for peS2o; and Common Pile's filtering pipeline (classifiers not yet verified against the Common Pile paper).
Personal information
No chat, forum, or social-media text. Case law and government documents name people as public records do; no additional PII scrubbing was applied.
Training
| Steps | 16,750 Γ 524,288 tokens = 8.78 B tokens (one epoch) |
| Schedule | WSD: 170 warmup steps, constant 6e-4 to step 13,399, linear decay to 6e-5 over the last 20% |
| Learning rate | 6e-4, chosen by a 1,000-step comparison on this data: 6e-4 reached val loss 4.824, 1.8e-3 reached 4.974 |
| Optimizer | AdamW (Ξ² 0.9/0.95, weight decay 0.1 on matrices), grad clip 1.0 |
| Precision | bf16 autocast, TF32 matmuls, torch.compile |
| Hardware | 4Γ H100 80GB SXM (RunPod), DDP, micro-batch 64 |
| Throughput | ~1.85M tokens/s |
| Training time | 85 min (one uninterrupted run) |
| Energy (GPU) | 3.79 kWh (driver energy counters; excludes the rest of the machine) |
| Cost | ~$28 including setup, a smoke test, and the learning-rate comparison |
Val loss (openpd-strict-v1 val split): about 3.20 at the end of the constant-LR phase (3.194 at step 13,000, 3.204 at 13,250) β 3.025 final.
Evaluation
Every number below comes from the same eval code, run the same way on all three models. All use the same tokenizer, so perplexities are directly comparable. Evaluated every 1,000 steps (chart).
| openpd-gpt2 v1 | openpd-gpt2 v0 | GPT-2 small | |
|---|---|---|---|
| Training data | 8.8B tokens, pd only | 2.4B tokens, pd + Ubuntu IRC | WebText (2019 web pages linked from Reddit) |
| Perplexity, 1930 novels (held out, 977k tokens) | 17.4 | 26.1 | 28.8 |
| Perplexity, WikiText-103 test (modern, 273k tokens) | 48.6 | 81.1 | 22.8 |
| BLiMP, mean of 67 paradigms (chance 0.5) | 0.771 | 0.736 | 0.823 |
| morphology | 0.893 | 0.880 | 0.930 |
| syntax | 0.725 | 0.673 | 0.791 |
| semantics | 0.710 | 0.681 | 0.745 |
| syntax/semantics | 0.725 | 0.686 | 0.785 |
| HellaSwag acc_norm (10,042; chance 0.25) | 0.261 | 0.255 | 0.298 |
| ActivityNet contexts | 0.314 | 0.302 | 0.353 |
| WikiHow contexts | 0.236 | 0.233 | 0.272 |
Period text: much better than GPT-2. The 1930 set is eight novels first published in 1930, which entered the US public domain on 1 January 2026 and were added to Project Gutenberg after the training data was collected (13-gram overlap with the training data: 2.2% on average, stock phrases). v1 has lower perplexity than GPT-2 on every one:
| Novel | v1 | v0 | GPT-2 |
|---|---|---|---|
| The Maltese Falcon (Hammett) | 13.4 | 19.3 | 20.8 |
| The Secret of the Old Clock (Keene) | 14.2 | 22.6 | 24.1 |
| The Murder at the Vicarage (Christie) | 16.7 | 24.7 | 27.8 |
| Cakes and Ale (Maugham) | 17.3 | 27.1 | 29.5 |
| Vile Bodies (Waugh) | 17.6 | 26.2 | 29.7 |
| Angel Pavement (Priestley) | 17.8 | 26.3 | 29.3 |
| Strong Poison (Sayers) | 20.1 | 30.6 | 33.4 |
| Mr. Fortune's Maggot (Townsend Warner) | 23.7 | 37.2 | 40.0 |
Modern text: better than v0, still about twice GPT-2. Most of the training data predates 1930, and its modern sources are specialized (case law, government documents, science papers). Dropping Ubuntu IRC (v0's only casual modern text) did not hurt: WikiText perplexity fell from 81.1 to 48.6. (Our WikiText numbers use whole-article sliding windows on detokenized text, not the GPT-2 paper's setup; compare them only within this table.)
Grammar (BLiMP): closer to GPT-2, but leveled off. BLiMP checks that the model prefers the grammatical sentence of a minimal pair. It rose to ~0.76 by step 10,000 and gained only ~0.01 after that; the remaining gap is mostly syntax (0.725 vs 0.791).
HellaSwag: still near chance. Its ActivityNet contexts (video captions of everyday activities) reach 0.31; its WikiHow contexts, about modern products and activities, stay slightly below chance for the whole run. 3.7Γ more training data than v0 didn't move them, which suggests the limit is what the data covers, not model size or training length.
The learning-rate decay (last 20% of steps) cut val loss from about 3.20 to 3.025, 1930-novel perplexity from 20.2 to 17.4, and WikiText from 58.8 to 48.6.
Samples (configs/prompts/openpd-v1.toml, 3 samples per prompt, temperature 0.9, top-k 50;
v1 vs v0 vs GPT-2). v1 stays on topic more than v0 (the natural-history prompt now gets plumage
and egg counts rather than a family story) and writes consistent period dialogue. With no chat in
its data, modern prompts become period fiction instead of v0's IRC chat; the IRC-style prompt
<mike> anyone know how to turns into prose about grammar. Example:
The old captain looked out across the harbor and called in the mate of the ship.
"Where is it?" asked the midshipman politely. "Are the schooners safe?"
"I think the schooners are safe," answered the skipper. "I have seen the schooners off the land now. Have them taken on board, and put in a fresh supply at least."
Known issues
From the dataset's content report (openpd-strict-v1_report.json):
- Very long documents. 739 documents (7.5% of tokens) are longer than one 524k-token training batch, up to 7.2M tokens; each fills several consecutive batches by itself. In training this showed up as stretches of very low training loss and temporary validation-loss bumps (e.g. 3.28 β 3.36 β back below 3.28 around step 8,500).
- Layout-heavy government text. 4.2% of tokens are in documents where β₯60% of tokens contain no letters (indentation, dot leaders, column layouts). 0.3% are numeric data tables.
- OCR noise. Book, LoC, BHL, and case-law text is OCR; BHL in particular is noisy. Greek and other non-Latin passages in older books are often garbled.
- No cross-source deduplication. The same work can appear in Project Gutenberg, the pre-1929 books, and LoC.
- Period bias. Most text predates 1930; the model knows little about the modern world, and its language and attitudes reflect its sources, including their prejudices.
- US-centric public-domain basis. Some works may still be under copyright outside the US.
- Tokenizer. The GPT-2 tokenizer was trained by OpenAI on WebText, which is not open data. The model's weights are trained only on open data, but its vocabulary is not.
- 8-bit ONNX weights (browser demo) are slightly worse than the full-precision model: on a held-out 1930 passage, loss 2.945 vs 2.928, same top prediction 89.5% of the time.
- Unverified upstream details. Common Pile's compilation license and the exact classifiers in its filtering pipeline are not yet verified against its paper.
Intended use
Experiments, comparisons with the original GPT-2, demonstrations of open-data training, winning petty arguments, and fun. Not for factual questions, advice, or any use that needs reliable or current information.
Attribution
Training data from the Common Pile v0.1 (Kandpal et al. 2025, The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text) and the Library of Congress Selected Digitized Books collection (compiled by Sebastian Majstorovic / storytracer).
- Downloads last month
- -