ikarth/openpd-gpt2-v0

openpd-gpt2 v0

I trained a language model from scratch! It's called GPT-2.

The code is based on the well-known nanoGPT tutorials, with the twist that the training data is entirely public domain and CC0 text (and therefore, so is the model). The nature of the particular public-domain data means that books from 1929 and before make up a large part of the dataset, so it's not as versed in modern topics as the original internet-poisoned GPT-2 is. On the other hand no Reddit posts were consulted in training this model.

It's trained on about 2.5B tokens, with a relatively strict definition of public domain text; I have plans for the full 10B-token dataset (with an even stricter limit) at a later date, but the point of this model is that it was trained entirely on my personal 3090. (A solid workhorse, but send better GPUs for faster results.)

Note that the data cleaning was relatively minimal, and I didn't alter the original architecture much. It's a base model with no safety tuning, guardrails, or instruction tuning, use at your own risk. It also doesn't have a lot of code in its training data, so there's a limit to what it can do other than say bad words at you.

Better data and architecture can produce a better model; this one is mostly proof-of-concept.

Try it

In your browser: openpd-gpt2 demo (runs locally via transformers.js and the 8-bit ONNX weights in onnx/; about 170 MB download).

With transformers:

from transformers import pipeline

generator = pipeline("text-generation", model="ikarth/openpd-gpt2-v0")
print(generator("The old captain looked out across the harbor and")[0]["generated_text"])

Sampling defaults (generation_config.json): temperature 0.9, top-k 50, 120 new tokens.

Model

Architecture	GPT-2 small: 12 layers, 12 heads, 768 dims, 1,024-token context, tied embeddings (vocab padded to 50,304 for training; exported with 50,257)
Tokenizer	GPT-2 BPE (tiktoken gpt2). Not trained on open data (see Known issues)
Parameters	~124M
Code	personally hand-coded core model (build-nanogpt lineage, so copied from there). Run segments: eb0b3af (start, uncommitted changes present), fe60116 (resume, uncommitted changes present), 2928c74 (final segment, clean)

Training data

As near as I can tell the whole thing is public domain (or CC0).

Tiers pd + pd-policy: every document is public domain (by law, mostly US) or CC0, checked per document where the source records licenses (metadata.license / oa_license), per source otherwise, except Ubuntu IRC (pd-policy), whose public-domain status rests on Ubuntu's IRC policy rather than law or per-author dedication. Dataset openpd-v1 (tokenized with GPT-2 BPE), fingerprint fbca048694dc8f04… (sha256 of the token stream).

Source	License basis	Dataset train tokens	Share	Cleaning
Project Gutenberg (Common Pile)	PD (US)	2.176 B	22.3%	prose unwrapped, verse kept
Pre-1929 books, Internet Archive (Common Pile)	PD (US)	2.113 B	21.6%	de-hyphenated
Library of Congress Selected Digitized Books (storytracer)	CC0 / PD (US)	1.156 B	11.8%	de-hyphenated, prose unwrapped
Ubuntu IRC (Common Pile)	pd-policy: PD by Ubuntu IRC policy + join notice	0.943 B	9.7%	timestamps removed, nicknames pseudonymized, emails/IPs replaced
Caselaw Access Project (Common Pile)	PD (US)	0.724 B	7.4%	—
arXiv abstracts (Common Pile)	CC0	0.550 B	5.6%	—
US Government Publishing Office (Common Pile)	PD (US gov works)	0.469 B	4.8%	de-hyphenated
Biodiversity Heritage Library (Common Pile)	PD	0.435 B	4.5%	de-hyphenated
PubMed Central, CC0 subset (Common Pile)	CC0	0.410 B	4.2%	—
peS2o, CC0/PD subset (Common Pile)	CC0 / PD	0.301 B	3.1%	—
arXiv papers, PD/CC0 subset (Common Pile)	PD / CC0	0.284 B	2.9%	de-hyphenated
Regulations.gov (Common Pile)	PD (US gov works)	0.202 B	2.1%	de-hyphenated
Python Enhancement Proposals (Common Pile)	PD	0.003 B	0.0%	—
Total		9.77 B

What this model actually trained on (from src.run_provenance on the released checkpoint, step 4,499): 2.36 B tokens, 0.24 epochs of the dataset. Gutenberg 0.526 B, pre-1929 books 0.511 B, LoC 0.279 B, Ubuntu IRC 0.228 B, case law 0.175 B, arXiv abstracts 0.133 B, USGPO 0.113 B, BHL 0.105 B, PubMed CC0 0.099 B, peS2o 0.073 B, arXiv papers 0.069 B, regulations 0.049 B, PEPs 0.001 B. Attribution for openpd-v1 is proportional: the dataset interleaves sources evenly, so each source's share of the trained tokens matches its dataset share to about a percent.

Provenance records (per source: upstream revision, per-file sha256, license review, cleaning, model involvement) live in the dataset registry; the tokenized dataset's manifest lists every input file with its hash.

Model involvement in the data

No synthetic text. Upstream processing involved models only for text extraction and filtering: OCR for the book, BHL, and case-law sources; Grobid PDF parsing for peS2o; and Common Pile's filtering pipeline (classifiers not yet verified against the Common Pile paper).

Ubuntu IRC licensing (pd-policy)

Ubuntu's IRC Guidelines stated that the contents of all Ubuntu channels are "considered to be in the public domain", its IRC Terms of Service stated that participants in logged channels agree to public logging and processing, and logged channels showed an entry notice pointing to those terms (also documented by Uthus & Aha 2013). I think it's fine but I'm not a lawyer and in the interest of having 100% rigidly public domain training data I'm planning on a future training run that drops it from the mix.

Personal information

Ubuntu IRC is the only source with chat participants. Nicknames are replaced with stable pseudonyms (salted xxh64 seeding an Elite (1984)-style syllable generator; consistent within and across logs, not reversible without the private salt). Emails and IPv4 addresses are replaced with user@example.com / 192.0.2.1. Limits: only nicknames that speak in a given day's channel log are recognized; a nickname that is an ordinary English word is not rewritten inside sentences; pseudonyms keep the original nickname's trailing digits. A bug, found after training, left /me action lines unmatched: about 5% of action lines (0.03% of IRC lines) kept the real nickname of someone who only used /me that day. Fixed for future datasets.

Pre-release leak check. From 2,048 IRC-style samples (256 new tokens each), every nickname the model wrote was compared with the 836k real nicknames in the raw logs and their pseudonyms. In speaker/action position, 0.4% matched a distinctive real nickname that is never a pseudonym, and on inspection these were ordinary names, words, or prompt echoes, except one. Of the 1,000 most active distinctive nicknames, three appeared at all, six times in total: the handles of long-time Ubuntu developers, learned from mentions on days they didn't speak. No real email addresses, no IP addresses from the logs, and no phone numbers appeared. So the model has picked up a few prominent community members' public handles, rarely; it does not reproduce the chat participants at large.

Training

Steps	4,500 × 524,288 tokens = 2.36 B tokens
Schedule	WSD: 170 warmup steps, constant 6e-4 to step 3,599, linear decay to 6e-5 over the last 20%
Optimizer	AdamW (β 0.9/0.95, weight decay 0.1 on matrices), grad clip 1.0
Precision	bf16 autocast, TF32 matmuls, torch.compile
Hardware	1× RTX 3090, power-limited to 300 W
Throughput	~56–60k tokens/s
Training time	11.6 h of active training (paused overnight at step 3,599; restarted from the step-4,000 checkpoint after a stall)
Energy (GPU)	3.69 kWh (driver energy counter; excludes the rest of the machine)

Evaluation

Evaluated every 500 steps (and through the decay phase, shaded). Perplexity on the held-out 1930 novels passed GPT-2's between steps 3,750 and 4,000. HellaSwag's ActivityNet contexts climbed steadily from chance to 0.30, while its WikiHow contexts stayed at or below chance throughout. BLiMP was still rising when training stopped: this model is undertrained, and the next run (8.8B tokens, about 4× longer) should show how far these curves go.

Every number below comes from the same eval code, run the same way on this model and on OpenAI's GPT-2 small (124M parameters, trained on WebText: 2019 web pages linked from Reddit). Both use the same tokenizer, so perplexities are directly comparable. "Before decay" is the step-3,599 checkpoint, the end of the constant-learning-rate phase. summary.md in this repo has the table; per-paradigm and per-book results are in the project repo.

	openpd-gpt2 v0	v0 before decay (step 3,599)	GPT-2 small
Val loss (openpd-v1 val split)	3.418	3.675 (step 3,500)	n/a (different data)
HellaSwag acc_norm (10,042; chance 0.25)	0.255	0.248	0.298
ActivityNet contexts	0.302	0.295	0.353
WikiHow contexts	0.233	0.225	0.272
BLiMP, mean of 67 paradigms (chance 0.5)	0.736	0.711	0.823
morphology	0.880	0.847	0.930
syntax	0.673	0.639	0.791
semantics	0.681	0.692	0.745
syntax/semantics	0.686	0.669	0.785
Perplexity, 1930 novels (held out, 977k tokens)	26.1	31.7	28.8
Perplexity, WikiText-103 test (modern, 273k tokens)	81.1	104.3	22.8
CBT common nouns (contaminated, see below)	0.791	0.739	0.863

Period text: better than GPT-2. The 1930 set is eight novels first published in 1930, which entered the US public domain on 1 January 2026 and were added to Project Gutenberg after the training data was collected: The Maltese Falcon (Hammett), The Murder at the Vicarage (Christie), Strong Poison (Sayers), The Secret of the Old Clock (Keene), Vile Bodies (Waugh), Cakes and Ale (Maugham), Angel Pavement (Priestley), and Mr. Fortune's Maggot (Townsend Warner). A 13-gram overlap scan against the training data finds 2.2% of their 13-token spans on average (stock phrases), so they are effectively unseen. This model has lower perplexity than GPT-2 on all eight books (26.1 vs 28.8 overall).

Modern text: much worse. On WikiText-103 (Wikipedia articles), perplexity is 3.5× GPT-2's. Most of the training data predates 1930, and its modern sources are specialized (case law, government documents, science papers, IRC chat) rather than general encyclopedic or web prose. (Our WikiText numbers use whole-article sliding windows on detokenized text, not the GPT-2 paper's setup; compare them only within this table.)

HellaSwag is near chance (0.25). HellaSwag asks for the most plausible ending to a description of an everyday activity. Its ActivityNet contexts (video captions) score 0.30; its WikiHow contexts, about modern products, technology, and activities that public-domain text never describes, score 0.23, slightly below chance. This reflects the training data at least as much as the model size.

BLiMP: grammar is mostly learned. BLiMP gives the model pairs of sentences that differ in one grammatical detail and checks that it prefers the grammatical one. The gap to GPT-2 is smallest in morphology (agreement, irregular forms: 0.88 vs 0.93) and largest in syntax (0.67 vs 0.79).

CBT is contaminated, so its score doesn't measure generalization. The Children's Book Test (fill in a missing common noun from 10 candidates) is built from Project Gutenberg children's books, and Project Gutenberg is in the training data: 66% of the test passages' 13-token spans appear in it. The score is listed for completeness only. The named-entity subset is omitted.

The learning-rate decay helped across the board. The last 20% of training (learning rate decayed linearly to 1/10) cut val loss from 3.675 to 3.418 and improved every metric above except BLiMP semantics (0.692 → 0.681).

Samples (configs/prompts/openpd-v1.toml, 3 samples per prompt, temperature 0.9, top-k 50; v0 vs v0-before-decay vs GPT-2): v0 writes in a consistent 19th-century register (quoted dialogue, epistolary and naturalist prose), produces convincing IRC support chat with pseudonymous nicks, and is grammatical over short spans but drifts over longer ones. Given modern prompts ("My favorite video game is"), it switches into IRC chat, its only modern register. Example:

The old captain looked out across the harbor and called in the old man, and a man in the cabin called out to him. The young man was his only answer when he heard the old captain's answer. "Well, sir, for a moment or two you are so anxious to get away in a storm…"

Known issues

From src/data_report.py over the training data (openpd-v1_report.json):

• Data tables in government sources. Some USGPO / Regulations.gov documents are large numeric data dumps (tariff codes, emissions tables, columns of 0.00). ~0.3% of tokens are in documents where ≥30% of tokens contain digits (256 documents).
• Layout-heavy government text. 3.8% of tokens (31,011 documents) are in documents where ≥60% of tokens contain no letters: indentation runs, dot leaders, rule lines, column layouts. Much of it wraps real text; it was not normalized.
• Very long documents. 739 documents (6.8% of tokens) are longer than one 524k-token training batch, up to 7.2M tokens; each fills several consecutive batches by itself. In training this showed up as occasional very-low-loss steps and gradient-norm spikes (handled by clipping).
• OCR noise. Book, LoC, BHL, and case-law text is OCR; BHL in particular is noisy ("thoiit^ht jfood policy"). Greek and other non-Latin passages in older books are often garbled.
• No cross-source deduplication. The same work can appear in Project Gutenberg, the pre-1929 books, and LoC. 3 copies is acceptable but deduplication might improve things.
• Period bias. Most text predates 1930; the model knows little about the modern world, and its language and attitudes reflect its sources, including their prejudices.
• US-centric public-domain basis. "Public domain" here means public domain in the United States; some works may still be under copyright elsewhere.
• Tokenizer. The GPT-2 tokenizer was trained by OpenAI on WebText, which is not open data. The model's weights are trained only on open data, but its vocabulary is not.
• Ubuntu IRC is pd-policy, not pd. See "Ubuntu IRC licensing" above.
• Unverified upstream details. Common Pile's compilation license and the exact classifiers in its filtering pipeline are not yet verified against its paper.

Intended use

Experiments, comparisons with the original GPT-2, demonstrations of open-data training, winning petty arguments, and fun. Not for factual questions, advice, or any use that needs reliable or current information.

Attribution

Training data from the Common Pile v0.1 (Kandpal et al. 2025, The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text) and the Library of Congress Selected Digitized Books collection (compiled by Sebastian Majstorovic / storytracer).