IPA-GPT

Small / medium / large GPT models pretrained on an 8-language corpus in three input representations — text, romanized, and IPA (ipa_stripped). Modded-nanoGPT architecture; vocab 100k (BPE). See the paper for training details, corpus, and evaluation.

Layout: <representation>/<size>/ each contains best_state.pt, config.json, and the tokenizer that model was trained with.

Usage

from load import load_pretrained
from tokenizers import Tokenizer
import torch

model = load_pretrained("text/medium", device="cuda")   # eval mode
tok = Tokenizer.from_file("text/medium/bpe-8lang-text-100k-tokenizer.json")

ids = torch.tensor(tok.encode("some input text").ids, dtype=torch.long, device="cuda")
logits = model.logits(ids)          # (1, T, 100096) softcapped logits
nll = model.loss(ids[:-1], ids[1:]) # teacher-forced cross-entropy

Requirements: pip install -r requirements.txt (PyTorch ≥ 2.1). Runs on CPU or GPU; no torch.compile / FlexAttention needed. modeling_gpt.py is a portable re-implementation of the training model's eval path (identical outputs within the model's sliding window).

Notes

• Document separator is token id 2 (<|eos|>) — emit it to mark end-of-text. (config.json also records eot_token.)
• Tokenizers are the exact ones used in training; a known BPE-merge issue and its (minimal) impact are documented in the paper.

Each checkpoint contains only the model weights under the model key. To fine-tune, start a fresh optimizer. Training metadata (step, recorded val loss, % trained) is in each config.json.