LMA mini project - monolingual Transformer LMs for Gujarati and Nepali
Ten checkpoints from five independently pretrained decoder-only Transformers,
written from scratch in PyTorch (no nn.Transformer*, no HuggingFace model
classes, no pre-built attention). Each model is ~25M parameters, trained for
one epoch over its own monolingual corpus.
Every run stores two checkpoints: best.pt (lowest validation loss) and
latest.pt (end of the run). Both are resume-capable - they carry model
weights, optimizer state, scheduler state, gradient-scaler state, step, token
count, both configs and the RNG states.
The five runs
| Folder | Language | Vocab | Layers | Positional | Params | Steps | Test PPL | Test BPB | What it is |
|---|---|---|---|---|---|---|---|---|---|
gujarati-model-h/16k-rope/ |
Gujarati (Model H) | 16,000 | 11 | rope | 25,625,856 | 15,826 | 51.62 | 0.5764 | reported model |
gujarati-model-h/12k-rope/ |
Gujarati (Model H) | 12,000 | 12 | rope | 25,860,864 | 16,418 | 44.64 | 0.5759 | vocabulary control |
gujarati-model-h/16k-nopos/ |
Gujarati (Model H) | 16,000 | 11 | none | 25,625,856 | 15,826 | 56.07 | 0.5885 | bonus ablation, no positional encoding |
nepali-model-l/16k-rope/ |
Nepali (Model L) | 16,000 | 11 | rope | 25,625,856 | 14,962 | 38.93 | 0.4247 | reported model |
nepali-model-l/12k-rope/ |
Nepali (Model L) | 12,000 | 12 | rope | 25,860,864 | 15,564 | 33.47 | 0.4236 | vocabulary control |
Perplexity is only comparable within a vocabulary. The 12k runs look better on perplexity purely because choosing 1 of 12,000 pieces is an easier question than 1 of 16,000. Bits-per-byte divides by bytes of text instead, so it is the number to compare across tokenizers.
Loading a checkpoint
import torch
blob = torch.load('best.pt', map_location='cpu', weights_only=False)
print(blob['model_config']) # architecture that produced these weights
print(blob['state']['step']) # training step
# rebuild with the DecoderLM class from the project repository
The model code, tokenizers, configs, training logs and full evaluation are in the project repository; these files are only the weights.