LMA Phase 1 — Hindi and Nepali tokenizers
Two independent sentencepiece tokenizers, one per language, trained from scratch for a pair of ~25M-parameter decoder-only Transformers.
They share nothing — not merges, not pieces, not a vocabulary file. Hindi and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two corpora and the two vocabularies separate is the central constraint of the project rather than an afterthought.
| language | algorithm | vocab | trained on | corpus tokens |
|---|---|---|---|---|
| Hindi | unigram | 10,000 | 2,016,377 of 4,032,755 lines (50%, sampled at random) | 655.2M |
| Nepali | unigram | 10,000 | 2,539,821 of 5,079,643 lines (50%, sampled at random) | 558.9M |
How these were chosen
Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k) across two algorithms (BPE, unigram), for both languages. Every model read the same 10% random sample of its language's training split, so differences between them are differences between models rather than between samples.
Unigram beat BPE at every vocabulary size in both languages — ten paired comparisons, no exceptions, by 1.2–2.7% fertility.
Vocabulary 10,000 was selected over 16,000 despite 16k tokenizing better.
The embedding matrix is vocab_size × 512 parameters against a 25M budget, so
16k spends 33% of the whole model on a lookup table while 10k spends 20%. The
selection rule is the smallest vocabulary whose fertility is within 8% of the
best, which trades ~5–7% fertility for ~3.1M parameters returned to the
transformer layers.
Hindi — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces. Nepali — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces.
UNK is impossible. byte_fallback=True decomposes any unseen character
into byte tokens, so the UNK count is zero by construction rather than by luck.
Training data
| Hindi | Nepali | |
|---|---|---|
| final corpus | 5,094,185 docs / 2.588B chars | 6,755,888 docs / 2.513B chars |
| training split | 3,564,832 docs / 1.798B chars | 4,726,832 docs / 1.752B chars |
Splits are document-level, stratified by source, 70/15/15 by characters, with one whole source held out per language as an unseen-domain test set. The tokenizers saw the training split only.
The corpora are at meet5568/lma_datasets.
Usage
import sentencepiece as spm
from huggingface_hub import hf_hub_download
path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("भारत एक विशाल देश है।", out_type=str))
Limitations
Unigram's memory scales with total corpus length — it builds a suffix array over every character — at roughly 10.6 GB of RAM per GB of text, measured. The full training split would need about 49 GB, so the final unigram models were trained on 50% of it. A control experiment found fertility differing in the fourth decimal place between 81.6% and 100% of the lines, so this is a hardware limit rather than a quality one, but it is stated rather than implied.