indictok‑acu — akshara‑constrained
A Unigram tokenizer for 22 Indian languages whose vocabulary is guaranteed almost never to split an akshara — the orthographic syllable a reader of a Brahmic script perceives as one character. A token that begins with a dependent vowel sign can only ever cut a consonant away from its own matra; this tokenizer removes that possibility rather than penalising it.
Full method and code: https://github.com/vivekvar-dl/indictok
The result
Like‑for‑like, 192k Unigram, same corpus, only the constraint differs:
| akshara‑splitting ↓ | avg fertility | |
|---|---|---|
| unconstrained | 8.58% | 2.43 |
| constrained (this) | 0.17% | 2.43 |
98% of akshara‑splitting removed at effectively no compression cost at this scale. 16,184 vocabulary pieces pruned, round‑trip verified.
For comparison, akshara‑splitting in other published tokenizers: Sarvam‑30B 3.24%, MuRIL 5.30%, Llama‑4 76%.
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("vivekvar/indictok-acu")
tok.encode("भारत தமிழ் മലയാളം")
How the constraint works
The design turned on one fact: Unigram is the right substrate and BPE is not. A Unigram vocabulary is an unordered set of pieces scored by log‑probability, so deleting an akshara‑splitting piece simply removes it from the Viterbi search and the segmentation reroutes through legal pieces. BPE merges form an ordered chain — dropping a piece breaks every merge that depended on it.
Bare combining marks (a lone matra) are kept: they are the atomic fallback units that let arbitrary text be represented at all. Only multi‑character pieces beginning with a restricted mark are pruned.
The restricted set itself is not asserted — it is derived from 50M words of real text per
script and ships with its evidence. One rule was rejected by that corpus (Malayalam
254‑ാം ordinal notation), and pieces beginning with that mark are correctly not
pruned.
What we do not claim
Akshara‑splitting rate has no established link to downstream model quality. It measures orthographic correctness, nothing more. Given that fertility and morphological alignment both fail to predict downstream performance in the literature, the honest expectation is that this does too. It is published as a correctness property, not a performance claim.
License
Apache‑2.0.