sajalregmi4/arkios-tokenizer

Arkios Tokenizer — Nepali + English, 65,536 tokens

A byte-level BPE tokenizer built for bilingual Nepali–English language modelling. It reaches 1.69 tokens per word on Nepali — the lowest of any tokenizer we tested that is usable for generative language modelling, at roughly a quarter of the vocabulary of the next best.

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")   # or: from_pretrained("<repo>")
ids = tok.encode("नेपाली भाषामा ठूला भाषा मोडेलहरू प्रशिक्षण गर्नु महत्त्वपूर्ण छ।").ids
print(len(ids))                # 11
print(tok.decode(ids))         # exact round-trip

Why it is efficient on Nepali

Most byte-level BPE pipelines inherit GPT-2's pre-tokenization regex, in which a word is \p{L}+ — Unicode letters. Devanagari writes its vowels as combining marks (\p{Mn}, \p{Mc}), so that pattern cuts every Nepali word at every vowel sign:

नेपाली  under \p{L}+        →  ['न', 'े', 'प', 'ा', 'ल', 'ी']      6 pre-tokens
नेपाली  under [\p{L}\p{M}]+ →  ['नेपाली']                          1 pre-token

BPE only merges pairs within a pre-token, so those cuts are walls the merge table can never cross. Nepali fertility is floored at about 4.7 tokens/word before a single merge is learned, and no amount of Nepali training data moves it.

This tokenizer uses [\p{L}\p{M}]+, the same mark-aware word class as GPT-4o's o200k pattern. Trained on an otherwise identical corpus and vocabulary, that one change takes Nepali from 4.78 → 1.58 tokens/word.

Trained three 268M-parameter models differing only in tokenizer, the fixed one reaches 4.43% lower held-out Nepali bits-per-byte at equal compute — so this is not only a compression result.

Code, harness and full results: https://github.com/sajalregmi/arkios-tokenizer Paper: Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility — arXiv:2608.26449

Measured performance

FLORES-200 devtest, all tokenizers fed byte-identical NFC text. "Lossless" means decode(encode(x)) loses no characters.

Tokenizer	Vocab	Nepali tok/word	Nepali B/tok	English tok/word	Lossless
IndicBERTv2	250,000	1.58	11.54	1.24	yes¹
Arkios (this)	65,536	1.69	10.79	1.30	yes
BLOOM	250,680	1.72	10.56	1.25	yes
NLLB-200	256,204	1.92	9.48	1.40	no
o200k (GPT-4o)	200,019	2.32	7.84	1.23	yes
mT5	250,100	2.64	6.90	1.54	no
Sarvam-1	68,096	2.66	6.85	1.50	yes
Gemma-2	256,000	3.13	5.81	1.28	yes
Mistral NeMo	131,072	3.17	5.75	1.27	yes
Llama-3	128,256	3.76	4.84	1.24	yes
DeepSeek-V3	128,815	4.29	4.24	1.24	yes
Qwen2.5	151,665	6.58	2.77	1.26	yes
cl100k (GPT-4)	100,277	6.98	2.61	1.24	yes
GPT-2	50,257	10.97	1.66	1.28	yes

¹ IndicBERTv2 is lossless on this text but is a WordPiece encoder tokenizer: it does not preserve whitespace and has no byte fallback, so it is not a drop-in for generative modelling. We report it because it beats us and pretending otherwise would be dishonest.

Design

Algorithm	byte-level BPE, byte_fallback=true
Vocabulary	65,536 = 256 bytes + 65,254 merges + 26 special tokens
Special tokens	ids 65,510–65,535 (at the top, not the bottom)
Normalisation	NFC
Pre-tokenization	digit split (1–3 digits, ASCII and Devanagari ०–९) → mark-aware word split → ByteLevel(add_prefix_space=false, use_regex=false)
Checksum	0x3259671B (crc32 of the packed binary form)

Three deliberate choices worth knowing about:

Specials at the top. HuggingFace normally assigns special tokens the lowest ids. Here they occupy 65,510–65,535, so the learned vocabulary is a contiguous block starting at 0.

Digit pre-split. Runs of digits are cut into groups of at most three before BPE sees them, covering both ASCII 0-9 and Devanagari ०-९. Without this, BPE learns arbitrary number chunks — 2023 one token, 2024 two — giving numbers uneven representations. No token in this vocabulary contains four or more consecutive digits.

Byte fallback. Nothing is ever out-of-vocabulary, in any language or script. Round-trip is exact for every input after NFC.

Files

File	For
tokenizer.json	HuggingFace tokenizers, transformers, llama.cpp
arkios_tokenizer.bin	the C/CUDA trainer (packed, mmap-able)
byte_lengths.npy	per-token UTF-8 byte length, all 65,536 entries — what makes bits-per-byte computable

byte_lengths.npy is the file to reach for if you are comparing models across tokenizers. Per-token loss is not comparable when tokenizers segment text differently; bits-per-byte is, and it needs this table.

Training data

Public web text — English (FineWeb-Edu), Nepali (FineWeb-2 npi_Deva, AI4Bharat Sangraha and IndicCorpV2, Nepali Wikipedia), code (permissively licensed GitHub) and mathematics (OpenWebMath) — together with a private custom corpus that we do not redistribute.

Because a BPE vocabulary is derived from its training text, we audited it before release: all 65,536 tokens were decoded and the Devanagari subset (14,754 tokens) checked for anything that could encode non-public text. There are zero multi-word Devanagari tokens, and the longest entries are ordinary civic vocabulary (पूर्वप्रधानमन्त्री "former prime minister", विश्वविद्यालय "university", अन्तर्राष्ट्रिय "international").

Nothing in the paper depends on the private corpus: every experiment — the matched pairs, the sweep, the bound, the ecosystem census and the downstream ablation — uses only public data.

Reproducing the numbers

Every figure above is regenerated by one command, with no GPU and no HuggingFace account:

make paper-harness

It downloads FLORES-200, streams a public Nepali–English corpus, trains a matched control/treatment pair differing only in the pre-tokenization word class, and writes a machine-readable results.json recording the resolved Hub revision of every baseline and the sha256 of every corpus.

Limitations

• Nepali and English. Other Devanagari languages benefit from the mark-aware word class but were not in the training mixture; Hindi measures 2.01 tokens/word, Marathi and other Indic scripts are worse. This is not a pan-Indic tokenizer.
• Fixed vocabulary trade-off. At 65,536 entries, Nepali words compete with English for merge slots. English fertility is about 2% worse than a Nepali-free tokenizer of the same size would give.
• Fertility is compression. Better compression at a fixed context and cost is a real benefit, but it is not by itself evidence of better downstream quality.

Citation

@misc{regmi2026vowelsigns,
  title         = {Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on
                   Multilingual Tokenizer Fertility},
  author        = {Regmi, Sajal and Pudasaini, Siddhartha and Pun, Chetan Phakami},
  year          = {2026},
  eprint        = {2608.26449},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2608.26449},
  url           = {https://arxiv.org/abs/2608.26449}
}

---

Built at Karela Technologies Inc..