OTK-BPE: Optimized BPE Tokenizers

This repository hosts the OTK-BPE family of production-grade Byte-Level BPE (BBPE) tokenizers for regional and multilingual language models. It now covers Swahili, Kinyarwanda, and a merged French + Kinyarwanda + English + Swahili vocabulary — each available at three vocab sizes: 50k, 100k, and 150k — so you can pick the tradeoff between compactness and coverage that fits your use case.

(Companion repo: olaverse/otk-bpe-50k covers the Nigerian-languages family — Yoruba, Igbo, Hausa, Pidgin, and a unified Naija tokenizer, all fixed at 50k.)

Which one should I use?

Your situation	Use
Only need Swahili	sw-150k (best fertility and entity handling in this family; drop to sw-50k/sw-100k only if embedding-table size is tightly constrained)
Only need Kinyarwanda	kin-150k
Need French, Kinyarwanda, English, and Swahili in one model	merged-150k
Storage/embedding-table size is a hard constraint	Step down to -100k or -50k in the same language — fertility degrades gradually, not a cliff (see Benchmarks)

150k is the recommended default across all three families — in every benchmark below, fertility and entity-handling both improved monotonically from 50k → 100k → 150k, with no exceptions. There's no size in this range where a smaller vocab wins outright.

Key Features

• Byte-Level BPE (BBPE): maps raw UTF-8 bytes to printable characters — 0.00% Out-Of-Vocabulary, zero [UNK] tokens, by construction.
• Diacritic Preservation & Normalization: NFC normalization inside the pre-tokenization chain, so accents and combining diacritics don't get split into decomposed code points.
• Code-Mixed English Support: the Swahili and Kinyarwanda tokenizers each blend a English-Wikipedia component into training; the merged tokenizer treats English as a full first-class language rather than a minority blend.
• Emoji & Symbol Merging: curated emoji vocab is injected during training so common emoji merge into single tokens instead of fragmenting.

Tokenizer Models in This Repository

All loaded from olaverse/otk-bpe, selected via the subfolder argument.

Subfolder	Languages	Vocab Size
sw-50k / sw-100k / sw-150k	Swahili	50,000 / 100,000 / 150,000
kin-50k / kin-100k / kin-150k	Kinyarwanda	50,000 / 100,000 / 150,000
merged-50k / merged-100k / merged-150k	French + Kinyarwanda + English + Swahili	50,000 / 100,000 / 150,000

Performance Benchmarks

Fertility = average tokens per word (lower is better). Entity fragmentation = share of capitalized/likely-proper-noun words split into more than one token (lower is better) — this metric predicts downstream NER difficulty before any model is even trained.

Swahili benchmark uses the MasakhaNEWS Swahili test split (42,494 entity-candidate words checked) — a real, curated held-out set, never seen during training.

Kinyarwanda, French, and English benchmarks use a held-out, non-training slice of the same streaming sources (FineWeb-2 / Wikipedia) rather than a curated test set — MasakhaNEWS does not include Kinyarwanda (only the related-but-distinct Rundi/Kirundi), so no equivalent curated benchmark exists for that language yet. Treat these three languages' numbers as directionally reliable, not as rigorously verified as the Swahili figures.

Swahili

Tokenizer	Vocab Size	Fertility	Entity Fragmentation
otk-bpe sw-150k	150,000	1.210 (Best!)	0.341 (Best!)
otk-bpe sw-100k	100,000	1.233	0.435
AfroXLMR	250,002	1.597	0.441
otk-bpe merged-150k	150,000	1.264	0.519
otk-bpe sw-50k	50,000	1.285	0.626
otk-bpe merged-100k	100,000	1.302	0.672
mBERT	~119,547	2.071	0.575
otk-bpe merged-50k	50,000	1.393	0.822
GPT-4o (o200k_base)	200,019	1.841	0.768
GPT-4 (cl100k_base)	100,277	2.462	0.820

Kinyarwanda

Tokenizer	Vocab Size	Fertility	Entity Fragmentation
otk-bpe kin-150k	150,000	1.377 (Best!)	0.334 (Best!)
otk-bpe kin-100k	100,000	1.409	0.420
otk-bpe merged-150k	150,000	1.465	0.475
otk-bpe kin-50k	50,000	1.483	0.561
otk-bpe merged-100k	100,000	1.523	0.567
otk-bpe merged-50k	50,000	1.662	0.709
AfroXLMR	250,002	2.495	0.718
mBERT	~119,547	2.702	0.715
GPT-4o (o200k_base)	200,019	2.189	0.798
GPT-4 (cl100k_base)	100,277	2.798	0.854

Kinyarwanda is where general-purpose multilingual tokenizers are weakest across the board — even the smallest dedicated tokenizer here (kin-50k) beats every baseline on fertility, and kin-150k beats every baseline on both metrics simultaneously.

French

Tokenizer	Vocab Size	Fertility	Entity Fragmentation
otk-bpe merged-150k	150,000	1.378 (Best!)	0.428
otk-bpe merged-100k	100,000	1.409	0.502
GPT-4o (o200k_base)	200,019	1.478	0.479
otk-bpe merged-50k	50,000	1.486	0.620
AfroXLMR	250,002	1.597	0.434
mBERT	~119,547	1.616	0.388 (Best!)
GPT-4 (cl100k_base)	100,277	1.720	0.578

English

Tokenizer	Vocab Size	Fertility	Entity Fragmentation
mBERT	~119,547	1.429 (Best!)	0.285 (Best!)
GPT-4o (o200k_base)	200,019	1.436	0.452
otk-bpe merged-150k	150,000	1.443	0.558
GPT-4 (cl100k_base)	100,277	1.460	0.496
otk-bpe merged-100k	100,000	1.500	0.659
AfroXLMR	250,002	1.538	0.425
otk-bpe merged-50k	50,000	1.632	0.803

English is the merged tokenizer's weakest relative showing — expected, since none of the 4 languages get a dedicated vocabulary in the merged design, and mBERT/AfroXLMR both have far larger, English-rich vocabularies to draw on. merged-150k still lands within ~1% of mBERT's fertility despite covering 3 additional languages in the same vocabulary.

Reading the entity fragmentation numbers honestly

At 150k vocab, OTK-BPE tokenizers win entity fragmentation outright, not just fertility, for Swahili and Kinyarwanda — sw-150k and kin-150k both have the lowest fragmentation rate of every tokenizer tested, dedicated or general-purpose. The vocab-size increase from 50k to 150k didn't just narrow the gap to larger multilingual tokenizers on this metric, it closed and then reversed it for these two languages specifically.

The exceptions are French and English, where mBERT — not AfroXLMR — has the best entity handling (0.388 and 0.285 respectively), edging out even the merged tokenizer. This makes sense: mBERT's vocabulary, while smaller than AfroXLMR's, was built with heavy exposure to major European-language Wikipedia text including large volumes of proper nouns, and English/French are exactly where that shows. The merged tokenizer's own English and French entity handling (0.558 and 0.428) is competitive but not best-in-class for those two languages specifically — a real, honest tradeoff of covering 4 languages in one shared vocabulary rather than dedicating full budget to either.

Net read: don't assume general-purpose tokenizers automatically win on entity handling just because they have larger vocabularies — that held at the smaller 50k vocab size tested earlier in this project, but stopped holding once vocab size reached 150k for the languages that needed it most.

How to Use

Installation

pip install tokenizers transformers huggingface_hub

Method A: Standard Transformers Loading

from transformers import AutoTokenizer

# Load the Swahili tokenizer at your chosen size
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")

text = "Habari yako? Leo ni siku nzuri sana 😊"
inputs = tokenizer(text)

print("Tokens:", tokenizer.tokenize(text))
print("IDs:", inputs["input_ids"])
print("Decoded:", tokenizer.decode(inputs["input_ids"]))

# Kinyarwanda
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="kin-150k")

# Merged (French + Kinyarwanda + English + Swahili)
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="merged-150k")

Method B: Lightweight Raw BPE Loading

from olaverse import Tokenizer

# Supports: "sw-50k", "sw-100k", "sw-150k", "kin-50k", "kin-100k", "kin-150k",
#           "merged-50k", "merged-100k", "merged-150k"
tokenizer = Tokenizer("sw-150k")

ids = tokenizer.encode("Habari yako? Leo ni siku nzuri sana 😊")
print("IDs:", ids)
print("Decoded:", tokenizer.decode(ids))

Datasets Used for Training

Source	License	Role
FineWeb-2 (swh_Latn, kin_Latn, fra_Latn)	ODC-By	Web text
Wikipedia (sw, rw, fr, en)	CC BY-SA 4.0	Encyclopedic corpora
MasakhaNEWS (Swahili train split)	CC BY 4.0	News text, dense with named entities
olaverse/qg-passages-multi	Apache-2.0	Domain coverage for Swahili, French, and English
Wikipedia (English)	CC BY-SA 4.0	Code-mixed blending (sw/kin) or full component (merged)
Curated Emoji Registry	—	Common social-media emoji

MasakhaNEWS Swahili test split was excluded from all training corpora and reserved exclusively for the fertility/entity benchmarks above.

Links

• Companion repo: Nigerian languages (otk-bpe-50k)
• Olaverse Library Docs

License

Apache-2.0. Training data licenses noted per source above; please retain attribution to upstream datasets.