sayurio/bn-tokenizer-16k

Bengali / Bangla Tokenizer 16k

sayurio/bn-tokenizer-16k is a compact Byte-Pair Encoding (BPE) tokenizer with a 16,384 vocabulary size, engineered specifically for native Bengali Natural Language Processing, Small Language Models (SLMs), edge-deployed LLMs, and Vision-Language Document OCR architectures where embedding-table size is a hard constraint.

Unlike multilingual tokenizers (such as LLaMA, Mistral, or Qwen) that split Bengali words and conjuncts into multiple sub-byte fragments, bn-tokenizer-16k keeps common roots, suffixes, and high-frequency conjuncts as unified tokens. This yields an average compression ratio of ~5.50 characters per token on formal Bengali text — at a quarter of the embedding-table footprint of a 64k vocabulary.

---

Key Features

• Compact Vocabulary (16,384): Quarters the embedding and output-projection parameters relative to a 64k tokenizer, making it suitable for on-device inference, small models (≤500M parameters), and tight VRAM budgets.
• Frequent Conjunct (যুক্তবর্ণ) Preservation: Unifies the most common ligatures (যেমন: ক্ষ, জ্ঞ, ঞ্চ, ণ্ড, ক্ত, ত্র, ঙ্গ) and multi-syllable roots (শিক্ষাক্রম, বুদ্ধিমত্তা) into single token representations. Rarer conjuncts fall back to character-level composition.
• Canonical NFC Normalization: Strictly enforces Unicode Normalization Form C (NFC) to eliminate split vowel diacritics (e.g., decomposing ো into ে + া).
• Metaspace Word Boundary Tracking: Uses Metaspace (▁) pre-tokenization and decoding for 100% lossless text reconstruction with exact whitespace preservation.
• Invisible Character Handling: Retains Zero-Width Non-Joiner (\u200c) and Zero-Width Joiner (\u200d) to ensure correct rendering of Bengali spelling rules.
• Smaller Output Head: A 16k softmax is ~4× cheaper per decoding step than a 64k softmax, which matters for token-by-token generation latency.

---

Tokenizer Specifications

Parameter	Value
Model Type	Byte-Pair Encoding (BPE)
Vocabulary Size	16,384
Pre-Tokenizer	Metaspace (Replacement: ▁, Prepend: always)
Decoder	Metaspace (Replacement: ▁, Prepend: always)
Normalizer	Unicode NFC + Strip
Training Corpus	~1.11 GB clean text (500,000+ deduplicated sentences)
Corpus Domains	Bengali Wikipedia, AI4Bharat Sangraha, National Journalism
Min Token Frequency	3

Special Tokens

Token	ID	Role
<pad>	0	Padding token
<s>	1	Beginning of sequence (BOS)
</s>	2	End of sequence (EOS)
<unk>	3	Unknown token

---

Compression & Tokenization Benchmarks

Example 1: Formal & Academic Text

• Input Text: স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড
• Tokens: ['<s>', ' স্বাধীন', ' বাংলাদেশ', 'ের', ' জাতীয়', ' শিক্ষা', 'ক্র', 'ম', ' ও', ' পাঠ্যপুস্তক', ' বোর্ড', '</s>']
• Token IDs: [1, 1223, 830, 118, 688, 3141, 902, 47, 156, 7642, 2455, 2]
• Efficiency: 56 characters $\rightarrow$ 10 content tokens (5.60 chars/token)

Example 2: Scientific & Technical Text

• Input Text: বিজ্ঞান ও প্রযুক্তির উৎকর্ষে কম্পিউটার প্রোগ্রামিং এবং কৃত্রিম বুদ্ধিমত্তা অত্যন্ত গুরুত্বপূর্ণ।
• Tokens: ['<s>', ' বিজ্ঞান', ' ও', ' প্রযুক্তি', 'র', ' উৎকর্ষ', 'ে', ' কম্পিউটার', ' প্রোগ্রামিং', ' এবং', ' কৃত্রিম', ' বুদ্ধিমত্তা', ' অত্যন্ত', ' গুরুত্ব', 'পূর্ণ', '।', '</s>']
• Token IDs: [1, 1959, 156, 4102, 62, 10092, 71, 3209, 5931, 187, 7270, 8140, 2747, 6610, 1203, 24, 2]
• Efficiency: 96 characters $\rightarrow$ 15 content tokens (6.40 chars/token)

Variant Comparison (same corpus, same preprocessing)

Variant	Vocab	Avg. chars/token (formal)	Embedding params @ d=2048
bn-tokenizer-16k	16,384	~5.5–6.5	~33.6M
bn-tokenizer-32k	32,768	~6.5–7.5	~67.1M
bn-tokenizer-64k	65,536	~8.0	~134.2M

(Embedding params = vocab × hidden dim, input embedding only; output projection adds the same again unless untied.)

---

Quickstart

Using Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sayurio/bn-tokenizer-16k")

text = "স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড"

# Encode
encoded = tokenizer(text)
tokens = tokenizer.convert_ids_to_tokens(encoded["input_ids"])

print("Tokens    :", tokens)
print("Token IDs :", encoded["input_ids"])

# Decode
decoded = tokenizer.decode(encoded["input_ids"], skip_special_tokens=True)
print("Decoded   :", decoded)

Using Hugging Face tokenizers (Fast Rust Backend)

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("sayurio/bn-tokenizer-16k")

output = tokenizer.encode("স্বাধীন বাংলাদেশ")
print("Tokens :", output.tokens)
print("IDs    :", output.ids)

Training Methodology

The tokenizer was trained on a multi-domain corpus of clean Bengali text comprising:

• Academic & Encyclopedic Prose: Extracted from Bengali Wikipedia dump.
• Formal & Legal Records: Derived from AI4Bharat's curated Bengali resources.
• Contemporary Regional News: Modern journalistic and colloquial usage across politics, economics, and culture.

All text underwent strict pre-filtering:

• Removal of foreign scripts, stray HTML/Markdown tags, and corrupted byte markers.
• Retention strictly of the Bengali Unicode block (\u0980–\u09FF), Bengali numerals (০-৯), and standard punctuation.
• Unicode NFC canonical composition applied before pair ranking.

License This tokenizer is released under the Apache 2.0 License. You are free to use, modify, and distribute it for both academic research and commercial applications.