sayurio/bn-tokenizer-64k

Bengali / Bangla Tokenizer 64k

sayurio/bn-tokenizer-64k is a high-efficiency Byte-Pair Encoding (BPE) tokenizer with a 65,536 vocabulary size, engineered specifically for native Bengali Natural Language Processing, Large Language Models (LLMs), and Vision-Language Document OCR architectures.

Unlike multilingual tokenizers (such as LLaMA, Mistral, or Qwen) that split Bengali words and conjuncts into multiple sub-byte fragments, bn-tokenizer-64k treats compound grapheme clusters and common roots as unified tokens. This yields an average compression ratio of ~8.00 characters per token on formal Bengali text.

---

Key Features

• High Compression Rate: Compresses Bengali text down to ~8.00 characters per token, reducing sequence lengths by up to 60–70% compared to standard multilingual subword tokenizers.
• Compound Conjunct (যুক্তবর্ণ) Preservation: Accurately unifies complex ligatures (যেমন: ক্ষ, জ্ঞ, ঞ্চ, ণ্ড, ক্ত, ত্র, ঙ্গ) and multi-syllable roots (শিক্ষাক্রম, পাঠ্যপুস্তক, বুদ্ধিমত্তা) into single token representations.
• Canonical NFC Normalization: Strictly enforces Unicode Normalization Form C (NFC) to eliminate split vowel diacritics (e.g., decomposing ো into ে + া).
• Metaspace Word Boundary Tracking: Uses Metaspace ( ) pre-tokenization and decoding for 100% lossless text reconstruction with exact whitespace preservation.
• Invisible Character Handling: Retains Zero-Width Non-Joiner (\u200c) and Zero-Width Joiner (\u200d) to ensure correct rendering of Bengali spelling rules.

---

Tokenizer Specifications

Parameter	Value
Model Type	Byte-Pair Encoding (BPE)
Vocabulary Size	65,536
Pre-Tokenizer	Metaspace (Replacement: , Prepend: always)
Decoder	Metaspace (Replacement: , Prepend: always)
Normalizer	Unicode NFC + Strip
Training Corpus	~1.11 GB clean text (500,000+ deduplicated sentences)
Corpus Domains	Bengali Wikipedia, AI4Bharat Sangraha, National Journalism
Min Token Frequency	3

Special Tokens

Token	ID	Role
<pad>	0	Padding token
<s>	1	Beginning of sequence (BOS)
</s>	2	End of sequence (EOS)
<unk>	3	Unknown token

---

Compression & Tokenization Benchmarks

Example 1: Formal & Academic Text

• Input Text: স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড
• Tokens: ['<s>', ' স্বাধীন', ' বাংলাদেশের', ' জাতীয়', ' শিক্ষাক্রম', ' ও', ' পাঠ্যপুস্তক', ' বোর্ড', '</s>']
• Token IDs: [1, 1223, 830, 688, 25508, 156, 26780, 2455, 2]
• Efficiency: 56 characters $\rightarrow$ 7 content tokens (8.00 chars/token)

Example 2: Scientific & Technical Text

• Input Text: বিজ্ঞান ও প্রযুক্তির উৎকর্ষে কম্পিউটার প্রোগ্রামিং এবং কৃত্রিম বুদ্ধিমত্তা অত্যন্ত গুরুত্বপূর্ণ।
• Tokens: ['<s>', ' বিজ্ঞান', ' ও', ' প্রযুক্তির', ' উৎকর্ষ', 'ে', ' কম্পিউটার', ' প্রোগ্রামিং', ' এবং', ' কৃত্রিম', ' বুদ্ধিমত্তা', ' অত্যন্ত', ' গুরুত্বপূর্ণ।', '</s>']
• Token IDs: [1, 1959, 156, 7613, 18092, 71, 3209, 11838, 187, 7270, 23209, 2747, 8446, 2]
• Efficiency: 96 characters $\rightarrow$ 12 content tokens (8.00 chars/token)

---

Quickstart

Using Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sayurio/bn-tokenizer-64k")

text = "স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড"

# Encode
encoded = tokenizer(text)
tokens = tokenizer.convert_ids_to_tokens(encoded["input_ids"])

print("Tokens    :", tokens)
print("Token IDs :", encoded["input_ids"])

# Decode
decoded = tokenizer.decode(encoded["input_ids"], skip_special_tokens=True)
print("Decoded   :", decoded)

Using Hugging Face tokenizers (Fast Rust Backend)

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("sayurio/bn-tokenizer-64k")

output = tokenizer.encode("স্বাধীন বাংলাদেশ")
print("Tokens :", output.tokens)
print("IDs    :", output.ids)

---

Training Methodology

The tokenizer was trained on a multi-domain corpus of clean Bengali text comprising:

1. Academic & Encyclopedic Prose: Extracted from Bengali Wikipedia dump.
2. Formal & Legal Records: Derived from AI4Bharat's curated Bengali resources.
3. Contemporary Regional News: Modern journalistic and colloquial usage across politics, economics, and culture.

All text underwent strict pre-filtering:

• Removal of foreign scripts, stray HTML/Markdown tags, and corrupted byte markers.
• Retention strictly of the Bengali Unicode block (\u0980–\u09FF), Bengali numerals (০-৯), and standard punctuation.
• Unicode NFC canonical composition applied before pair ranking.

---

License

This tokenizer is released under the Apache 2.0 License. You are free to use, modify, and distribute it for both academic research and commercial applications.