ManipuriGPT-Tokenizer-v1.0

The official SentencePiece BPE tokenizer for the ManipuriGPT foundation model ecosystem, trained on the deduplicated ManipuriGPT-Corpus-v1.0.

Key Metrics

Metric	Value
Algorithm	SentencePiece BPE
Vocabulary Size	32,000 subwords
Vocabulary Utilization	91.09%
Unknown Token Rate (<unk>%)	0.0000%
Avg Tokens / Sequence	160.3
Compression Ratio	3.653 chars/token
Token Entropy	11.64 bits

Special Tokens

Token	ID
<pad>	0
<unk>	1
<s>	2
</s>	3
<meitei>	4
<bengali>	5
<romanized>	6
<mask>	7

Supported Scripts

1. Meitei Mayek — Primary native Unicode script
2. Bengali Script — Historical and legacy literary texts
3. English / Latin — Romanized Manipuri and code-switching

Usage

import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.Load("tokenizer.model")

text = "ꯃꯅꯤꯄꯨꯔꯤ ꯂꯣꯟ ꯑꯁꯤ ꯑꯆꯧꯕ ꯂꯣꯟ ꯑꯃꯅꯤ"
tokens = sp.Encode(text, out_type=str)
print(tokens)

# Decode back
decoded = sp.Decode(tokens)
print(decoded)

Files

File	Description
tokenizer.model	SentencePiece binary model (BPE)
tokenizer.vocab	Full vocabulary with log-probabilities
tokenizer_config.json	Training metadata, special tokens, and evaluation metrics
special_tokens_map.json	HF-compatible special token mapping
tokenizer_qualitative_samples.json	50 qualitative encoding/decoding samples for manual inspection

Training Details

• Training Corpus: ManipuriGPT-Corpus-v1.0 (147,065 deduplicated sequences)
• Character Coverage: 100%
• Byte Fallback: Enabled (zero <unk> guarantee)
• Script-Aware Special Tokens: <meitei>, <bengali>, <romanized> for script-conditioned generation

Citation

@misc{manipurigpt_tokenizer_v10,
  author       = {ManipuriGPT Team},
  title        = {ManipuriGPT-Tokenizer-v1.0: SentencePiece BPE Tokenizer for Manipuri},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/nanskong/ManipuriGPT-Tokenizer-v1}
}

Related Resources

• Corpus: nanskong/ManipuriGPT-Corpus-v1.0
• GitHub: NansKong/ManipuriGPT-Foundational-Language-