YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Sindhi-TinyLlama Tokenizer
This tokenizer is specifically designed for Sindhi-English multilingual training with TinyLlama.
Features
- Vocabulary size: 42,681 tokens
- Base: TinyLlama tokenizer (32,000 tokens)
- Additional: 10,681 Sindhi-specific tokens
- Training data: 30,000 examples from Sindhi-English corpus
- Languages: Sindhi (ุณฺูู), English, and mixed text
Performance
- Perfect roundtrip encoding/decoding
- Efficient tokenization: ~4.17 chars per token
- Handles 2048-sequence length (TinyLlama compatible)
- Batch processing optimized
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("jamalimubashirali/sindhi-tinyllama-tokenizer")
sindhi_text = "ูู ฺูช ุณฺูู ุฌู
ูู ุขูู"
encoded = tokenizer.encode(sindhi_text)
decoded = tokenizer.decode(encoded)
print(f"Original: {sindhi_text}")
print(f"Decoded: {decoded}")
print(f"Match: {decoded == sindhi_text}")
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model.resize_token_embeddings(len(tokenizer))
Training Details
Trained on 3.6M Sindhi-English sequences
2048 max sequence length
Preserves TinyLlama's original vocabulary
Optimized for both Sindhi and English text
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support