haparoot/fat5-zwnj-base

Reduced Persian-centric version of mt5-base which is optimized for Persian (Farsi) while retaining English and Arabic capabilities by Removing the unused vocabulary.

• Original vocabulary: 250,112 tokens
• New vocabulary: 29,900 tokens (12% of original)
• Removed: ~220,212 tokens (88% of vocabulary) Model Size Impact:
• Original model: 582M parameters
• New model: 244M parameters (42% of original size)
• Major reduction in embedding layers (66% of original model size)

The new vocabulary includes:

• Top 1K tokens from original mT5 (safety buffer)
• Top 5K English tokens from English corpus (Cover about 90% of vocabulary)
• Top 2K Arabic tokens from Arabic corpus (Cover about 90% of vocabulary)
• Top Persian tokens (up to ~27K, Cover about 99.9% of vocabulary)
• 100 special tokens (T5's extra tokens like <extra_id_0>)

This model can tackle Zero-Width-Non-Joiner (ZWNJ - نیم‌فاصله) unlike the original one.