Tokenizer silently drops ө (U+04E9) and ш (U+0448) — most Mongolian text is corrupted on input
Summary
vocab.json contains ѳ (U+0473 CYRILLIC SMALL LETTER FITA) and щ (U+0449 SHCHA)
instead of ө (U+04E9 BARRED O) and ш (U+0448 SHA).
ө and ш are among the most common letters in Mongolian. ѳ is an archaic Church
Slavonic letter unused in modern Mongolian; щ appears only in a handful of Russian
loanwords. The characters look nearly identical when rendered, but they are different
codepoints.
Because VitsTokenizer filters unknown characters rather than mapping them to <unk>,
affected letters are removed with no error or warning.
Reproduction
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("facebook/mms-tts-mon")
print(tok("өвөл")["input_ids"]) # [0, 20, 0, 10, 0]
print(tok("вл")["input_ids"]) # [0, 20, 0, 10, 0] <- identical
print(tok("шинэ")["input_ids"]) # [0, 7, 0, 3, 0, 2, 0]
print(tok("инэ")["input_ids"]) # [0, 7, 0, 3, 0, 2, 0] <- identical
өвөл ("winter") is synthesized as "вл". шинэ ("new") is synthesized as "инэ".
Confirming the vocab contents:
v = tok.get_vocab()
for ch in ["\u04E9", "\u0473", "\u0448", "\u0449"]:
print(f"U+{ord(ch):04X} {ch} {'present' if ch in v else 'ABSENT'}")
# U+04E9 ө ABSENT
# U+0473 ѳ present
# U+0448 ш ABSENT
# U+0449 щ present
The audio is fine — only the labels are wrong
The vocab IDs are frequency-ordered, and the ordering shows the training text was ordinary
Mongolian. Letters that occur only in Russian loanwords cluster at the bottom of the range:
| Token | ID | Expected frequency in Mongolian |
|---|---|---|
п |
32 | rare (loans) |
к |
33 | rare (loans) |
ф |
34 | rare (loans) |
ъ |
35 | rare |
ѳ |
17 | should be ~0 if it were really FITA |
щ |
26 | should be ~0 if it were really SHCHA |
ѳ ranking 17th of 35 is where a common native vowel belongs, not an archaic letter.
This is consistent with training text typeset in a pre-Unicode Mongolian font that reused
the fita and shcha codepoints for ө and ш — the embeddings should already encode
/ɵ/ and /ʃ/ correctly.
[LISTENING TEST — replace or delete:
Verified by ear with a native Mongolian speaker: өвөл and овол are clearly
distinct, and шинэ renders as [ʃinɛ]. The acoustic content is correct.]
Proposed fix
Add aliases to vocab.json, keeping the existing keys for backward compatibility:
"ө": 17,
"ш": 26
No change to model weights, and vocab_size is unaffected (IDs 0–63 are already dense;
this only adds keys pointing at existing IDs).
Workaround for anyone hitting this today:
FIX = str.maketrans({"ө": "ѳ", "Ө": "ѳ", "ш": "щ", "Ш": "щ"})
text = text.lower().translate(FIX)
[FIXED REPO — replace or delete:
A patched checkpoint is available at nugjii/mms-tts-mon-fixed.]
Environment
transformers 4.34.0, torch 2.11.0+cpu, Python 3.13.15