X-MDD (Qwen3-ASR-1.7B, canonical speech + phones, ICL + GRPO)

Main checkpoint of X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References (arXiv:2610.23079).

Given the canonical speech (TTS-synthesized, voice-cloned from the learner), the canonical phones and the learner speech, plus optional in-context demonstrations, the model generates the phone sequence the learner actually produced. Aligning it to the canonical phones gives detection and diagnosis.

• Backbone: Qwen3-ASR-1.7B (AuT + projector + LLM decoder); all parameters fine-tuned except the AuT Conv2D front end
• Training: SFT on Mandarin + English (7k steps, 0–4 demos), then GRPO (300 steps, 1–2 demos, reward = 0.5·F1 + 0.5·(1−PER)); this is GRPO step 275
• Canonical speech TTS: Fun-CosyVoice3-0.5B (zh/en), SILMA TTS (ar)

Test set	Demos	PER ↓	F1 ↑	DER ↓
Mandarin	4	17.66	79.00	31.32
English (L2-ARCTIC)	4	12.86	58.88	19.47
Arabic, QuranMB (unseen)	128	7.75	38.07	40.33

Files

File	Content
model.safetensors	Weights (bf16; keys asr.* = Qwen3-ASR; ctc_head.* = an unused CTC head, ignored at inference)
config.yaml	Inference config (prompt layout, language prefixes, model settings)
processor/	Qwen3-ASR processor and tokenizer, including the added prompt boundary tokens
phone_vocabulary.txt	The 242-phone training vocabulary (zh tonal phones + CMU 39 phones), in index order
arabic_phone_symbols.txt	Phone symbols used for constrained decoding on Arabic (QuranMB)

Usage

Install the X-MDD code (xmdd package), then:

hf download walston/X-MDD --local-dir pretrained/X-MDD
python -m xmdd.infer --model-dir pretrained/X-MDD --language en \
    --manifest my_en_test.jsonl --support-manifest my_en_test.jsonl \
    --constrain-phones my_en_phones.txt --shots 4 --out-dir results/en

from xmdd import XMDD, load_manifest, select_demos
model = XMDD.from_pretrained("pretrained/X-MDD")
queries = load_manifest("my_en_test.jsonl", "en", model.cfg)
print(model.predict(queries[0], select_demos(queries[0], queries, shots=4)))

For a language other than zh/en/ar also pass --language-name. The base Qwen/Qwen3-ASR-1.7B-hf is downloaded for the architecture and then overwritten by these weights.

Input manifest rows (JSONL) look like:

{"id": "...", "speaker_id": "...", "l2_wav": "learner.wav", "canonical_wav": "canonical_tts.wav",
 "canonical_phones": ["F", "AO", "R"], "actual_phones": ["F", "AO", "R"]}

actual_phones is only needed for demonstrations and for scoring. Demonstrations are picked from the same speaker first. For an unseen language, pass raw phones, the language prefix (ar is already in config.yaml) and optionally constrain decoding to the target phone set.

Citation

@article{wang2026xmdd,
  title   = {X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References},
  author  = {Wang, Xintong and Zhao, Junchuan and Wang, Ye},
  journal = {arXiv preprint arXiv:2610.23079},
  year    = {2026}
}