sarapd/Qwen3-ASR-1.7B_Jordanian_Dialect_Arabic

Qwen3-ASR-1.7B — Jordanian Dialect Arabic

A full fine-tune of Qwen/Qwen3-ASR-1.7B on ~10.8 hours of Jordanian Arabic speech.

WER 31.63% → 27.59%, CER 12.84% → 11.11% on a held-out 2,067-utterance (~3.3 hour) test set. A LoRA variant was also trained for comparison but full fine-tuning gave the larger accuracy gain for a similar training cost — see Training.

---

Model details

Base architecture	Audio encoder (300M, FastConformer-style, 1–8s variable chunking, ~12.5 tokens/s) → projector → Qwen3-1.7B language model decoder
Fine-tuning method	Full fine-tune (all LM-decoder parameters updated; audio encoder + projector frozen)
Language	Arabic — Jordanian dialect
Checkpoint	checkpoint-560 (selected for best eval WER)
License	Apache-2.0 (inherited from base model)

Only the language-model component is touched during fine-tuning — it's the part responsible for learning dialect and vocabulary; the audio encoder and projector stay frozen.

---

Performance

Test set: 2,067 utterances (~3.32 hours), held out from training. Same normalizer, same helper, same checkpoint used for both scoring runs — the only variable is fine-tuned vs. base weights.

	Baseline	Full fine-tune	LoRA
WER	31.63%	27.59%	30.14%
CER	12.84%	11.11%	12.49%

By category (WER)

Category	Baseline	Full-FT	LoRA	n
Code-switching	47.21%	37.97%	41.75%	98
Dialect numbers	32.99%	27.56%	29.98%	253
Plain	30.44%	27.01%	29.69%	1,747

By source (WER)

Baseline	Full-FT	LoRA	n
27.71%	25.33%	26.96%	1,465
45.88%	35.80%	41.70%	602

Inference speed (A100, batch=16, full test set)

	Time	Speed vs. real-time
Baseline	3:19	~59×
Full fine-tune	3:51	~53×
LoRA	7:52	~26×

Full fine-tuning barely changes inference speed relative to baseline. LoRA is roughly 2× slower here, still comfortably real-time.

Streaming simulation (2-second chunks, single stream): ~1.5s average latency per chunk; the slowest 5% of chunks took 3.1–3.3s. Latency here is architecture-bound and doesn't meaningfully change with fine-tuning.

---

Training

Method: full fine-tune of the language-model decoder. A LoRA run (r=16, alpha=32) was also trained for comparison; the training script required manual modification to support LoRA, as it wasn't built in.

	Full fine-tune	LoRA
Learning rate	2e-5	1e-4
Epochs	8	8
Batch size	6 (effective 96, grad accum 16)	same
Hardware	1× A100 (Modal)	same
Training time	~1h15m	~55min

A full fine-tune at 4 epochs was also tried first (~40min): WER 31.63% → 28.21%, CER 12.84% → 11.70% — better than baseline but worse than the 8-epoch run, which is why 8 epochs / checkpoint 560 was used going forward.

Data

Split	Utterances	Hours
Train pool	7,056	~10.82
Eval (held out of train pool)	353	—
Test	2,067	~3.32

• Train and test draw from disjoint source-recording sets — 223 unique recordings in train, 141 in test, with no overlap found among the recordings checked.
• Audio mix: a combination of YouTube-style recordings and phone-call-style recordings (roughly a quarter of unique sources in both train and test).
• No speaker-level metadata. A "source recording" is not the same as a speaker; number of distinct speakers is unknown.
• Source concentration: the largest single source recording accounts for ~10% of train and ~19% of test — headline numbers are partly a statement about performance on a small number of recordings, not a uniformly balanced sample.
• WER/CER computed via jiwer after normalization: punctuation stripped, alef variants (أ إ آ → ا) folded, teh marbuta (ة → ه) folded, whitespace collapsed — applied identically to references and hypotheses.

---

Quantization

Post-training quantization to W4A16 (4-bit weights, 16-bit activations) was performed via GPTQ (llmcompressor), applied to the full fine-tune checkpoint. The audio encoder and lm_head were excluded from quantization; only the language-model decoder layers were quantized.

• Checkpoint size: 3.81 GB → 1.86 GB (~51% reduction)
• Spot-checked on a small held-out sample: transcription is functional and mostly accurate, but shows a consistent accuracy cost relative to the full fine-tune, concentrated in the same two weak categories as the unquantized model — dialect numbers (e.g. one sample reverted to MSA ثمانية عشر instead of the dialectal ثمنطعشر) and code-switching.
• Full test-set WER/CER for the quantized checkpoint has not yet been measured — treat this as a proof of concept rather than a validated deployment artifact.

Usage

import os, shutil, torch
from huggingface_hub import hf_hub_download

CKPT_DIR = "path/to/checkpoint-560"

# Add required config files if not already present
for fname in ["preprocessor_config.json", "chat_template.json"]:
    dest = os.path.join(CKPT_DIR, fname)
    if not os.path.exists(dest):
        src = hf_hub_download("Qwen/Qwen3-ASR-1.7B", fname)
        shutil.copy(src, dest)

model = Qwen3ASRModel.from_pretrained(
    CKPT_DIR,
    dtype=torch.bfloat16,
    device_map="cuda:0",
)

output = model.transcribe(audio=(audio, sample_rate), language="Arabic")
print(output)

Can also be served with vLLM.

---

Limitations

• Code-switching is the weakest category — 37.97% WER even on the best checkpoint (baseline: 47.21%). English brand names and technical terms are frequently mangled or transliterated phonetically (e.g. "catering" → كيتيرين).
• Dialect numbers (e.g. ثمنطعش، سبعطعش، ستطعش) are inconsistently transcribed by the baseline and LoRA models; the full fine-tune is noticeably more consistent here.
• Test set is source-imbalanced — one recording accounts for ~19% of all test utterances, so reported WER is partly a statement about that recording's characteristics.
• No speaker metadata — number of distinct speakers, gender balance, and recording conditions are unknown beyond the source-recording split.
• Single training run per method — no repeated-seed runs, so the full-FT vs. LoRA gap (27.59% vs. 30.14%) hasn't been checked for run-to-run variance.

What would improve this model

• More code-switching examples in training, the single highest-leverage fix (47%→38% still leaves a large error rate on this category)
• A larger base model, or one with more Levantine Arabic exposure — dialect numbers and code switching may be hitting a ceiling tied to the base model's pretraining rather than fine-tuning data volume