najdi-classifier: Arabic spoken dialect vs MSA

What it does: given an Arabic speech clip, it returns P(dialect), the probability that the clip sounds like spoken (colloquial) Arabic rather than Modern Standard Arabic (MSA).

Despite the repo name, this model does not detect Najdi specifically. It was built for a Najdi TTS project as an automatic check that synthesized speech "sounds like dialect, not MSA". A 5-class Najdi / Hijazi / Khaliji / MSA / Other version did not generalize to unseen TV shows (Najdi-vs-rest AUC ≈ 0.7), so only the dialect-vs-MSA model is released.

Part of the MSc project Emotional Najdi TTS (task 3.1, evaluation pipeline).

Model

• Encoder (frozen, not included): openai/whisper-large-v3-turbo, float32. For each clip (16 kHz, up to 30 s): hidden states of encoder layers 20, 24, 28 and 32, each averaged over the real audio frames only (50 frames/s; Whisper's silence padding is left out), then concatenated (5,120 values).
• Head (this repo, model.safetensors): standardization + logistic regression (C = 1e-3, balanced class weights). Plain arrays, no pickle; inference.py needs only numpy, torch and transformers.
• Selection: logistic regression, RBF-SVM, gradient boosting and a layer-weighted neural head were compared by 5-fold cross-validation grouped by TV show; logistic regression won (CV AUC 0.923).

Training data

SADA (Saudi Audio Dataset for Arabic, SDAIA, Kaggle version 23, CC BY-NC-SA 4.0): single-speaker segments of 3–15 s from Saudi TV shows. Labels come from SADA's SpeakerDialect; "MSA" = ModernStandardArabic, "Dialect" = every other label (Najdi, Hijazi, Khaliji, Shamali, Janubi, Levantine, Egyptian, Iraqi, Yemeni, Maghrebi).

• Clean audio for all classes, plus Music / Noisy audio with the same number of dialect and MSA segments, so background music or noise does not predict "MSA".
• Every TV show is in exactly one split (SADA speaker IDs are only unique within a file), so the test set has unseen shows and speakers.

Split	Clips	Hours	Shows	MSA clips (shows)
Train + validation (used for the released model)	4,830	8.2	54	603 (25)
Test (held out)	1,109	2.0	10	308 (9)

Evaluation

On the held-out test shows:

	AUC	Balanced accuracy (threshold 0.5)
All test clips	0.937	0.868
Clean audio (828 clips)	0.932
Music (198) / Noisy (83)	0.907 / 0.877

Mean P(dialect) by true SADA label on the test shows: Najdi 0.83, Hijazi 0.80, Khaliji 0.82, Other 0.85, MSA 0.26.

Check against a native listener (TTS output): 80 blind clips (real Najdi recordings and two pretrained TTS systems) labelled by one native Najdi speaker as dialect / MSA / unclear: AUC 0.935 for P(dialect) against the dialect-vs-MSA labels, agreement 0.85 at 0.5. The labels split cleanly by system, so this validates ranking systems, not clip-level differences.

Use

import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("nsa01n/najdi-classifier"))
from inference import DialectScorer
DialectScorer().p_dialect("clip.wav")   # e.g. 0.83 for colloquial Saudi speech, 0.2 for MSA news-style

or python inference.py clip.wav after downloading the repo.

Limitations

• Dialect vs MSA only. It does not tell Najdi from Hijazi, Khaliji or other dialects.
• Unintelligible speech can score as "dialect" (it learned "not MSA"). When scoring TTS output, report it only for clips a speech recognizer can follow (the project uses CER ≤ 0.3).
• Trained on Saudi TV speech; MSA comes from only 34 shows, so it rests on few MSA speakers and channels. Other domains (read speech, phone audio, non-Saudi MSA) are untested.
• SADA labels are per-segment speaker dialect on TV; short segments often carry few dialect cues.
• Synthetic speech: validated on a small listening check only; confirm with listeners before relying on it for conclusions.

License and attribution

The head was trained on SADA, so it is released under CC BY-NC-SA 4.0 (non-commercial, share-alike, attribution). SADA: Saudi Data and AI Authority (SDAIA), SADA: Saudi Audio Dataset for Arabic, Kaggle sdaiancai/sada2022. Whisper (the encoder this head reads) is MIT-licensed by OpenAI and is downloaded from its own repository.