zainab11/whisper-small-ar-fleurs

Whisper Small — Arabic (fine-tuned on FLEURS ar_eg)

openai/whisper-small fine-tuned on the Arabic (ar_eg) subset of Google FLEURS (~6 hours of read MSA speech by Egyptian-accented speakers). Fine-tuning reduces normalized WER on the held-out FLEURS test split from 28.0% → 19.9% (−29% relative), at unchanged inference cost.

Results (FLEURS ar_eg test, 427 samples, corpus-level)

Model	Raw WER	Normalized WER	Normalized CER	RTF (T4, fp16)
openai/whisper-small (zero-shot)	33.07%	28.02%	8.51%	0.079
this model	25.82%	19.90%	6.77%	0.077
openai/whisper-large-v3-turbo (zero-shot, reference)	16.61%	10.10%	3.13%	0.050

Normalization: NFC, diacritics/tatweel removal, alef & ya unification, punctuation stripping — applied identically to references and predictions. One undecodable test sample excluded (documented).

Training

• Data: FLEURS ar_eg train (2,102 clips ≤30s ≈ 6h); labels = dataset's transcription field
• 5 epochs, lr 1e-5 (warmup 50), fp16, effective batch 16, gradient checkpointing, seed 42
• Best checkpoint selected by validation WER (epoch 4, val WER 19.33% — consistent with test 19.90%)

Error analysis (worst-50 rematch vs the zero-shot baseline)

Acoustic substitutions improved most (16/19 samples), numbers 7/7, named entities 9/12; rare technical terms unchanged. Number-format gains partly reflect adaptation to FLEURS's orthographic conventions and may not transfer out of domain.

Usage

# pip install transformers torch librosa soundfile

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

# 1. Load the model
MODEL_ID = "zainab11/whisper-small-ar-fleurs"
processor = WhisperProcessor.from_pretrained(MODEL_ID)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_ID)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()

# 2. Load any audio file (WAV/MP3/...) — converted to mono 16 kHz automatically
audio_path = "my_audio.wav"          # <-- put your file here
audio, sr = librosa.load(audio_path, sr=16000, mono=True)
print(f"Audio duration: {len(audio)/sr:.2f}s")

# 3. Audio -> log-Mel features (+ attention mask)
inputs = processor(audio, sampling_rate=16000,
                   return_attention_mask=True, return_tensors="pt")
input_features = inputs.input_features.to(device)
attention_mask = inputs.attention_mask.to(device)

# 4. Transcribe (Arabic)
with torch.no_grad():
    ids = model.generate(input_features,
                         attention_mask=attention_mask,
                         language="ar", task="transcribe")

# 5. Decode
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Note: clips longer than 30s are truncated by this simple example (Whisper's window). For long audio, use the chunked pipeline or segment first.

Intended use & limitations

Fine-tuned on read MSA speech with Egyptian-accented speakers — not spontaneous Egyptian dialect, noisy telephony, or code-switched speech. Inherits Whisper's known long-form repetition/hallucination behavior. Evaluation on dialectal data (e.g. Casablanca Egyptian subset) is planned future work.

Part of an end-to-end Arabic ASR study (baselines → normalization → error analysis → fine-tuning): github.com/zainabayman11/whisper-ar-skeleton