whisper-large-v3-urdu-v2

Whisper-large-v3 fine-tuned for Pakistani Urdu in its three written registers, selected at decode time by a control token:

token	register	output looks like
`<	scripturdu	>`
`<	scriptroman	>`
`<	scriptmixed	>`

The mixed register keeps English islands in Latin letters inside an Urdu-script matrix — the convention of real Pakistani speech (dramas, call-centres, interviews) — including English function words and phrases, not just borrowed nouns.

Usage

The script token goes after <|notimestamps|> in the decoder prefix:

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

repo = "emads/whisper-large-v3-urdu-v2"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo).eval()

def prefix(script_token: str) -> torch.Tensor:
    toks = ["<|startoftranscript|>", "<|ur|>", "<|transcribe|>",
            "<|notimestamps|>", script_token]
    return torch.tensor([processor.tokenizer.convert_tokens_to_ids(toks)])

feats = processor(audio_16khz, sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(feats, decoder_input_ids=prefix("<|scripturdu|>"), max_new_tokens=200)
print(processor.batch_decode(out, skip_special_tokens=True)[0])

Swap "<|scripturdu|>" for "<|scriptroman|>" or "<|scriptmixed|>" to change the output convention on the same audio.

Evaluation (2026-08-25)

Greedy decoding, register-matched control token, text normalization as in the training repo (github.com/emad-siddiq/urdu-llm, urduwhisper/text_norm.py).

benchmark	metric	this model	whisper-large-v3 fine-tuned v1
FLEURS ur_pk test (n=299)	WER	0.1665	0.1687
Common Voice Urdu test (n=1000)	WER	0.1598	0.1574
Roman-Urdu test, 300 clips	CER	0.1470	0.857 (v1 cannot write Roman)
Urdu–English code-switch, 1000 clips	WER	0.3540	0.3581 (v1 writes islands in Urdu script)

On the code-switch set, 37% of English islands are written in Latin (v1: 0%) with 80% heard in some script; script fidelity is 1.000 for the Urdu register and 0.981 for Roman — the registers do not leak into each other.

Provenance notes: the Roman references are LLM-generated chat-Roman, mechanically audited (no content errors in 300/300; native naturalness review pending). The code-switch benchmark is the manually-corrected gold slice of UrduSpeech (CC-BY-4.0, arXiv:2605.17846).

Training

Fine-tuned from the v1 Urdu fine-tune of openai/whisper-large-v3 (2,500 steps, batch 32, lr 5e-6) on a register-tagged mix: Urdu-script speech (Common Voice, IndicVoices, Kathbath, PRUS, transliterated IndicVoices-Hindi), Roman-Urdu targets (same audio, romanized labels), and a mixed bucket under <|scriptmixed|>: MUCS Hinglish (re-scripted to the Urdu convention), English audio with Latin labels (FLEURS-en, LibriSpeech clean-100), on-the-fly spliced Urdu+English utterances, and paired loanword copies. Latin-letter label tokens of mixed rows carry 2× loss weight. All sources permissively licensed (CC-0/CC-BY); the benchmark corpora never enter training (leakage-filtered).

Limitations

• Long-form audio needs external chunking (30 s window).
• Code-switch WER is dominated by conversational-domain recognition, not script choice; Pakistani conversational audio is the known gap.
• Roman spelling follows chat conventions; expect variant spellings (ke/kay, hai/he) — score with variant folding.
• Decoding without a script token is unsupported; always pass the prefix.