Nawah-ASR-50M-v1

Arabic speech recognition where the text is generated by a 50M Arabic LLM, not by an ASR decoder. A frozen Whisper-small encoder feeds a small projector, and oddadmix/50M-2048-Emhotob writes the transcript.

audio 16kHz
  └─ Whisper-small encoder (frozen)      88.15M
       └─ Linear(768 -> 512) projector    0.39M   trained
            └─ Emhotob-50M decoder       51.82M   trained  (+16.42M lm_head)
                 └─ Arabic text
                                        ───────
                                        156.78M

This is the same graft as Nawah-VL-50M-Grounding, with the vision tower swapped for a speech encoder. No custom modelling code: Qwen2AudioConfig resolves its sub-configs through AutoConfig, so a llama text config composes directly and generate(), the KV cache and save_pretrained all come for free.

Results

Trained on 60 hours of MASC. Every row is the same 3,165-clip video-disjoint held-out split, same normalizer, greedy decoding:

model	decoder	WER	CER
openai/whisper-small (zero-shot)	Whisper 153M	0.5739	0.2062
oddadmix/whisper-small-arabic-dialectal	Whisper 153M	0.6775	0.2244
Nawah-ASR-50M-v1 (60 h)	Emhotob 50M	0.6145	0.2936
Nawah-ASR-50M-v1.1 (300 h)	Emhotob 50M	0.4109	0.1815

Use v1.1 instead. This checkpoint is kept for the data-scaling comparison: 5x the data took WER 0.6145 -> 0.4109 with no recipe change, so data was the only variable.

Note these Whisper numbers are out-of-domain on MASC and are not the checkpoints' published scores on their own test sets — whisper-small-arabic-dialectal reports 0.428 on the dialectal set it was tuned for. Comparing a number from one test set against a number from another is meaningless, so everything above was re-run here.

Zero empty hypotheses; ~190x realtime.

Usage

import soundfile as sf, torch
from transformers import (Qwen2AudioForConditionalGeneration, AutoTokenizer,
                          WhisperFeatureExtractor)

repo = "oddadmix/Nawah-ASR-50M-v1"
model = Qwen2AudioForConditionalGeneration.from_pretrained(repo).eval()
tok   = AutoTokenizer.from_pretrained(repo)
fe    = WhisperFeatureExtractor.from_pretrained(repo)

wav, sr = sf.read("clip.wav", dtype="float32")          # 16 kHz mono
feats = fe([wav], sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
_, n = model.model.audio_tower._get_feat_extract_output_lengths(feats.attention_mask.sum(-1))

cv = tok.convert_tokens_to_ids
prefix = [[tok.bos_token_id, cv("<|audio_start|>")] + [cv("<audio>")] * int(n[0]) + [cv("<|audio_end|>")]]
ids = torch.tensor(prefix)
out = model.generate(input_ids=ids, attention_mask=torch.ones_like(ids),
                     input_features=feats.input_features,
                     feature_attention_mask=feats.attention_mask,
                     max_new_tokens=96, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

The audio token count must match the clip. <audio> is repeated exactly _get_feat_extract_output_lengths(...) times — 25 tokens per second of audio. This is the point of the Qwen2Audio shell: a 3-second clip costs 75 tokens, not the 375 a fixed 30-second window would spend on silence.

Training notes

Two stages: projector only (3 epochs), then projector + LM (8 epochs). AdamW, betas (0.9, 0.95), wd 0.05, cosine with 3% warmup, LR 2e-4 projector / 1e-4 LM. The encoder stays frozen and in .eval() throughout.

The projector must be calibrated before training starts. A freshly-initialised projector emitted audio embeddings at RMS 0.81 against Emhotob's embedding RMS of 0.033 — 24x too large — and alignment does not begin until that is corrected by a single measured scalar. The sibling vision project hit the same wall at 36x. If training stalls, check this before touching the LR.

Scripts (build_model.py, collate.py, train_asr.py, eval_asr.py, prepare_masc.py) are included in this repo.

Limitations

MASC is scraped from YouTube and labelled MSA, but carries substantial dialect — expect a mix, not clean MSA. Audio longer than 30 s is truncated by the feature extractor and must be chunked.