whisper-tiny-grpo-0029b82-cv17-tiny-sft

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 49 languages): WER 0.472, CER 0.175

Training data

Fine-tuned on cv17_index — all 49 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
ar	0.257	0.161
ast	0.511	0.196
be	0.714	0.203
bg	0.542	0.123
bn	0.443	0.310
br	0.963	0.515
cs	0.467	0.107
cy	0.587	0.226
da	0.833	0.298
de	0.373	0.114
el	0.429	0.135
en	0.281	0.139
es	0.404	0.186
et	0.565	0.140
fa	0.724	0.311
fi	0.513	0.174
fr	0.373	0.108
gl	0.404	0.080
ha	0.558	0.168
hi	0.367	0.181
hu	0.658	0.180
it	0.409	0.140
ja	1.167	0.262
ka	0.676	0.269
ko	0.558	0.243
lt	0.824	0.238
lv	0.586	0.159
mk	0.610	0.162
ml	0.429	0.284
mn	0.654	0.219
mr	0.216	0.111
nl	0.200	0.071
oc	0.882	0.325
pl	0.571	0.234
pt	0.368	0.115
ro	0.421	0.146
ru	0.375	0.112
sk	0.727	0.176
sl	0.389	0.100
sr	0.111	0.029
sv-SE	0.375	0.156
sw	0.608	0.188
ta	0.141	0.068
te	0.353	0.200
th	0.486	0.168
tr	0.630	0.114
uk	0.282	0.055
ur	0.568	0.266
vi	0.500	0.219

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv17_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.04
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-0029b82-cv17-tiny-sft")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.