whisper-tiny-grpo-4bddbd7-cv22-tiny-klbeta02

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 77 languages): WER 0.656, CER 0.270

Training data

Fine-tuned on cv22_index — all 77 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
af	0.800	0.253
am	1.000	0.831
ar	0.647	0.253
as	0.422	0.207
az	0.767	0.220
ba	0.906	0.349
be	0.741	0.203
bg	0.800	0.218
bn	0.500	0.237
br	0.902	0.423
ca	0.720	0.249
cs	0.594	0.125
cy	0.575	0.206
da	0.577	0.271
de	0.327	0.123
el	0.500	0.130
en	0.366	0.180
es	0.543	0.201
et	1.056	0.243
eu	0.625	0.158
fa	1.033	0.456
fi	0.692	0.173
fr	0.325	0.155
gl	0.312	0.081
ha	0.516	0.134
he	0.882	0.366
hi	0.302	0.154
ht	0.905	0.389
hu	1.056	0.703
hy-AM	0.960	0.342
id	0.615	0.270
is	0.935	0.414
it	0.553	0.132
ja	1.000	0.542
ka	1.095	0.528
kk	0.700	0.224
ko	0.522	0.232
lo	0.879	0.648
lt	0.913	0.411
lv	0.750	0.175
mk	0.545	0.167
ml	0.651	0.439
mn	0.706	0.283
mr	0.229	0.104
mt	0.719	0.220
ne-NP	0.460	0.223
nl	0.571	0.214
nn-NO	0.944	0.345
oc	0.760	0.248
pa-IN	0.486	0.364
pl	0.577	0.200
ps	0.792	0.539
pt	0.333	0.113
ro	0.444	0.083
ru	0.455	0.174
sk	0.783	0.242
sl	0.667	0.192
sq	0.852	0.462
sr	0.444	0.226
sv-SE	0.655	0.285
sw	0.733	0.201
ta	0.309	0.185
te	0.571	0.288
tg	0.900	0.304
th	0.621	0.211
tk	1.000	0.490
tt	0.875	0.317
uk	0.750	0.193
ur	0.600	0.252
uz	0.818	0.226
vi	0.478	0.293
yi	0.905	0.431
yo	0.903	0.387
yue	1.200	0.255
zh-CN	1.000	0.271
zh-HK	1.000	0.286
zh-TW	1.000	0.292

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv22_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.2
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-4bddbd7-cv22-tiny-klbeta02")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.