dziaineka/whisper-small-belarusian

Whisper Small Belarusian

This model is a fine-tuned version of openai/whisper-small on the mozilla-foundation/common_voice_11_0 be dataset. It achieves the following results on the evaluation set:

• Loss on validation: 0.0706
• WER on validation set: 6.3672
• WER on test set: 6.79

FUTO Keyboard variant (run_5/, supersedes run_4/)

run_5/ contains a separate model derived from this one, built specifically for use as a custom voice recognition model in FUTO Keyboard (which runs models via whisper.cpp). It is not a replacement for the root model above - it lives alongside it as an additional, purpose-built artifact. In short, across run_4 and run_5:

1. Continued fine-tuning on a larger, refreshed Common Voice release (CV26, ~1,818 validated hours vs. CV11's smaller set) - improved WER from 6.79% to 5.88% (run_4).
2. run_4 additionally adapted the model with futo-org/whisper-acft so it tolerates whisper.cpp's dynamic audio_ctx optimization without a WER blowup. Real-world testing then found a deeper bug: on audio longer than one short utterance, whisper.cpp only transcribed the first sentence of each 30-second window and silently dropped the rest, because the training recipe forced <|notimestamps|> on every label, so the model never learned to report where its own output ends.
3. run_5 fixes this with --predict_timestamps=True (real per-clip timestamps instead of forced <|notimestamps|>) - 5.98% WER, essentially unchanged from run_4. This also turned out to make whisper-acft adaptation unnecessary: the timestamp-trained checkpoint alone already scores identically at fixed and dynamic audio_ctx (6.90% both), so run_5 ships without acft applied. Important caveat: this fixes segment-timestamp accuracy, not multi-segment continuation - a single dictation utterance up to ~30s now transcribes completely and accurately, but a continuous multi-minute recording still only captures the first sentence per 30s window (Common Voice's single-utterance data can't teach otherwise; see run_5/long_audio_test_results.md for the full before/after comparison and what a further fix would require).
4. Converted to GGML and quantized to q8_0 (252 MB) for direct use in whisper.cpp/FUTO Keyboard: run_5/ggml/ggml-model-q8_0.bin.

See run_5/readme.md for the full writeup (the acft-adaptation detour and why it was dropped, exact commands, and all metrics) and run_4/readme.md for the earlier CV26/acft work it builds on. New tooling: src/prepare_local_common_voice.py, src/acft_adapt.py, src/eval_acft.py, and --predict_timestamps on src/run_speech_recognition_seq2seq_streaming.py.

Source code

All the source coude is located both in:

• GitHub repository
• and under src folder

Code in these 2 places should be the same. GitHub is used to make development and training of multiple models (small, base, etc.) easier.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 12000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.1907	0.08	1000	0.2546	25.4639
0.1482	0.17	2000	0.1641	17.1676
0.1175	0.25	3000	0.1454	15.5940
0.0958	0.33	4000	0.1261	13.2625
0.099	0.42	5000	0.1012	10.6143
0.028	1.05	6000	0.1053	9.8794
0.0473	1.13	7000	0.1029	10.3078
0.0391	1.21	8000	0.0924	9.2419
0.0423	1.3	9000	0.0797	7.9249
0.0604	1.38	10000	0.0688	7.0150
0.0121	2.01	11000	0.0696	6.4638
0.0155	2.1	12000	0.0706	6.3672

Framework versions

• Transformers 4.26.0.dev0
• Pytorch 1.13.0+cu117
• Datasets 2.7.1.dev0
• Tokenizers 0.13.2