whisper-small-santali-coded

A full fine-tune of openai/whisper-small for automatic speech recognition on Santali, transcribed into Sanlish — a Latin-script romanization scheme for Santali speech.

Model description

Unlike partial-freeze / adapter-style approaches, every parameter of the base Whisper-small checkpoint (encoder, decoder, and output head) was fine-tuned. No language tag was pinned during tokenization or generation (generation_config.language = None), so the model was not steered toward Bengali or any other Whisper-supported language — it learned Sanlish surface forms purely from the fine-tuning data.

• Base model: openai/whisper-small (~244M params)
• Fine-tuning strategy: full fine-tune, no frozen layers
• Task: speech transcription (audio → Sanlish text)
• Language tag: none pinned (language-agnostic decoding)

Training data

• Source: custom Santali audio + Sanlish transliteration dataset
• Splits: train / validation / test (santali_train.csv, santali_val.csv, santali_test.csv)
• Format: audio_path (16kHz audio) → sanlish (target transliteration text)

Training procedure

Hyperparameter	Value
Base checkpoint	openai/whisper-small
Learning rate	2e-5
Weight decay	0.1
Batch size (train)	16
Epochs (planned / completed)	20 planned, 17 completed (training interrupted by a runtime disconnect)
Warmup steps	computed dynamically as 10% of total training steps
Eval/save strategy	every epoch
Early stopping	patience 5 (on validation WER) — not yet triggered when training stopped
Metric for best model	WER (lower is better)

Evaluation results

Per-epoch validation metrics (normalized WER/CER — lowercased, punctuation-stripped):

Epoch	Train Loss	Val Loss	WER (%)	CER (%)
1	1.8145	0.6641	54.51	11.61
3	0.1810	0.4263	36.37	6.60
8	0.0142	0.4845	33.30	5.92
10	0.0068	0.4903	31.50	5.58
14 (best WER)	0.0006	0.5197	31.07	5.39
17 (checkpoint on main)	0.0003	0.5266	31.18	5.41

Note: validation loss bottoms out around epoch 3 and rises afterward while WER/CER keep improving slightly — a sign of overfitting on the loss objective that hasn't yet hurt transcription accuracy. Epoch 14 had the lowest validation WER of the run; epoch 17 (the last checkpoint pushed before the training runtime disconnected) is marginally behind it and is what's currently on main. Earlier epoch checkpoints are available in this repo's commit history.

Test set: 28.82% WER, 4.89% CER — evaluated on the held-out test split (193 examples), using the epoch-17 checkpoint currently on main.

Limitations

• Training did not complete its full 20-epoch schedule (interrupted at epoch 17), and early stopping had not yet triggered.
• Fine-tuned on a relatively small custom dataset for a low-resource language variant; may not generalize to speakers, dialects, or recording conditions outside the training distribution.
• No language tag is pinned, so downstream users should explicitly pass task="transcribe" at inference and not rely on Whisper's automatic language detection for this checkpoint.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-coded")
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-coded", task="transcribe")

model.generation_config.language = None
model.generation_config.task = "transcribe"

# input_features = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
# predicted_ids = model.generate(input_features)
# transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)