whisper-small-danish

A Danish fine-tune of Whisper Small. The architecture and tokenizer are unchanged. The released weights are the exponential moving average after 200,000 optimizer steps.

Usage

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "thorhojhus/whisper-small-danish"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, dtype=dtype, low_cpu_mem_usage=True,
).to(device)

transcribe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)
result = transcribe(
    "audio.wav",
    generate_kwargs={"language": "da", "task": "transcribe", "num_beams": 1},
)
print(result["text"])

Training

Training was started from stock Whisper Small. Filtered pseudo-labelled Danish speech was combined with the Danish portions of CoRal v3, Common Voice 26.0, FLEURS, FTSpeech and NST. Pseudo-labels were generated with whisper-large-v3-turbo-danish. The retained pool contained approximately 13,411 unique audio hours. Labels were filtered by confidence, no-speech probability, hallucination rules and a Danish/English vocabulary check.

Pseudo-labelled speech accounted for 100% of the sampling weight through step 50,000, 80% through step 140,000 and 70% thereafter. Human-labelled sources made up the remainder. Targets retained their casing, punctuation and numeral style; normalization was applied only for scoring.

The use of weak supervision and no-speech examples was inspired by the original Whisper paper. Waveform augmentation, SpecAugment and synthetic no-speech examples were included.

Setting	Value
Steps	200,000
Optimizer	Muon for hidden matrices; AdamW for remaining parameters
Peak learning rate	Muon 5e-5; AdamW 2.5e-5
Weight decay	0.01
Schedule	1,000-step warmup, stable to 50,000, linear decay to 10% at 200,000
EMA	Per-step decay 0.9999, applied every 50 steps
Precision	BF16 parameters with compensated updates; FP32 EMA
Maximum training segment	30 seconds
Training hardware	One NVIDIA GeForce RTX 4090
Logged active training time	Approximately 55.7 hours, excluding validation, checkpoint I/O and pauses

The following figures are cumulative audio sampled on the final training trajectory, including repeated passes. They are not unique dataset durations.

Source	Hours seen
CoRal conversation	1,151.29
CoRal read-aloud	2,584.61
Common Voice Danish	139.94
FLEURS Danish	140.21
FTSpeech	4,182.40
NST Danish	1,938.15
Pseudo-labelled data	44,344.90
Total	54,481.50

Additional settings and provenance are recorded in training_recipe.json.

Evaluation

The checkpoint was evaluated using the public Danish ASR Leaderboard protocol with Transformers FP16 greedy Danish transcription. Standard decoding was used for short clips; timestamped sequential decoding was used above 30 seconds. Scoring included the leaderboard's Danish numeral normalization and did not remove filler words.

Test set	Samples	WER (%)	CER (%)
CoRal conversation	8,438	24.01	15.02
CoRal read-aloud	9,122	14.16	5.73
Common Voice (cv17_da)	2,756	9.53	3.65
FLEURS da_dk	930	9.94	3.94
FTSpeech test_balanced	5,534	9.22	5.29
Unweighted domain mean	26,780	13.37	6.73

Limitations

• Performance outside Danish has not been validated after fine-tuning.
• Long-form behavior was not directly trained and should be checked for the intended use.
• Silence, noise and unfamiliar names can still produce incorrect text.

License

The model weights are released under the MIT License. Training data are not redistributed and remain subject to their source terms. Attribution and the upstream Whisper notice are included in THIRD_PARTY_NOTICES.md.

Citation

@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}