whisper-base-he

A 50.2M-parameter Hebrew speech recognition model, built by replacing Whisper's multilingual vocabulary with a Hebrew one and training on 3,113 hours of human-transcribed Hebrew.

It is the most accurate open Hebrew recogniser of its size by a wide margin, and second only to models roughly sixteen times larger.

Results

Word error rate, scored with a harness that reproduces the ivrit.ai Hebrew leaderboard on 40 of 40 published model and dataset pairs.

Benchmark	this model, 50.2M	whisper-small, 242M	whisper-base, 72.6M	ivrit.ai large-v3-turbo, 809M
ivrit-ai/eval-d1	14.87%			5.5%
imvladikon/hebrew_speech_kan	18.47%	37.42%	68.12%	8.10%
ivrit-ai/eval-whatsapp	25.77%			6.1%
google/fleurs he	33.74%	45.94%	68.19%	18.72%

Against every open Hebrew model we could find, on FLEURS-he:

Model	Parameters	WER
ivrit-ai/whisper-large-v3-turbo	809M	18.72%
whisper-base-he	50.2M	33.74%
openai/whisper-small	242M	45.94%
mike249/whisper-tiny-he-2	37.8M	66.98%
openai/whisper-base	72.6M	68.19%
Alex2575/aleksis_heb_base	72.6M	68.98%
imvladikon/wav2vec2-xls-r-300m-hebrew	315M	71.85%
openai/whisper-tiny	37.8M	79.54%

It runs at 500 to 800 times realtime on an RTX 5070 Laptop: a 47-minute recording transcribes in 3.6 seconds.

Why it is smaller than the model it came from

Whisper's vocabulary holds 51,865 tokens for 99 languages. On 24,225 words of held-out Hebrew speech it spends 3.17 tokens per Hebrew word, against 1.28 for English, and its embedding table is 26.6M of whisper-base's 72.6M parameters, nearly all of it tokens Hebrew never emits.

An 8,192-token Hebrew byte-level BPE brings that to 1.76 tokens per word and the model to 50.23M parameters, 30.8% smaller. Every new embedding row is initialised from the pretrained table, averaging the old rows for the same text, so none start from noise.

Measured on identical data and schedule, that vocabulary is worth 7.5 WER points on its own, not merely the size saving.

Usage

from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer, AutoFeatureExtractor
from transformers.generation.utils import GenerationMixin

model = AutoModelForSpeechSeq2Seq.from_pretrained("itayinbar/whisper-base-he")
tokenizer = AutoTokenizer.from_pretrained("itayinbar/whisper-base-he")
features = AutoFeatureExtractor.from_pretrained("itayinbar/whisper-base-he")

inputs = features(audio_16khz, sampling_rate=16000,
                  return_tensors="pt", padding="max_length")
# Monolingual: no language token, no task token, no timestamps. Whisper's own
# generate() expects all three, so call the generic implementation.
ids = GenerationMixin.generate(model, **inputs, max_new_tokens=200)
print(tokenizer.decode(ids[0], skip_special_tokens=True))

Three things differ from stock Whisper:

1. Call the generic generate. This model has no language or task tokens.
2. Pad mel features to the full 30-second window, as Whisper always requires.
3. Cut audio longer than 30 seconds into overlapping windows and stitch the results. Without that, eval-d1 scores 91.94% instead of 14.87%, because everything past the first thirty seconds is scored as deleted.

ONNX weights are included for transformers.js. A browser downloads 160.2 MB (fp16 encoder plus fp32 decoder), against 563 MB for the large Hebrew model it is meant to replace.

Only those two weight files are shipped. An fp16 decoder and an int8 encoder were built and then removed: the fp16 decoder fails to load in ONNX Runtime, and the int8 encoder loads but changes the transcript. Every combination offered here was checked by transcribing with it and comparing against the fp32 output, not by loading it.

Training

3,112.7 hours over 552,327 utterances, all human-transcribed.

Corpus	Hours
ivrit-ai/knesset-plenums-whisper-training	2,756.6
ivrit-ai/crowd-transcribe-v5	295.0
ivrit-ai/crowd-recital-whisper-training	44.7
google/fleurs he train	9.5
imvladikon/hebrew_speech_kan train	6.9

Schedule-free AdamW, learning rate 1e-4, batch 16, bf16, 103,551 steps on a single RTX 5070 Laptop GPU with 8 GB. Corpora are sampled by target share of audio time rather than of utterances, since Knesset ships 30-second windows and crowd-transcribe averages 5.2 seconds.

What was tried and did not work

Measured on identical data and budget, so the comparisons are like for like:

Change	Effect on WER
Hebrew tokenizer instead of Whisper's	-7.5 points, and 30.8% fewer parameters
700 hours to 3,113 hours	-2.7 points on average
SpecAugment	none
Hybrid CTC objective	none, at 19% lower throughput

SpecAugment and hybrid CTC are standard practice in speech recognition, and neither did anything here.

Licence and provenance

Weights are Apache-2.0, following openai/whisper-base.

Training data from ivrit.ai under the ivrit.ai licence, which permits training models including commercially and requires attribution. Credit for that data belongs to ivrit.ai, whose own Hebrew models are the state of the art this one is measured against. FLEURS is CC-BY-4.0.

imvladikon/hebrew_speech_kan declares no licence on the Hub. It contributed 6.9 of 3,112.7 hours and roughly 4% of the audio the model heard after weighting. It is named here so anyone relying on this model's provenance can judge that themselves.

Code, evaluation harness and full method: github.com/itayinbarr/Hebrew-small-asr.