whisper-medium-onnx

openai/whisper-medium exported to a single ONNX graph holding the encoder, the decoder loop and beam search, fused into ONNX Runtime's WhisperBeamSearch contrib op (com.microsoft, opset 1; default domain opset 17). One InferenceSession.run() transcribes one 30 s window; no KV cache crosses the graph boundary.

Files

file	bytes	sha256
whisper-medium_beamsearch.int8.onnx	997,613,039	98fcf400f539cba4b702ed8bc2a3c91c23d89906e11df367e3cf6f02245ea1c2
whisper-medium_beamsearch.onnx	1,753,381	d30f610c1687bd074920e625603522da393eff713d1166a296f533cdf8ef2cd9
whisper-medium_beamsearch.onnx.data	3,267,870,720	97d5e7c335e9132963452a8c395be8fd99e034d85f1e3c9f34415b009e60911b

whisper-medium_beamsearch.int8.onnx is self-contained. The fp32 model stores its weights externally; whisper-medium_beamsearch.onnx.data must sit next to whisper-medium_beamsearch.onnx.

Export

Exporter onnxruntime.transformers.models.whisper.convert_to_onnx, with onnxruntime 1.22.0, torch 2.6.0+cpu, transformers 4.46.3, onnx 1.22.0, Python 3.11.15.

int8:

python -m onnxruntime.transformers.models.whisper.convert_to_onnx \
  -m openai/whisper-medium \
  --output ./medium-int8 \
  --precision int8 \
  --use_external_data_format \
  --use_forced_decoder_ids \
  --overwrite

fp32:

python -m onnxruntime.transformers.models.whisper.convert_to_onnx \
  -m openai/whisper-medium \
  --output ./medium-fp32 \
  --precision fp32 \
  --use_external_data_format \
  --use_forced_decoder_ids \
  --overwrite

The int8 model was then rewritten to a single file (onnx.save(..., save_as_external_data=False)); the graph is unchanged.

Quantization is onnxruntime.quantization.quantize_dynamic, per-tensor, op_types_to_quantize=["MatMul", "Gemm"] with MatMulConstBOnly=True. The token embedding (Gather) is left at fp32. Graph fusion (--optimize_onnx) was not applied to either model.

Signature

Both models take and return the same tensors.

	name	type	shape
input	input_features	float32	[batch_size, feature_size, sequence_length]
input	max_length	int32	[1]
input	min_length	int32	[1]
input	num_beams	int32	[1]
input	num_return_sequences	int32	[1]
input	length_penalty	float32	[1]
input	repetition_penalty	float32	[1]
input	decoder_input_ids	int32	[batch_size, initial_sequence_length]
output	sequences	int32	[batch_size, num_return_sequences, max_length]

input_features is the standard Whisper log-mel spectrogram: 80 mel bins, 3000 frames — 30 s at 16 kHz, 400-point window, hop 160.

Usage notes

• Timestamps: pass <|0.00|> (token 50364) as the last forced decoder id. Given <|notimestamps|> (50363) instead, the model returns bare text. Timestamps come back as paired tokens >= 50364, at 0.02 s per step.
• Language detection: pass only <|startoftranscript|> (50258) as decoder_input_ids. The next generated token is the language token, in 50259-50357.

import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("whisper-medium_beamsearch.int8.onnx",
                            providers=["CPUExecutionProvider"])
sequences = sess.run(None, {
    "input_features": mel[None],                      # (1, 80, 3000) float32
    "max_length": np.array([200], np.int32),
    "min_length": np.array([1], np.int32),
    "num_beams": np.array([5], np.int32),
    "num_return_sequences": np.array([1], np.int32),
    "length_penalty": np.array([1.0], np.float32),
    "repetition_penalty": np.array([1.0], np.float32),
    # <|startoftranscript|>, <|es|>, <|transcribe|>, <|0.00|>
    "decoder_input_ids": np.array([[50258, 50262, 50359, 50364]], np.int32),
})[0]

Measured

One 30 s window, num_beams=5, max_length=200, on a Ryzen CPU and an RTX 4090 (onnxruntime-gpu 1.22.0, CUDA 12 wheels, cuDNN 9.8.0.87). Best of three runs.

model	CPU	CUDA
int8	9.18 s	5.40 s
fp32	13.14 s	1.64 s

int8 on CUDA is held back by MatMulInteger falling back to CPU. Use int8 on CPU and fp32 on CUDA.

On a Spanish test clip both models returned the same transcript, and timestamp boundaries agreed to within 0.04 s.

Tokenizer

vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, tokenizer_config.json, normalizer.json, preprocessor_config.json, config.json and generation_config.json are copied unchanged from openai/whisper-medium.

License

apache-2.0, as declared by openai/whisper-medium. OpenAI released the original Whisper code and weights under MIT.