pdnhan/whisper-small-mobile

Whisper small for mobile (ONNX)

An int8 ONNX copy of OpenAI's multilingual Whisper small model, for on-device speech recognition with ONNX Runtime. The Thì Thầm app uses it for German, with the decoder forced to <|de|>.

It is made from Xenova/whisper-small at commit 2d67713f236afa48a18992566e7647f6ca848e13. The only change is that the attention maps are no longer graph outputs. ONNX Runtime keeps declared graph outputs in memory, so Xenova's files need much more memory for each transcription than the model needs. The weights, the nodes and the results are unchanged.

Peak memory

Net peak memory of one transcription (both sessions loaded, the encoder, then 20 decoder steps), with ONNX Runtime 1.31.1, 4 intra-op threads and graph optimization ENABLE_ALL, measured in fresh processes:

Model	Xenova file	This copy
small	2231 MB	1034 MB

Check against the original

On 2 FLEURS German clips, 30 greedy decoder steps from the German prompt give identical tokens, and the encoder output is identical, for Xenova's file and for this copy.

Inputs and outputs

• Encoder onnx/encoder_model_quantized.onnx: input input_features, float32 [1, 80, 3000] (log-mel of 30 s of audio); output last_hidden_state.
• Decoder onnx/decoder_model_merged_quantized.onnx: inputs input_ids (int64), encoder_hidden_states, use_cache_branch (bool) and past_key_values.*; outputs logits and present.*.
• German prompt: [50258, 50261, 50359, 50363] (<|startoftranscript|>, <|de|>, <|transcribe|>, <|notimestamps|>). Stop at 50257 (<|endoftext|>). Another language token works the same way.

Credits and license

• Whisper: OpenAI, openai/whisper-small (Apache 2.0 on Hugging Face; the code and weights in openai/whisper are under the MIT License).
• ONNX export: Xenova, Xenova/whisper-small, made for transformers.js.
• This copy: made with make_mobile.py (included here for provenance; it runs from tools/whisper_de of the app's repository). License: Apache 2.0, see LICENSE, as the files it is made from.
• Used by the Thì Thầm Android app for German speech recognition.