pdnhan/PhoWhisper-medium-mobile

PhoWhisper-medium for mobile (ONNX)

vinai/PhoWhisper-medium (769M parameters), VinAI's Vietnamese fine-tune of Whisper, converted to ONNX for on-device speech recognition with ONNX Runtime on Android and iOS. The weights are unchanged; only the format differs (and the precision, for the int8 files).

Files

File	Precision	Size	SHA-256
onnx/decoder_model_merged.onnx	fp32	1,828.0 MB	52507d5bbbbff1264d820afd28f42dbf24cba3b910a1a0fc8f2d564d0cd6b810
onnx/decoder_model_merged_quantized.onnx	int8	513.5 MB	e46abe4257bf3d1d3cc1635bc9f873459c5cda6cc9f98b3d6b253688ff13917f
onnx/encoder_model.onnx	fp32	1,229.1 MB	58db3e78ccbb37a37cd0e0f984973b97e6c1a8fc870bef56835490958711f206
onnx/encoder_model_quantized.onnx	int8	323.5 MB	0ca0afb92ebfa2e73e951b9a522539e071b2f05534172310ee7a48a2cc39cb10

Mobile apps should download the two *_quantized.onnx files (dynamic int8: MatMul and Gather weights). The fp32 files are the reference export, for desktop use or further conversion. The tokenizer and feature-extractor files are the original PhoWhisper ones.

Accuracy

Word error rate on the first 30 clips of the FLEURS Vietnamese test split, greedy decoding, text lower-cased with punctuation removed:

Precision	WER
fp32	4.9%
int8	4.6%

This is a sanity check of the conversion, not a benchmark: see the PhoWhisper paper for proper evaluation.

Model interface

These are standard Optimum exports using only standard ONNX operators (opset 17, IR version 9), so the stock ONNX Runtime packages for Android and iOS run them, with no custom-op library. The int8 files give identical tokens on ONNX Runtime 1.20.0 and 1.31.1.

Encoder: input_features float32 [1, 80, 3000] (log-Mel spectrogram of 30 s of 16 kHz mono audio, computed as in WhisperFeatureExtractor: 400-sample FFT window, hop 160, padded with zeros) → last_hidden_state float32 [1, 1500, 1024].

Decoder (merged, with KV cache): inputs input_ids int64 [1, n], encoder_hidden_states, use_cache_branch bool [1], and past_key_values.<layer>.{decoder,encoder}.{key,value}; outputs logits and present.<layer>.{decoder,encoder}.{key,value}.

Decoding loop (greedy):

1. Run the encoder once.
2. First decoder step: input_ids = [50258, 50278, 50359, 50363] (<|startoftranscript|><|vi|><|transcribe|><|notimestamps|>), use_cache_branch = false, and every past_key_values.* an empty tensor [1, 16, 0, 64].
3. Take the argmax of the last position of logits. Stop on <|endoftext|> (50257).
4. Next steps: input_ids = [[token]], use_cache_branch = true. Feed back present.*.decoder.* every step and present.*.encoder.* from the first step only (the encoder cache does not change).
5. Decode the tokens with tokenizer.json (byte-level BPE), skipping ids ≥ 50257.

Android (Kotlin)

// build.gradle.kts: implementation("com.microsoft.onnxruntime:onnxruntime-android:1.20.0") or newer
val env = OrtEnvironment.getEnvironment()
val options = OrtSession.SessionOptions().apply { setIntraOpNumThreads(4) }
val encoder = env.createSession(File(dir, "encoder_model_quantized.onnx").path, options)
val decoder = env.createSession(File(dir, "decoder_model_merged_quantized.onnx").path, options)

val features = OnnxTensor.createTensor(env, FloatBuffer.wrap(logMel), longArrayOf(1, 80, 3000))
val hidden = encoder.run(mapOf("input_features" to features)).use { it[0].value as Array<*> }
// ...then the decoding loop above with decoder.run(...)

iOS (Swift)

// Swift Package Manager: https://github.com/microsoft/onnxruntime-swift-package-manager (1.20.0 or newer)
// or CocoaPods: pod 'onnxruntime-objc'
import onnxruntime_objc

let env = try ORTEnv(loggingLevel: .warning)
let options = try ORTSessionOptions()
try options.setIntraOpNumThreads(4)
// Optional: try options.appendCoreMLExecutionProvider(with: ORTCoreMLExecutionProviderOptions())
let encoder = try ORTSession(env: env, modelPath: dir + "/encoder_model_quantized.onnx", sessionOptions: options)
let decoder = try ORTSession(env: env, modelPath: dir + "/decoder_model_merged_quantized.onnx", sessionOptions: options)

let features = try ORTValue(tensorData: NSMutableData(data: logMelData), elementType: .float,
                            shape: [1, 80, 3000])
let out = try encoder.run(withInputs: ["input_features": features], outputNames: ["last_hidden_state"], runOptions: nil)
// ...then the decoding loop above with decoder.run(...)

Python check

import onnxruntime as ort
encoder = ort.InferenceSession("onnx/encoder_model_quantized.onnx")
decoder = ort.InferenceSession("onnx/decoder_model_merged_quantized.onnx")

How it was made

optimum-cli export onnx --task automatic-speech-recognition-with-past --opset 17, then onnxruntime.quantization.quantize_dynamic (QInt8, per-tensor, MatMul and Gather, including the merged decoder's subgraphs). Before quantization, the LM head's run-time transpose of the shared token embedding is folded into a constant, so the LM head is quantized too. The scripts that made and checked these files are in scripts/ (export.py, validate.py).

License and citation

BSD-3-Clause, the license of the original weights. All credit for the model goes to VinAI Research:

@inproceedings{PhoWhisper,
  title     = {{PhoWhisper: Automatic Speech Recognition for Vietnamese}},
  author    = {Thanh-Thien Le and Linh The Nguyen and Dat Quoc Nguyen},
  booktitle = {Proceedings of the ICLR 2024 Tiny Papers track},
  year      = {2024}
}