pdnhan/PhoWhisper-large-mobile

PhoWhisper-large for mobile (ONNX)

vinai/PhoWhisper-large (1.55B parameters), VinAI's Vietnamese fine-tune of Whisper, converted to ONNX for on-device speech recognition with ONNX Runtime on Android and iOS. The weights are unchanged; only the format differs (and the precision, for the int8 files).

Files

File	Precision	Size	SHA-256
onnx/decoder_model_merged.onnx	fp32	2.1 MB	eb36fba92fe9f856c834594f08cf364431a6aae83a836235e68a7293cdbba544
onnx/decoder_model_merged.onnx_data	fp32	3,626.1 MB	3825d5f4a09ac098cb9653c8772ea378f34ee22fd4a69ff516c53c39ea0031c0
onnx/decoder_model_merged_quantized.onnx	int8	978.5 MB	a1dd5c1d7b819e1a7b35656c22a1bf5ea36478b97344cc439ddd98f072336af3
onnx/encoder_model.onnx	fp32	0.4 MB	6f8e490b040a72de7adc7a8fd9cc9bab342b557a057032646eb8fdb7fd00f8f3
onnx/encoder_model.onnx_data	fp32	2,547.1 MB	6a221ae0d91edc8bdf83931968b8ec61eb34fb6c103cd50cea7bf98b0b559eeb
onnx/encoder_model_quantized.onnx	int8	660.5 MB	57cfd599038a8f0df27b0c2ac120c07101d67c2f514f7d741ac0ad0e7eb95e80

Mobile apps should download the two *_quantized.onnx files (dynamic int8: MatMul and Gather weights). The fp32 files are the reference export, for desktop use or further conversion. The tokenizer and feature-extractor files are the original PhoWhisper ones.

Accuracy

Word error rate on the first 30 clips of the FLEURS Vietnamese test split, greedy decoding, text lower-cased with punctuation removed:

Precision	WER
fp32	4.0%
int8	4.1%

This is a sanity check of the conversion, not a benchmark: see the PhoWhisper paper for proper evaluation.

Model interface

These are standard Optimum exports using only standard ONNX operators (opset 17, IR version 9), so the stock ONNX Runtime packages for Android and iOS run them, with no custom-op library. The int8 files give identical tokens on ONNX Runtime 1.20.0 and 1.31.1.

Encoder: input_features float32 [1, 80, 3000] (log-Mel spectrogram of 30 s of 16 kHz mono audio, computed as in WhisperFeatureExtractor: 400-sample FFT window, hop 160, padded with zeros) → last_hidden_state float32 [1, 1500, 1280].

Decoder (merged, with KV cache): inputs input_ids int64 [1, n], encoder_hidden_states, use_cache_branch bool [1], and past_key_values.<layer>.{decoder,encoder}.{key,value}; outputs logits and present.<layer>.{decoder,encoder}.{key,value}.

Decoding loop (greedy):

1. Run the encoder once.
2. First decoder step: input_ids = [50258, 50278, 50359, 50363] (<|startoftranscript|><|vi|><|transcribe|><|notimestamps|>), use_cache_branch = false, and every past_key_values.* an empty tensor [1, 20, 0, 64].
3. Take the argmax of the last position of logits. Stop on <|endoftext|> (50257).
4. Next steps: input_ids = [[token]], use_cache_branch = true. Feed back present.*.decoder.* every step and present.*.encoder.* from the first step only (the encoder cache does not change).
5. Decode the tokens with tokenizer.json (byte-level BPE), skipping ids ≥ 50257.

Android (Kotlin)

// build.gradle.kts: implementation("com.microsoft.onnxruntime:onnxruntime-android:1.20.0") or newer
val env = OrtEnvironment.getEnvironment()
val options = OrtSession.SessionOptions().apply { setIntraOpNumThreads(4) }
val encoder = env.createSession(File(dir, "encoder_model_quantized.onnx").path, options)
val decoder = env.createSession(File(dir, "decoder_model_merged_quantized.onnx").path, options)

val features = OnnxTensor.createTensor(env, FloatBuffer.wrap(logMel), longArrayOf(1, 80, 3000))
val hidden = encoder.run(mapOf("input_features" to features)).use { it[0].value as Array<*> }
// ...then the decoding loop above with decoder.run(...)

iOS (Swift)

// Swift Package Manager: https://github.com/microsoft/onnxruntime-swift-package-manager (1.20.0 or newer)
// or CocoaPods: pod 'onnxruntime-objc'
import onnxruntime_objc

let env = try ORTEnv(loggingLevel: .warning)
let options = try ORTSessionOptions()
try options.setIntraOpNumThreads(4)
// Optional: try options.appendCoreMLExecutionProvider(with: ORTCoreMLExecutionProviderOptions())
let encoder = try ORTSession(env: env, modelPath: dir + "/encoder_model_quantized.onnx", sessionOptions: options)
let decoder = try ORTSession(env: env, modelPath: dir + "/decoder_model_merged_quantized.onnx", sessionOptions: options)

let features = try ORTValue(tensorData: NSMutableData(data: logMelData), elementType: .float,
                            shape: [1, 80, 3000])
let out = try encoder.run(withInputs: ["input_features": features], outputNames: ["last_hidden_state"], runOptions: nil)
// ...then the decoding loop above with decoder.run(...)

Python check

import onnxruntime as ort
encoder = ort.InferenceSession("onnx/encoder_model_quantized.onnx")
decoder = ort.InferenceSession("onnx/decoder_model_merged_quantized.onnx")

How it was made

optimum-cli export onnx --task automatic-speech-recognition-with-past --opset 17, then onnxruntime.quantization.quantize_dynamic (QInt8, per-tensor, MatMul and Gather, including the merged decoder's subgraphs). Before quantization, the LM head's run-time transpose of the shared token embedding is folded into a constant, so the LM head is quantized too. The scripts that made and checked these files are in scripts/ (export.py, validate.py).

License and citation

BSD-3-Clause, the license of the original weights. All credit for the model goes to VinAI Research:

@inproceedings{PhoWhisper,
  title     = {{PhoWhisper: Automatic Speech Recognition for Vietnamese}},
  author    = {Thanh-Thien Le and Linh The Nguyen and Dat Quoc Nguyen},
  booktitle = {Proceedings of the ICLR 2024 Tiny Papers track},
  year      = {2024}
}