pdnhan/PhoWhisper-base-mobile

PhoWhisper-base for mobile (ONNX)

vinai/PhoWhisper-base (74M parameters), VinAI's Vietnamese fine-tune of Whisper, converted to ONNX for on-device speech recognition with ONNX Runtime on Android and iOS. The weights are unchanged; only the format differs (and the precision, for the int8 files).

Files

File	Precision	Size	SHA-256
onnx/decoder_model_merged.onnx	fp32	208.4 MB	2dbcbf9e4d3867c9a6af4cfa3357dbf639fd80081c7b9ba491ccc53057e61a3c
onnx/decoder_model_merged_quantized.onnx	int8	79.4 MB	5037ae417dd1b6387d15a0785f567db493bde230ba3f03352a827ff9246c9797
onnx/encoder_model.onnx	fp32	82.4 MB	8c8738cf38a8cfb71e78a7bcdac68bb060cc9fde7ba03c188fc4b1e512ddb665
onnx/encoder_model_quantized.onnx	int8	25.9 MB	e277fe5bbf55fc434edce192c57b049124805f474687a13396035566904eaddd

Mobile apps should download the two *_quantized.onnx files (dynamic int8: MatMul and Gather weights). The fp32 files are the reference export, for desktop use or further conversion. The tokenizer and feature-extractor files are the original PhoWhisper ones.

Accuracy

Word error rate on the first 30 clips of the FLEURS Vietnamese test split, greedy decoding, text lower-cased with punctuation removed:

Precision	WER
fp32	8.2%
int8	8.8%

This is a sanity check of the conversion, not a benchmark: see the PhoWhisper paper for proper evaluation.

Model interface

These are standard Optimum exports using only standard ONNX operators (opset 17, IR version 9), so the stock ONNX Runtime packages for Android and iOS run them, with no custom-op library. The int8 files give identical tokens on ONNX Runtime 1.20.0 and 1.31.1.

Encoder: input_features float32 [1, 80, 3000] (log-Mel spectrogram of 30 s of 16 kHz mono audio, computed as in WhisperFeatureExtractor: 400-sample FFT window, hop 160, padded with zeros) → last_hidden_state float32 [1, 1500, 512].

Decoder (merged, with KV cache): inputs input_ids int64 [1, n], encoder_hidden_states, use_cache_branch bool [1], and past_key_values.<layer>.{decoder,encoder}.{key,value}; outputs logits and present.<layer>.{decoder,encoder}.{key,value}.

Decoding loop (greedy):

1. Run the encoder once.
2. First decoder step: input_ids = [50258, 50278, 50359, 50363] (<|startoftranscript|><|vi|><|transcribe|><|notimestamps|>), use_cache_branch = false, and every past_key_values.* an empty tensor [1, 8, 0, 64].
3. Take the argmax of the last position of logits. Stop on <|endoftext|> (50257).
4. Next steps: input_ids = [[token]], use_cache_branch = true. Feed back present.*.decoder.* every step and present.*.encoder.* from the first step only (the encoder cache does not change).
5. Decode the tokens with tokenizer.json (byte-level BPE), skipping ids ≥ 50257.

Android (Kotlin)

// build.gradle.kts: implementation("com.microsoft.onnxruntime:onnxruntime-android:1.20.0") or newer
val env = OrtEnvironment.getEnvironment()
val options = OrtSession.SessionOptions().apply { setIntraOpNumThreads(4) }
val encoder = env.createSession(File(dir, "encoder_model_quantized.onnx").path, options)
val decoder = env.createSession(File(dir, "decoder_model_merged_quantized.onnx").path, options)

val features = OnnxTensor.createTensor(env, FloatBuffer.wrap(logMel), longArrayOf(1, 80, 3000))
val hidden = encoder.run(mapOf("input_features" to features)).use { it[0].value as Array<*> }
// ...then the decoding loop above with decoder.run(...)

iOS (Swift)

// Swift Package Manager: https://github.com/microsoft/onnxruntime-swift-package-manager (1.20.0 or newer)
// or CocoaPods: pod 'onnxruntime-objc'
import onnxruntime_objc

let env = try ORTEnv(loggingLevel: .warning)
let options = try ORTSessionOptions()
try options.setIntraOpNumThreads(4)
// Optional: try options.appendCoreMLExecutionProvider(with: ORTCoreMLExecutionProviderOptions())
let encoder = try ORTSession(env: env, modelPath: dir + "/encoder_model_quantized.onnx", sessionOptions: options)
let decoder = try ORTSession(env: env, modelPath: dir + "/decoder_model_merged_quantized.onnx", sessionOptions: options)

let features = try ORTValue(tensorData: NSMutableData(data: logMelData), elementType: .float,
                            shape: [1, 80, 3000])
let out = try encoder.run(withInputs: ["input_features": features], outputNames: ["last_hidden_state"], runOptions: nil)
// ...then the decoding loop above with decoder.run(...)

Python check

import onnxruntime as ort
encoder = ort.InferenceSession("onnx/encoder_model_quantized.onnx")
decoder = ort.InferenceSession("onnx/decoder_model_merged_quantized.onnx")

How it was made

optimum-cli export onnx --task automatic-speech-recognition-with-past --opset 17, then onnxruntime.quantization.quantize_dynamic (QInt8, per-tensor, MatMul and Gather, including the merged decoder's subgraphs). Before quantization, the LM head's run-time transpose of the shared token embedding is folded into a constant, so the LM head is quantized too. The scripts that made and checked these files are in scripts/ (export.py, validate.py).

License and citation

BSD-3-Clause, the license of the original weights. All credit for the model goes to VinAI Research:

@inproceedings{PhoWhisper,
  title     = {{PhoWhisper: Automatic Speech Recognition for Vietnamese}},
  author    = {Thanh-Thien Le and Linh The Nguyen and Dat Quoc Nguyen},
  booktitle = {Proceedings of the ICLR 2024 Tiny Papers track},
  year      = {2024}
}