pdnhan/PhoWhisper-tiny-mobile

PhoWhisper-tiny for mobile (ONNX)

vinai/PhoWhisper-tiny (39M parameters), VinAI's Vietnamese fine-tune of Whisper, converted to ONNX for on-device speech recognition with ONNX Runtime on Android and iOS. The weights are unchanged; only the format differs (and the precision, for the int8 files).

Files

File	Precision	Size	SHA-256
onnx/decoder_model_merged.onnx	fp32	118.5 MB	c0c88bdddb79e460b1be73db62673fec1db071a3789c552e98fda3be4fb8e543
onnx/decoder_model_merged_quantized.onnx	int8	50.0 MB	5849d47d2fa8655ba9ae3860b12049032cc6cfc8058e5a34b9a91071d18e1901
onnx/encoder_model.onnx	fp32	32.9 MB	f1ceaa57ff91f20ef032ffac2d4cdb60021c639e783128f4992f7ad4e742b6b2
onnx/encoder_model_quantized.onnx	int8	11.7 MB	1136f10145c2f33902f8e7aee46d84ee9d8bb6493224c1c0f3883b5cb80b3721

Mobile apps should download the two *_quantized.onnx files (dynamic int8: MatMul and Gather weights). The fp32 files are the reference export, for desktop use or further conversion. The tokenizer and feature-extractor files are the original PhoWhisper ones.

Accuracy

Word error rate on the first 30 clips of the FLEURS Vietnamese test split, greedy decoding, text lower-cased with punctuation removed:

Precision	WER
fp32	12.6%
int8	14.5%

This is a sanity check of the conversion, not a benchmark: see the PhoWhisper paper for proper evaluation.

Model interface

These are standard Optimum exports using only standard ONNX operators (opset 17, IR version 9), so the stock ONNX Runtime packages for Android and iOS run them, with no custom-op library. The int8 files give identical tokens on ONNX Runtime 1.20.0 and 1.31.1.

Encoder: input_features float32 [1, 80, 3000] (log-Mel spectrogram of 30 s of 16 kHz mono audio, computed as in WhisperFeatureExtractor: 400-sample FFT window, hop 160, padded with zeros) → last_hidden_state float32 [1, 1500, 384].

Decoder (merged, with KV cache): inputs input_ids int64 [1, n], encoder_hidden_states, use_cache_branch bool [1], and past_key_values.<layer>.{decoder,encoder}.{key,value}; outputs logits and present.<layer>.{decoder,encoder}.{key,value}.

Decoding loop (greedy):

1. Run the encoder once.
2. First decoder step: input_ids = [50258, 50278, 50359, 50363] (<|startoftranscript|><|vi|><|transcribe|><|notimestamps|>), use_cache_branch = false, and every past_key_values.* an empty tensor [1, 6, 0, 64].
3. Take the argmax of the last position of logits. Stop on <|endoftext|> (50257).
4. Next steps: input_ids = [[token]], use_cache_branch = true. Feed back present.*.decoder.* every step and present.*.encoder.* from the first step only (the encoder cache does not change).
5. Decode the tokens with tokenizer.json (byte-level BPE), skipping ids ≥ 50257.

Android (Kotlin)

// build.gradle.kts: implementation("com.microsoft.onnxruntime:onnxruntime-android:1.20.0") or newer
val env = OrtEnvironment.getEnvironment()
val options = OrtSession.SessionOptions().apply { setIntraOpNumThreads(4) }
val encoder = env.createSession(File(dir, "encoder_model_quantized.onnx").path, options)
val decoder = env.createSession(File(dir, "decoder_model_merged_quantized.onnx").path, options)

val features = OnnxTensor.createTensor(env, FloatBuffer.wrap(logMel), longArrayOf(1, 80, 3000))
val hidden = encoder.run(mapOf("input_features" to features)).use { it[0].value as Array<*> }
// ...then the decoding loop above with decoder.run(...)

iOS (Swift)

// Swift Package Manager: https://github.com/microsoft/onnxruntime-swift-package-manager (1.20.0 or newer)
// or CocoaPods: pod 'onnxruntime-objc'
import onnxruntime_objc

let env = try ORTEnv(loggingLevel: .warning)
let options = try ORTSessionOptions()
try options.setIntraOpNumThreads(4)
// Optional: try options.appendCoreMLExecutionProvider(with: ORTCoreMLExecutionProviderOptions())
let encoder = try ORTSession(env: env, modelPath: dir + "/encoder_model_quantized.onnx", sessionOptions: options)
let decoder = try ORTSession(env: env, modelPath: dir + "/decoder_model_merged_quantized.onnx", sessionOptions: options)

let features = try ORTValue(tensorData: NSMutableData(data: logMelData), elementType: .float,
                            shape: [1, 80, 3000])
let out = try encoder.run(withInputs: ["input_features": features], outputNames: ["last_hidden_state"], runOptions: nil)
// ...then the decoding loop above with decoder.run(...)

Python check

import onnxruntime as ort
encoder = ort.InferenceSession("onnx/encoder_model_quantized.onnx")
decoder = ort.InferenceSession("onnx/decoder_model_merged_quantized.onnx")

How it was made

optimum-cli export onnx --task automatic-speech-recognition-with-past --opset 17, then onnxruntime.quantization.quantize_dynamic (QInt8, per-tensor, MatMul and Gather, including the merged decoder's subgraphs). Before quantization, the LM head's run-time transpose of the shared token embedding is folded into a constant, so the LM head is quantized too. The scripts that made and checked these files are in scripts/ (export.py, validate.py).

License and citation

BSD-3-Clause, the license of the original weights. All credit for the model goes to VinAI Research:

@inproceedings{PhoWhisper,
  title     = {{PhoWhisper: Automatic Speech Recognition for Vietnamese}},
  author    = {Thanh-Thien Le and Linh The Nguyen and Dat Quoc Nguyen},
  booktitle = {Proceedings of the ICLR 2024 Tiny Papers track},
  year      = {2024}
}