pdnhan/Qwen2.5-3B-Instruct-mobile

Qwen2.5-3B-Instruct for mobile (ONNX, int4)

Qwen/Qwen2.5-3B-Instruct converted to a 4-bit ONNX model that ONNX Runtime runs on Android and iOS, with ready-to-use Kotlin and Swift code: the Qwen tokenizer, the generation loop with a KV cache, and a speech-transcript refiner for Vietnamese.

The refiner turns raw speech-recognition output (for example PhoWhisper) into clean written text: it adds punctuation and capitals, fixes misheard words from context, removes fillers and stutters, and writes spoken lists as bullet points. It is what the Thì Thầm app runs after PhoWhisper.

Transcript	Refined
ờ hôm nay chúng ta sẽ họp về ba vấn đề một là ngân sách quý tới hai là tuyển thêm nhân viên và ba là kế hoạch du lịch công ty	Hôm nay chúng ta sẽ họp về ba vấn đề: - Ngân sách quý tới. - Tuyển thêm nhân viên. - Kế hoạch du lịch công ty.
thủ tướng phạm minh chính đã tiếp bộ trưởng nông nghiệp mỹ tại hà lội vào sáng nay	Thủ tướng Phạm Minh Chính đã tiếp Bộ trưởng Nông nghiệp Mỹ tại Hà Nội vào sáng nay.
bạn có biết mấy giờ rồi không	Bạn có biết mấy giờ rồi không?
nhớ mua giúp mình mấy thứ nhé trứng sữa tươi bánh mì và một ít táo	Nhớ mua giúp mình mấy thứ nhé: - Trứng - Sữa tươi - Bánh mì - Một ít táo
các bước cài đặt như sau đầu tiên tải ứng dụng sau đó đăng nhập bằng số điện thoại cuối cùng là bật thông báo	Các bước cài đặt như sau: - Đầu tiên tải ứng dụng. - Sau đó đăng nhập bằng số điện thoại. - Cuối cùng là bật thông báo.
ừm thì là cuộc họp ngày mai được dời sang hai giờ chiều nha mọi người	Ừm, thì là cuộc họp ngày mai được dời sang hai giờ chiều nha. Mọi người.

The weights are those of the original model; only the format and the precision differ.

Files

File	What	Size	SHA-256
onnx/model.onnx	int4 model graph	287.8 kB	492f2a94c23b7f086f8ea18c0d780f107a76b0a7521ebd756f73b2232221f898
onnx/model.onnx.data	int4 weights	1,946.3 MB	c03d0e208d93f2bb29b35e5a3c276f3396841d48e171624b3863b1326de376c9
onnx/genai_config.json	model settings	1.6 kB	2915685a9fe18a6de5764e8b6efe5615ed68060bc63866a88f937cce3db298dd
vocab.txt	tokenizer vocabulary	1.5 MB	f79f19c08fbb0c35f098f503b219845c7fa03f42432e991fe82b52ff8f257014
merges.txt	tokenizer merges	1.7 MB	599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3
onnx/tokenizer.json	tokenizer for Python	11.4 MB	3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8

Mobile apps download onnx/model.onnx, onnx/model.onnx.data, onnx/genai_config.json, vocab.txt and merges.txt (about 1950 MB in total) and use the code in android/ or ios/.

• onnx/model.onnx + onnx/model.onnx.data: built with the ONNX Runtime GenAI model builder (-p int4 -e cpu): GroupQueryAttention with a KV cache, 4-bit MatMulNBits (symmetric, blocks of 32, accuracy_level 4) and the tied embedding as a 4-bit GatherBlockQuantized shared with the LM head, which only computes the last position's logits. Plain ONNX Runtime (1.22 or newer; tested with 1.31.1) runs it; the folder also works as is with ONNX Runtime GenAI.
• vocab.txt: the byte-level BPE vocabulary, one token per line, line number = token id (special tokens included).
• merges.txt: the BPE merges, one a b pair per line, by rank.
• onnx/tokenizer.json, onnx/tokenizer_config.json, onnx/chat_template.jinja: the original tokenizer files, for Python and ONNX Runtime GenAI.

Model interface

	Name	Type	Shape
input	input_ids	int64	[batch, tokens]
input	attention_mask	int64	[batch, past + tokens]: 1 for every token so far
input	past_key_values.{0..35}.key / .value	float32	[batch, 2, past, 128] (past = 0 for the first run)
output	logits	float32	[batch, 1, 151936]: next-token scores after the last input token
output	present.{0..35}.key / .value	float32	[batch, 2, past + tokens, 128]: the next run's past_key_values

Accuracy of the refiner

On 300 PhoWhisper transcripts of FLEURS Vietnamese test clips (the first 150 clips, transcribed by the int8 PhoWhisper tiny and small models of pdnhan/PhoWhisper-small-mobile and its siblings), against the original transcriptions:

Recognizer	Word error rate of the transcript	after the refiner
PhoWhisper small (int8)	12.3%	11.8%
PhoWhisper tiny (int8)	28.9%	28.6%

Word error rate is measured case- and punctuation-insensitively; FLEURS writes numbers as both digits and words, so a number the refiner writes in digits can count as an error. Punctuation (. , ? ! : on the words both texts share): precision 0.67, recall 0.61, F1 0.64. Time per transcript on an x86 desktop CPU (8 threads): 1.34 s; phones are several times slower.

The tokenizer gives the same ids as Hugging Face tokenizers on all 716 test texts, and the prompt is token for token the model's own chat template. The Kotlin code (JVM, ONNX Runtime 1.31.1) and the Swift code give the same text as the Python reference on the test cases.

How the refiner works

python/qwen_refine.py is the readable reference (numpy, onnxruntime and regex); the Kotlin and Swift code follow it step by step.

1. Tokenizer: NFC, Qwen2's pre-tokenizer regex, byte-level BPE. Special tokens are never parsed from the transcript: the prompt inserts their ids, so a transcript cannot start a new chat turn.
2. Prompt: a system message and four example turns (fillers, a spoken list, a question that must not be answered, misheard words, unknown names kept). They are the same for every transcript, so their KV cache is computed once when the model loads.
3. Decoding: greedy, until <|im_end|>, at most 1.5 × the transcript's tokens + 32.
4. Merge: small models reword, drop words and "correct" names they do not know. The output is aligned word by word with the transcript (longest common subsequence) and only trusted edits are kept: punctuation, capitals, line breaks and bullets; fillers (ờ, ừm, …), stutters (tôi tôi) and list markers (thứ hai là) left out; a word replaced by a similar-sounding one (hà lội → Hà Nội: at most one letter apart, two for long words, ignoring tone marks); a spoken number written in digits with the same value (hai nghìn không trăm hai mươi sáu → 2026). Anything else is undone with the transcript's own words.

Android (Kotlin)

Copy android/QwenRefiner.kt into the app. It uses only the Kotlin standard library and the ONNX Runtime Java API.

// build.gradle.kts: implementation("com.microsoft.onnxruntime:onnxruntime-android:1.22.0") or newer
// dir: a directory with vocab.txt, merges.txt, onnx/genai_config.json, onnx/model.onnx, onnx/model.onnx.data
val refiner = QwenRefiner.load(dir, threads = 4)   // keep one instance: loading takes a few seconds
val text = refiner.refine("thủ tướng phạm minh chính đã tiếp bộ trưởng nông nghiệp mỹ tại hà lội vào sáng nay")
// "Thủ tướng Phạm Minh Chính đã tiếp Bộ trưởng Nông nghiệp Mỹ tại Hà Nội vào sáng nay."

Call refine off the main thread. The model needs about 2.5 GB of RAM while it runs.

iOS (Swift)

Add the Swift package in ios/ (it depends on onnxruntime-swift-package-manager), or copy ios/Sources/QwenRefiner/*.swift into an app that already uses ONNX Runtime (SPM module OnnxRuntimeBindings or CocoaPods onnxruntime-objc, 1.22 or newer).

import QwenRefiner

let refiner = try QwenRefiner(directory: dir, threads: 4)
let text = try refiner.refine("thủ tướng phạm minh chính đã tiếp bộ trưởng nông nghiệp mỹ tại hà lội vào sáng nay")

The Swift code was compiled and checked on Linux (Swift 6.0) against a stand-in for the ONNX Runtime module that replays the Python reference's generated tokens, not yet on an iPhone.

Python

from huggingface_hub import snapshot_download
import sys

repo = snapshot_download("pdnhan/Qwen2.5-3B-Instruct-mobile")
sys.path.insert(0, f"{repo}/python")
from qwen_refine import Refiner  # numpy, onnxruntime>=1.22, regex

refiner = Refiner(repo)
print(refiner.refine("bạn có biết mấy giờ rồi không"))  # Bạn có biết mấy giờ rồi không?

How it was made

The scripts are in scripts/: export.py runs the ONNX Runtime GenAI model builder and writes the tokenizer files, asr_hyps.py transcribes the FLEURS clips with PhoWhisper, validate.py checks the tokenizer and the prompt and measures the table above. The Kotlin and Swift checks are in scripts/kotlin and scripts/swift-check.

License and credit

The Qwen Research License of the original model: non-commercial use only. The code in this repo (android/, ios/, python/, scripts/) is Apache 2.0, the same as in pdnhan/Qwen2.5-1.5B-Instruct-mobile, which uses the Apache 2.0 1.5B model. All credit for the model goes to the Qwen team (Built with Qwen). FLEURS (CC BY 4.0) and PhoWhisper (BSD-3-Clause) were used for evaluation only.