quickasr/Qwen3-ASR-1.7B-q8

Qwen3-ASR-1.7B int8 for quickasr

An int8 copy of Qwen/Qwen3-ASR-1.7B (revision 7278e1e70fe206f11671096ffdd38061171dd6e5), made with quickasr quantize for quickasr, a low-latency streaming speech-to-text runtime for CPUs (with an optional GPU path). It also powers livesub, real-time bilingual captions for livestreams, videos and calls.

The weights were modified, and the file runs only on quickasr: it is not loadable by transformers, vLLM or the qwen_asr package, and it is not GGUF.

Usage

Needs quickasr 0.4.0 or later.

git clone https://github.com/nexustar/quickasr && cd quickasr && make blas
./download_model.sh --model large --q8
# or: hf download quickasr/Qwen3-ASR-1.7B-q8 --local-dir qwen3-asr-1.7b-q8
./quickasr -d qwen3-asr-1.7b-q8 -i audio.wav
./quickasr -d qwen3-asr-1.7b-q8 -i audio.wav --stream

The int8 format is detected from the weights; no option is needed.

What changed

• The seven linear weights of each decoder layer (q/k/v/o, gate, up, down) are int8, one f32 scale per 32 weights (d = max|v| / 127, q = round(v / d)).
• The audio encoder keeps its precision, and so do the token embeddings, which are also the output layer (bf16): an int8 output layer flips near-tied tokens.
• The file is a plain safetensors file marked "quickasr_format":"q8-1" in its metadata. q8-1 is quickasr's own layout, version 1, and is unrelated to ggml's Q8_1 type. See MODEL.md.

To make it yourself from the original model (or a fine-tune of it):

./quickasr quantize qwen3-asr-1.7b qwen3-asr-1.7b-q8

Performance

Apple M1 Pro, CPU, real-time streaming with 2 s chunks, 180 s each of Chinese (AISHELL-1 test) and English (LibriSpeech test-clean). Error is CER for zh and WER for en; latency is the median time from the end of a word to its first appearance.

Format	err zh / en	CPU s per s of audio (zh / en)	latency p50 zh / en	Peak RSS
bf16	1.12% / 2.48%	1.76 / 2.10	2.00 / 2.30 s	7.0 GB
int8	1.12% / 2.48%	1.49 / 1.80	1.88 / 2.15 s	4.5 GB

On 66 Japanese conversation clips (4 s chunks) the CER was 11.1% for both bf16 and int8. Text can differ slightly from the source model. More numbers, including the GPU path: docs/performance.md.

License

Apache-2.0, the license of the source model.