quickasr/Qwen3-ASR-0.6B-q8

Qwen3-ASR-0.6B int8 for quickasr

An int8 copy of Qwen/Qwen3-ASR-0.6B (revision 5eb144179a02acc5e5ba31e748d22b0cf3e303b0), made with quickasr quantize for quickasr, a low-latency streaming speech-to-text runtime for CPUs (with an optional GPU path). It also powers livesub, real-time bilingual captions for livestreams, videos and calls.

The weights were modified, and the file runs only on quickasr: it is not loadable by transformers, vLLM or the qwen_asr package, and it is not GGUF.

Usage

Needs quickasr 0.4.0 or later.

git clone https://github.com/nexustar/quickasr && cd quickasr && make blas
./download_model.sh --model small --q8
# or: hf download quickasr/Qwen3-ASR-0.6B-q8 --local-dir qwen3-asr-0.6b-q8
./quickasr -d qwen3-asr-0.6b-q8 -i audio.wav
./quickasr -d qwen3-asr-0.6b-q8 -i audio.wav --stream

The int8 format is detected from the weights; no option is needed.

What changed

• The seven linear weights of each decoder layer (q/k/v/o, gate, up, down) are int8, one f32 scale per 32 weights (d = max|v| / 127, q = round(v / d)).
• The audio encoder keeps its precision, and so do the token embeddings, which are also the output layer (bf16): an int8 output layer flips near-tied tokens.
• The file is a plain safetensors file marked "quickasr_format":"q8-1" in its metadata. q8-1 is quickasr's own layout, version 1, and is unrelated to ggml's Q8_1 type. See MODEL.md.

To make it yourself from the original model (or a fine-tune of it):

./quickasr quantize qwen3-asr-0.6b qwen3-asr-0.6b-q8

Performance

Apple M1 Pro, CPU, real-time streaming with 2 s chunks, 180 s each of Chinese (AISHELL-1 test) and English (LibriSpeech test-clean). Error is CER for zh and WER for en; latency is the median time from the end of a word to its first appearance.

Format	err zh / en	CPU s per s of audio (zh / en)	latency p50 zh / en	Peak RSS
bf16	2.68% / 2.48%	0.73 / 0.92	1.33 / 1.57 s	3.0 GB
int8	2.46% / 2.48%	0.64 / 0.80	1.33 / 1.55 s	2.3 GB

Text can differ slightly from the source model. More numbers, including the GPU path: docs/performance.md.

License

Apache-2.0, the license of the source model.