katolikov/qwen-vknn

Qwen2.5-Coder-0.5B — VKNN .vxm (GPU, on-device)

Qwen2.5-Coder-0.5B (with-past decoder) compiled to VKNN's .vxm format for on-device GPU inference via the Vulkan backend.

file	what
qwen2.5-coder-0.5b-instruct-c1024-prefill256-int4.vxm	Instruct, int4 weights (-Os: calibration-free weight quantization + AWQ outlier columns), C = 1024 + a 256-token prefill bucket (~517 MB, 2.4x smaller than fp16), 100% Vulkan — fastest decode, whole-window prompt prefill
qwen2.5-coder-0.5b-instruct-c1024-prefill256.vxm	Instruct, fp16, C = 1024 + a 256-token prefill bucket (~1.27 GB) — same two-bucket plan at full fp16 quality
qwen2.5-coder-0.5b-instruct-c1024.vxm	Instruct, fp16, C = 1024, single bucket (~1.26 GB) — one plan serves prefill token-by-token and every decode step
qwen2.5-coder-0.5b-decode-c1024.vxm	base completion model, C = 1024 — continues raw text; stop on <|endoftext|> (id 151643)
qwen2.5-coder-0.5b-decode-c256.vxm	base completion model, C = 256
tokenizer.json / vocab.json / merges.txt / *_config.json	the Qwen byte-level BPE tokenizer

Instruct variants answer ChatML-formatted prompts; stop on <\|im_end\|> (id 151645).

The prefill bucket processes the prompt in one 256-token forward instead of token-by-token, cutting time-to-first-token by an order of magnitude on long prompts; the decode bucket then generates with the engine-resident KV cache. vknn_chat selects the buckets automatically (single-bucket files keep the token-by-token path). Two-bucket files need a VKNN build with multi-bucket prompt prefill (2026-07 main or later).

Two-bucket compile (fp16 shown; the int4 file adds -Os --quant-samples 0 instead of --fp16):

vknn_compile out.vxm --fp16 \
  --graph "model.onnx;dim:sequence_length=256;dim:past_sequence_length=1024;dim:total_sequence_length=1280" \
  --graph "model.onnx;dim:sequence_length=1;dim:past_sequence_length=1024;dim:total_sequence_length=1025"

Single-bucket compile:

vknn_compile model.onnx qwen2.5-coder-0.5b-instruct-c1024.vxm --fp16 -O1 \
  --dim sequence_length=1 --dim past_sequence_length=1024

Each fp16 model's greedy decode stream matches its HuggingFace fp32 reference token-for-token; the int4 variant stays coherent under the same ChatML contract (weight-only int4 with per-layer error guards). The VKNN app-demo on-device chat app ships the instruct model wrapped in ChatML.