sverinn/Qwopus3-5-9B-Coder-Flash-AWQ

Qwopus3-5-9B-Coder-Flash — AWQ W4A16

4-bit activation-aware quantization (AWQ, weight-only W4A16) of Jackrong/Qwopus3-5-9B-Coder-Flash (a Qwen3.5-9B hybrid-attention coder VLM), produced with llm-compressor and ready for vLLM via the compressed-tensors format.

8.5 GB on disk (vs 19.3 GB bf16) while keeping the full 262k context window comfortably inside a single 24 GB consumer GPU (RTX 3090).

Quantization recipe

Algorithm	AWQ (activation-aware scaling, weight-only int4)
Scheme	W4A16 — int4, group size 128, symmetric; activations bf16
Tool	llm-compressor 0.15.0 (compressed-tensors 0.20.1)
Calibration	HuggingFaceH4/ultrachat_200k (train_sft), 256 samples, max seq 2048
Targets	all Linear modules
Left unquantized	lm_head, norms, and all vision/multimodal modules (re:.*visual.*, re:.*vision_tower.*, re:.*vision_model.*, re:.*vision.*, re:.*multi_modal_projector.*, re:.*merger.*)
MTP head	copied unchanged (bf16)

The hybrid GatedDeltaNet linear-attention projections are quantized; AWQ mappings were inferred per attention type (8 full-attn / 24 linear-attn layers).

Benchmarks (RTX 3090 @ 350 W, vLLM 0.32)

Quality — 24-task exec-verified suite (8 code, 8 terminal, 8 tool-calling), temp 0:

Checkpoint	Code	Terminal	Tool-calls	Total
AWQ W4A16 (this repo)	7/8*	7/8	8/8	22/24
Base bf16 (reference)	7/8*	7/8	8/8	22/24

No measurable quality loss from quantization — both checkpoints fail the same two tasks (fizzbuzz*, rm_pycache) and pass the other 22.

* the fizzbuzz failure is temperature-0 batching nondeterminism (the model's code is correct and passes on retest), identical in both checkpoints.

Long context — needle-in-a-haystack at 201,643 tokens: exact passphrase retrieved ✅

Speed (c1, --max-model-len 262144, measured on RTX 3090 @ 350 W):

	AWQ W4A16
Prefill	~3,850 tok/s
Decode	~106 tok/s
VRAM @ 262k ctx	~20.7 GB (KV cache fits 353k tokens)

Usage (vLLM)

vllm serve sverinn/Qwopus3-5-9B-Coder-Flash-AWQ \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Vision inputs work out of the box (the vision tower is kept in bf16).

Notes

• Marlin int4 kernels are used automatically on Ampere+ GPUs.
• The base model's chat template (enable_thinking, XML tool-call format) is preserved unchanged.

Credits

• Jackrong for the base Qwopus3-5-9B-Coder-Flash model.
• unsloth and the Qwen team for the underlying Qwen3.5-9B base.
• The vLLM project for llm-compressor (AWQ quantization) and vLLM (serving).
• Quantization, benchmarks and this model card by sverinn. Inference speeds measured on a single RTX 3090 @ 350 W.