groxaxo/Qwen3.6-27B-W8A16-AutoRound

Qwen3.6-27B — W8A16 (INT8 weights, FP16 activations)

INT8 weight-only quantization of Qwen/Qwen3.6-27B, produced with Intel AutoRound and exported in the compressed-tensors (pack-quantized) format for direct loading in vLLM. The MTP (multi-token-prediction) head is shipped and works for speculative decoding — see below.

Quantization details

Setting	Value
Scheme	W8A16 (weights INT8, activations FP16)
Algorithm	AutoRound RTN (--algorithm rtn, iters=0)
Group size	128
Symmetric	yes
Format	compressed-tensors / pack-quantized
AutoRound	0.13.1
Vision tower	left in original precision (not quantized)
MTP head	left in original precision + listed in quant ignore (see note)

The language-model Linear layers are quantized to INT8 with group size 128. The visual encoder blocks and the MTP head are kept at full precision, so the checkpoint stays compatible with both multimodal and speculative-decoding paths in vLLM.

Usage (vLLM)

vllm serve groxaxo/Qwen3.6-27B-W8A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --dtype half \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

MTP speculative decoding

vllm serve groxaxo/Qwen3.6-27B-W8A16-AutoRound \
  --tensor-parallel-size 2 --dtype half \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Observed on 2× RTX 3090 with this checkpoint, draft acceptance is workload-dependent: ~80–90 % on factual / repetitive text, dropping to ~60–70 % on diverse generation, with mean acceptance length ≈ 2.2–2.8 (i.e. ~2–3 tokens per decode step). Actual decode speedup depends on batch size and prompt; benchmark on your own hardware.

Important: MTP requires the mtp.* layers in the quant ignore list

AutoRound leaves the mtp.* layers unquantized but does not add them to the quantization_config.ignore list. Without that, vLLM builds the MTP drafter as a W8A16 (Marlin) module, fails to match the plain-FP16 checkpoint weights (Parameter ... not found in params_dict, skip loading), and the drafter runs with garbage weights → 0 % acceptance.

This repo already includes the fix: the 8 unfused MTP Linear names are present in the ignore list of both config.json and quantization_config.json:

mtp.fc
mtp.layers.0.self_attn.{q,k,v,o}_proj
mtp.layers.0.mlp.{gate,up,down}_proj

(compressed-tensors un-fuses qkv_proj/gate_up_proj, so the individual shard names are listed.) If you re-quantize this model yourself, apply the same edit. Note that vLLM's torch.compile cache is not invalidated by per-layer quant-scheme changes — if you change the ignore list on an already-booted model, rm -rf ~/.cache/vllm/torch_compile_cache before restarting.

Notes

W8A16 roughly halves the weight footprint versus BF16 (~30 GB vs ~54 GB) while keeping activations in FP16. Ampere GPUs (e.g. RTX 3090) support INT8 weight-only kernels natively. Calibration-free RTN was used; for the smallest possible accuracy gap you can re-quantize with the iterative AutoRound algorithm and a calibration dataset.