sokada4/Qwen3.6-27B-GPTQ-Int4

Qwen3.6-27B — GPTQ 4-bit (W4A16)

4-bit GPTQ quantization of Qwen/Qwen3.6-27B, produced with GPTQModel.

Quantization details

Setting	Value
Method	GPTQ, W4A16
Bits	4
Group size	128
Symmetric	yes
desc_act	false
Quantizer	GPTQModel 7.3.4
Calibration data	allenai/c4 (zh/en/ja) + codeparrot/codeparrot-clean-valid, 256 samples total (64 per source), seq_len 4096

The mtp.* tensors (the model's Multi-Token-Prediction / speculative-decoding draft head, 15 tensors) are kept in bf16, not quantized — GPTQModel's qwen3_5.py model definition preserves them via its out_of_model_tensors mechanism rather than passing them through the quantization loop at all (upstream transformers doesn't load MTP weights into the normal inference graph in the first place, so there is nothing to quantize there). This means the MTP speculative-decoding path below works out of the box.

Vision tower weights are similarly untouched — only the language_model.* weights are quantized (this checkpoint is intended for text-only serving).

Serving with vLLM

Tested with vLLM 0.26.0. A dense 27B model at 4-bit needs ~17.7GiB just for weights, which is tight on a single 24GB consumer GPU once you add KV cache — 2x24GB via tensor parallelism gives comfortable headroom, including for MTP speculative decoding:

vllm serve <this-repo> \
  --served-model-name qwen3.6-27b-gptq \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 200000 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Notes:

• --tool-call-parser qwen3_xml and qwen3_coder are aliases for the identical parser class in vLLM — either works.
• --max-model-len 200000 (vs. the model's native 262144) is needed to leave enough KV-cache headroom for the MTP draft path's own verification buffers on 2x24GB. If you have more VRAM (e.g. 4x24GB via --tensor-parallel-size 4, or 2x48GB+ cards) you likely won't need this cut.
• Single-GPU (24GB) serving is only practical without MTP speculative decoding, and even then needs --max-model-len reduced well below native (a single 24GB card OOMs on the CUDA-graph-capture step at native length regardless of KV-cache tuning).

Benchmark (vllm bench serve, random dataset, input-len 128 / output-len 64, TP=2, 2x RTX 3090)

config	concurrency	mean TTFT	mean TPOT	output tok/s	MTP acceptance rate
no speculative decoding	1	145.8ms	16.4ms	54.4	—
+ MTP	1	165.0ms	12.0ms	69.4	84.4%
no speculative decoding	8	815.8ms	22.5ms	222.4	—
+ MTP	8	507.0ms	37.4ms	174.5	88.3%

MTP speculative decoding is a clear win at low concurrency (the realistic case for an interactive coding-agent backend) but trades away batched throughput at higher concurrency — the draft model's extra forward pass competes for GPU compute that's already well-utilized once the batch is large. Choose based on your actual traffic pattern.

Using as a Claude Code / Codex CLI backend

Both point their OpenAI/Anthropic-compatible client at your vLLM server.

Claude Code (Anthropic Messages API, vLLM implements this natively):

export ANTHROPIC_BASE_URL="http://<your-vllm-host>:8000"
export ANTHROPIC_API_KEY="dummy"
export ANTHROPIC_AUTH_TOKEN="dummy"
export ANTHROPIC_DEFAULT_OPUS_MODEL="qwen3.6-27b-gptq"
export ANTHROPIC_DEFAULT_SONNET_MODEL="qwen3.6-27b-gptq"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen3.6-27b-gptq"
# vLLM's prefix caching is broken by Claude Code's per-request attribution header:
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
claude

Codex CLI (OpenAI-compatible):

codex \
  -c preferred_auth_method="apikey" \
  -c model="qwen3.6-27b-gptq" \
  -c model_provider="local-vllm" \
  -c model_providers.local-vllm.name="local-vllm" \
  -c model_providers.local-vllm.base_url="http://<your-vllm-host>:8000/v1" \
  -c model_providers.local-vllm.wire_api="responses" \
  -c model_providers.local-vllm.env_key="DUMMY_KEY" \
  -c model_reasoning_effort="high"

Tool-call reliability caveat: in independent field testing, local Qwen3.6 deployments have shown a meaningfully higher malformed-tool-call rate than Claude/GPT-class hosted models (community reports put it around 10x higher). Gate tool calls / keep a human in the loop rather than running fully autonomous agent loops against this checkpoint.