Qwen3-Coder-Next-GPTQ

GPTQ (error-compensated weight quantization) (W4A16) quantization of Qwen/Qwen3-Coder-Next, made with llm-compressor in the compressed-tensors format for efficient serving with vLLM and SGLang.

Chinese version: README_zh.md

Why W4A16 (and not the official FP8)

Qwen publishes only an FP8 checkpoint for this model. This repo is the 4-bit (W4A16) build: ~40 GiB on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left untouched (see the protected layers below), so generation tracks the base model.

Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the original on a sparse MoE where the wrong rounding on a router would cascade.

Model

Base model	Qwen/Qwen3-Coder-Next
Architecture	qwen3_next MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing
Context	256K
Method	GPTQ (error-compensated weight quantization), W4A16, group size 128
Format	compressed-tensors (safetensors)
Size	~40 GiB
Tool	llm-compressor
License	Apache-2.0 (inherited from base model)

What is quantized, and what is protected

GPTQ quantizes weights column-by-column and compensates the residual error into the not-yet-quantized columns using second-order (Hessian) information. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:

Tensors	Kept	Reason
*.mlp.gate (router)	bf16	a compressed router sends tokens to the wrong experts
lm_head	bf16	the output projection; a global 4-bit type would squeeze it hardest
everything else (expert + attention linears)	W4A16	the bulk of the weights, where 4-bit pays off

Quality (perplexity)

Perplexity on wikitext-2-raw (test), n_ctx = 512, 12 chunks, measured with vLLM — the same recipe across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.

Weights	PPL	vs bf16
bf16 (master, reference)	7.74	—
this repo (GPTQ W4A16)	9.34	+20.5%

Serving

vllm serve qtum/Qwen3-Coder-Next-GPTQ

The quantization scheme is declared in config.json and detected automatically — no extra flags. Works with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it comfortably; on 24 GB use tensor/pipeline parallel or offload.

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Other quantizations

Format	Repo	Note
AWQ (W4A16)	qtum/Qwen3-Coder-Next-AWQ	the other 4-bit build
FP8 (official)	Qwen/Qwen3-Coder-Next-FP8	8-bit, ~2× this size

---

Quantized by qtum with llm-compressor. Weights are unmodified apart from quantization; all model behavior and license obligations follow the base model.