Qwen3.5-4B-SWQ

This is a quantized version of Qwen/Qwen3.5-4B with SWQ (Surogate Whole-model Quantization), a statistically lossless quantization method. The weights are 64% smaller on disk than the BF16 original and take 48% less VRAM to serve, and across six benchmarks spanning coding, mathematics, science and tool calling no test separates it from that original.

Against the same model in FP8 — the usual production quantization — it is 40% smaller on disk, uses 15% less VRAM, and is faster on single requests. It is also markedly more dependable on long-form code generation: on LiveCodeBench, FP8 drops 14 points below the original while this model stays within noise of it.

Against a GGUF Q5 quantization of comparable size, it reproduces the original's next-token choices more often and serves about 6x more throughput under concurrent load.

SWQ builds on GPTQ-style Hessian-compensated rounding and the statistically-lossless framing of arXiv:2605.02404, adding whole-model rate accounting, the hierarchical codec, and the Marlin tensor-core serving path.

Every weight matrix in the model is quantized:

• transformer layers (attention, Gated DeltaNet, MLPs) at 5 bits
• token embedding and LM head at 6 bits
• vision encoder at 8 bits

Language model total: 5.66 bits per weight. The vision encoder adds 345 MB and is quoted separately, outside that rate.

Quality

Statistically lossless refers to measured behaviour, not bit-exactness: across six benchmarks no test separates the quantized model from the original, and it reproduces the original's own word-by-word choices as closely as the measurement can detect.

Benchmark	Problems	BF16	FP8	Qwen3.5-4B-SWQ
HumanEval+ (pass@1)	164	75.6	75.0	79.3
MBPP+ (pass@1)	378	84.9	81.5	86.0
MATH-500	500	94.4	93.0	93.2
LiveCodeBench v6 (pass@1)	182	51.6	37.4	49.5
GPQA Diamond	197	72.6	73.1	70.1
BFCL-v3 (tool calling)	1,000	86.0	87.9	84.1

Against the BF16 original, no benchmark separates them. Scored problem by problem the two agree on the overwhelming majority — 478 of 500 on MATH-500, 160 of 182 on LiveCodeBench — and no difference survives correction for testing six benchmarks. Tool calling is the closest: 86.0 against 84.1, which is nominally significant before correction and worth knowing if your workload is agentic.

Against FP8 the picture splits. This model is far more reliable on long-form code generation: on LiveCodeBench, FP8 fails 33 problems the original solves and recovers only 7, a 14-point drop, while this model stays within noise of the original. It also leads FP8 on both execution- scored coding benchmarks. FP8 is the better choice for tool calling, where it beats this model by 3.8 points.

All columns are our own measurements, one greedy sample per problem, so they run below Qwen's published figures, which average several samples. Compare the columns to each other, not to numbers published elsewhere.

Against other methods

GGUF Q5_K_S is the closest alternative at this size. The GGUF has no vision encoder — llama.cpp ships that separately — so this compares text-only against text-only:

	GGUF Q5_K_S	Qwen3.5-4B-SWQ
Bits per weight	5.73	5.66
Matches the original's next word	92.3%	93.7%
MBPP+ vs the original	−4.50	+1.1
Serving 32 requests at once	653 tok/s	3,562 tok/s
Weights in VRAM (text-only)	2.8 GiB	4.2 GiB

It stays closer to the original — both in the words it picks and in what that costs on a coding benchmark — and it serves batched traffic 5.5x faster, because it runs on vLLM's tensor-core kernels rather than llama.cpp.

Those kernels want the weights in a wider layout than either format stores, so serving costs more memory than the 3.1 GiB file: 4.2 GiB text-only, 4.5 GiB with the vision encoder. llama.cpp runs its stored format directly and fits in 2.8 GiB, keeping the token embedding on the CPU. If VRAM is the binding constraint and you serve one request at a time, Q5_K_S is the better fit; for concurrent traffic this model does far more with the memory.

Against FP8, the usual production quantization of this model. Both include the vision encoder, both measured on the same GPU:

	Qwen3.5-4B FP8	Qwen3.5-4B-SWQ
Weights on disk	5.14 GiB	3.09 GiB
Weights in VRAM	5.26 GiB	4.49 GiB
Single request	177 tok/s	195 tok/s
32 requests at once	3,798 tok/s	3,562 tok/s

Smaller to download, smaller to serve, and faster for a single request; FP8 keeps a 6% edge on heavily batched traffic, where its native tensor-core path has no dequantization to do at all.

AWQ is the other option in vLLM, but on this architecture it leaves attention and the embeddings in 16-bit, which nearly doubles the size to 10.28 bits per weight.

Serving

Requires vLLM with the Surogate package installed; its plugin registers the weight format automatically:

pip install vllm
pip install git+https://github.com/invergent-ai/surogate
vllm serve surogate/Qwen3.5-4B-SWQ

Text and images go through the standard OpenAI API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
out = client.chat.completions.create(
    model="surogate/Qwen3.5-4B-SWQ",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
)
print(out.choices[0].message.content)

Weights convert to Marlin tensor-core kernels at load; serialized bytes never change. Measured on one RTX 5090 (SM120) at 128 tokens in and out: 195 tok/s single-request, 3,562 tok/s at 32 concurrent requests, using 4.49 GiB of weight memory. Text-only deployments can pass --language-model-only to skip the vision tower, which brings the weights to 4.17 GiB. For tool calling use --tool-call-parser qwen3_coder.

Prompting, thinking-mode switches, and agentic usage are unchanged from the base model — see the Qwen3.5-4B card.

Quantization details

SWQ stores weights on a two-level grid. Each matrix is cut into 256-weight superblocks and each superblock into eight 32-weight sub-blocks; every sub-block gets its own scale and zero point, but rather than storing those as full-precision numbers they are kept as 6-bit indices into two FP16 constants held once per superblock. Storing a scale and zero directly for every 32 weights would cost a full bit per weight; the two-level scheme buys the same granularity for half that, and the finer the scaling, the less each individual weight has to be distorted to fit 5 bits. The values themselves are not simply rounded: each matrix is fitted against activation statistics gathered from calibration text, using GPTQ-style error compensation on the hierarchical grid followed by a weighted least-squares refit of the sub-block parameters, so error introduced in one column is partly corrected in the ones that follow. Every matrix goes through this, the token embedding and LM head included — how those two are quantized turns out to matter more for output quality than anything done to the transformer layers, and methods that leave them in 16 bits end up far larger than their advertised width. The quoted bit rate counts every serialized byte, and at load time the weights are repacked into Marlin tensor-core kernels for serving without changing what is stored on disk.

• Language model. Uniform 5-bit projections, 6-bit embedding/LM head, calibrated on 512 text sequences.
• Vision encoder. INT8, group-128.
• Not included. The MTP speculative-decoding heads.
• Validated on a single GPU. The hierarchical format does not currently support tensor parallelism.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. These weights are a derived work of the original BF16 release; all credit for the base model to the Qwen team.

Quantized with Surogate

This model was quantized with Surogate, an open-source LLM training toolkit. SWQ is its quantization method — the recipe, the evaluation harness, and the vLLM plugin that serves this format all live in the repository, so these results can be reproduced and the method applied to other models.