Qwen3.6-35B-A3B-int4-AutoRound

This is a W4A16 (4-bit weight, 16-bit activation) quantization of Qwen/Qwen3.6-35B-A3B using Intel AutoRound v0.12.2.

Qwen3.6-35B-A3B is a Mixture-of-Experts model with 35B total parameters and ~3B active parameters per token (256 experts, 8 active). It features a hybrid architecture with Gated Delta Net (GDN) linear attention layers alternating with full attention, plus a Multi-Token Prediction (MTP) head for speculative decoding.

Quantization Details

Parameter	Value
Method	Intel AutoRound (SignRound)
AutoRound version	0.12.2
Scheme	W4A16 (4-bit symmetric integer weights, BF16 activations)
Group size	128
Symmetric	Yes
Packing format	auto_round:auto_gptq
Calibration iterations	400
Calibration samples	256
Sequence length	2048
Batch size	8
Learning rate	auto
Calibration dataset	AutoRound default
Hardware	NVIDIA DGX Spark (GB10, Blackwell SM_121, 128 GB unified memory)
Quantization time	~10.5 hours
Peak RAM	28.19 GB
Peak VRAM	25.11 GB

Layers kept in BF16

Certain layers are excluded from quantization to preserve model quality and ensure compatibility with tensor-parallel inference:

Layer pattern	Reason
shared_expert (gate_proj, up_proj, down_proj)	Intel's recommended MoE recipe — shared experts are critical for output quality
shared_expert_gate	Shape not divisible by 32 (skipped automatically)
mtp.* (entire MTP block)	Required for vLLM speculative decoding — AutoRound's calibration never activates MTP, so these would receive RTN-only rounding with near-zero acceptance rate
linear_attn.in_proj_a / in_proj_b	GDN layers with out_features=32; vLLM fuses these into in_proj_ba (dim 64), and Marlin requires >=64 per TP shard — quantizing breaks TP>=2
mlp.gate (MoE router)	Standard practice; routing decisions are sensitive to quantization noise
model.visual.*	Vision encoder preserved in full precision
lm_head	Tied embeddings, kept in BF16

The quantization_config.json includes both checkpoint-level (in_proj_a, in_proj_b) and vLLM fused-module (in_proj_ba) FP override entries so that vLLM correctly identifies these layers as unquantized under tensor parallelism.

Quantization summary

• Quantized: 30,850 / 31,181 linear layers
• Block-wise loss (representative): layer 0 loss 0.000011→0.000001, layer 39 loss 0.002693→0.001283

Model size

Format	Size
BF16 (original)	~70 GB
INT4 (this model)	~21 GB

How to use

With transformers + gptqmodel

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "nchapman/Qwen3.6-35B-A3B-int4-AutoRound",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("nchapman/Qwen3.6-35B-A3B-int4-AutoRound")

messages = [{"role": "user", "content": "Explain mixture-of-experts architectures briefly."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Note: gptqmodel must be installed for the W4 dequantization kernels. Without it, transformers will silently produce garbage output.

With vLLM

vllm serve nchapman/Qwen3.6-35B-A3B-int4-AutoRound \
    --dtype bfloat16 \
    --quantization auto_round \
    --num-speculative-tokens 1 \
    --speculative-model [draft_model]

The MTP head is preserved in BF16, enabling vLLM's speculative decoding with the native MTP draft. Tensor parallelism (TP>=2) works correctly thanks to the in_proj_ba FP overrides.

Reproducing this quantization

Using AutoRound's Python API:

from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-35B-A3B",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-35B-A3B")

autoround = AutoRound(
    model,
    tokenizer,
    bits=4,
    group_size=128,
    sym=True,
    iters=400,
    nsamples=256,
    seqlen=2048,
    batch_size=8,
    dataset="NeelNanda/pile-10k",
    ignore_layers=[
        "mtp.fc",
        "linear_attn.in_proj_a",
        "linear_attn.in_proj_b",
        "shared_expert",
    ],
)
autoround.quantize()
autoround.save_quantized("Qwen3.6-35B-A3B-int4-AutoRound", format="auto_round")

The ignore_layers patterns keep in_proj_a and in_proj_b in BF16. AutoRound's saved quantization_config.json will contain the checkpoint-level overrides; the vLLM fused-module entries (*.linear_attn.in_proj_ba) need to be added as a post-save step — our quantization script does this automatically, but if using the raw API you'll need to add {"bits": 16, "data_type": "fp"} entries for each *.linear_attn.in_proj_ba to extra_config in both quantization_config.json and config.json for TP>=2 compatibility.

Acknowledgments

• Quantization method: Intel AutoRound — SignRound: optimization-based weight rounding for post-training quantization
• MTP preservation approach informed by Lorbus/Qwen3.6-27B-int4-AutoRound
• Shared-expert BF16 recipe from Intel/Qwen3.5-122B-A10B-int4-AutoRound
• GDN TP fix validated against vLLM's Marlin kernel constraints