mgoin/Qwen3.6-35B-A3B-2Bit-GSQ-ct

Qwen3.6-35B-A3B — 2-bit GSQ

2-bit quantization of Qwen/Qwen3.6-35B-A3B (MoE, 35B total / 3B active) produced with GSQ (Gumbel-Softmax Quantization). The model is compressed from BF16 down to ≈2.13 bpp while preserving most of the base model's reasoning, coding, and long-context behaviour.

• Paper: GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv:2604.18556)
• Paper page on HF: https://huggingface.co/papers/2604.18556
• Code: https://github.com/IST-DASLab/GSQ
• Collection: https://huggingface.co/collections/ISTA-DASLab/gsq

Evaluation Results

We evaluate the quantized checkpoint against the original Qwen/Qwen3.6-35B-A3B across reasoning, instruction-following, science QA, and math benchmarks.

Benchmark	Base Model	2-bit GSQ
AIME 2025	93.33	86.67
MATH-500	85.2	85.8
GPQA Diamond	83.84	76.77
IFEval	91.25	88.01
MMLU-Pro	84.87	81.37
GSM8K	96.21	93.63

Quantization details

• Base model: Qwen/Qwen3.6-35B-A3B
• Bits / weight (effective): ≈2.13 bpp
• Codebook: 2-bit symmetric scalar {-2, -1, 0, +1} × scale
• Group size: 128
• Format: Humming (quant_method: "humming", b_dtype: "uint2")
• Pipeline: GPTQ initialization → Gumbel-Softmax refinement (Lion optimizer)
• What's quantized: routed-expert MLPs (gate_proj, up_proj, down_proj).
• Kept in BF16: attention (self_attn, linear_attn), embeddings, layernorms, LM head, MoE routing gate, and the shared experts.

Storage layout (why the HF UI shows I32 + BF16)

The Hugging Face "Tensor types" widget reports the container dtype of each safetensor on disk, not the effective precision of the underlying weights. This checkpoint uses the Humming on-disk layout (exact-width packing — no sub-byte values are padded into a wider container). For every quantized expert-MLP Linear with original weight shape [out_features, in_features], the following tensors are stored:

Tensor	Dtype	Shape on disk	Meaning
<layer>.weight	I32	[out_features, in_features × 2 / 32] = [out_features, in_features / 16]	2-bit values bit-packed along the input dim, LSB-first: 16 weights per INT32 word.
<layer>.weight_scale	BF16	[out_features, in_features / 128]	One symmetric scale per group of group_size = 128 weights along the input dim.
Attention / norms / embed / LM-head / MoE gate / shared experts	BF16	unchanged	Not quantized; copied from the base checkpoint.

So although the UI says "I32 + BF16", the effective storage per quantized weight is:

2 bits (packed) + 16 bits / 128 (group scale) ≈ 2.13 bpp

The quantization_config block in config.json is:

{
  "quant_method": "humming",
  "b_dtype": "uint2",
  "weight_scale_group_size": 128,
  "weight_scale_type": "group",
  "has_zero_point": false,
  "ignore": [
    "lm_head",
    "re:.*self_attn.*",
    "re:.*linear_attn.*",
    "re:.*visual.*",
    "re:mtp.*",
    "re:.*mlp\\.gate$",
    "re:.*mlp\\.shared_expert_gate$",
    "re:.*shared_expert.*"
  ]
}

Loading this checkpoint requires vLLM plus the humming MoE kernels (pip install humming-kernels). See Serving with vLLM below.

Note: GSQ training first writes shards in compressed-tensors pack-quantized format (where the 2-bit codebook is padded into a 4-bit INT32 container). The published checkpoint here has been re-packed via convert_to_humming.py into exact-width 2-bit Humming storage, hence the 2 / 32 shape factor on weight.

Serving with vLLM

Temporary vLLM compatibility note: the upstream vLLM Humming MoE implementation currently has a bug that prevents this checkpoint from being served correctly. Until the fix is available upstream, use one of the following workarounds:

1. Build vLLM from our fork:

   git clone https://github.com/adotdad/vllm.git
   cd vllm
   pip install -e .
   

2. Or patch your existing vLLM installation by replacing:

   vllm/model_executor/layers/quantization/humming.py
   

   with the version from our fork:

   https://github.com/adotdad/vllm
   

Install the Humming kernels (required for vLLM to load this checkpoint):

pip install humming-kernels

Ampere (sm ≥ 80) or Hopper GPUs are required for serving.

vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ

Text-only VRAM note: the full checkpoint is ≈14.4 GB, of which about ≈2.6 GB comes from the MTP and vision components. These components are optional for ordinary text-only generation: the vision weights are only needed for multimodal inputs, and the MTP weights are only useful when the serving backend enables MTP/speculative decoding. If you only use the text-generation path without MTP, the model weights require roughly ≈11.8 GB of VRAM, excluding KV-cache and runtime overhead.

Citation

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026},
  url    = {https://arxiv.org/abs/2604.18556}
}