guru87/Qwen3.6-35B-A3B-MTP

Qwen3.6-35B-A3B MTP head (BF16, single file)

The multi-token-prediction (MTP) draft head from the official Qwen/Qwen3.6-35B-A3B MoE checkpoint, packed into a single 1.6 GB safetensors file.

This repo is only useful as a drafter for a quantized Qwen3.6-35B-A3B checkpoint that is missing its MTP weights (e.g. a PARO build that declares mtp_num_hidden_layers: 1 in its config but ships zero mtp.* tensors). With the MTP head re-attached via the sharded-symlinks layout (see usage below), vLLM speculative decoding works out of the box.

Same recipe we already used for:

• guru87/Qwen3.6-27B-MTP
• guru87/Qwen3.5-9B-MTP
• guru87/Qwen3.5-4B-MTP
• guru87/Qwen3.5-2B-MTP
• guru87/Qwen3.5-0.8B-MTP

Contents

File	Purpose
mtp.safetensors	19 BF16 tensors, 1611 MiB, lifted verbatim from the base
SHA256SUMS	Per-tensor full SHA256 for audit / authenticity check
extract_from_qwen_base.py	Reproducible extraction script (the file was made with this)

Provenance

Extracted from Qwen/Qwen3.6-35B-A3B/model-00025-of-00026.safetensors and model-00026-of-00026.safetensors (the two shards that contain the mtp.* weight-map entries) on a local copy of the public Qwen base model.

To verify authenticity yourself:

python extract_from_qwen_base.py --src /path/to/Qwen3.6-35B-A3B \
                                 --out /tmp/mtp.safetensors --sums /tmp/SHA256SUMS
diff /tmp/SHA256SUMS SHA256SUMS    # must be byte-identical

Cross-format verification

We additionally verified by SHA256 that the BF16-passthrough MTP tensors (layernorms, fc projection, MoE gate, shared-expert gate) are bit-identical to those shipped inside the public AWQ-4bit quant:

Source	matches?
Qwen/Qwen3.6-35B-A3B (BF16, this repo's source)	10/10
Qwen3.6-35B-A3B-AWQ-4bit (BF16-kept tensors)	10/10

The expert and self-attention projections in AWQ are stored in packed-INT4 layout (per-expert weight_packed + weight_scale + weight_shape), so they can't bit-match the stacked BF16 expert tensors here — but the non-quantized tensors that stay BF16 in AWQ are all bit-equal. This shows the MTP head here is canonical Qwen output, identical to what's already redistributed inside community quants.

Tensor manifest

mtp.fc.weight                                         shape=[2048, 4096]            bf16
mtp.norm.weight                                       shape=[2048]                  bf16
mtp.pre_fc_norm_embedding.weight                      shape=[2048]                  bf16
mtp.pre_fc_norm_hidden.weight                         shape=[2048]                  bf16
mtp.layers.0.input_layernorm.weight                   shape=[2048]                  bf16
mtp.layers.0.post_attention_layernorm.weight          shape=[2048]                  bf16
mtp.layers.0.self_attn.q_norm.weight                  shape=[256]                   bf16
mtp.layers.0.self_attn.k_norm.weight                  shape=[256]                   bf16
mtp.layers.0.self_attn.q_proj.weight                  shape=[8192, 2048]            bf16
mtp.layers.0.self_attn.k_proj.weight                  shape=[512, 2048]             bf16
mtp.layers.0.self_attn.v_proj.weight                  shape=[512, 2048]             bf16
mtp.layers.0.self_attn.o_proj.weight                  shape=[2048, 4096]            bf16
mtp.layers.0.mlp.gate.weight                          shape=[256, 2048]             bf16
mtp.layers.0.mlp.shared_expert_gate.weight            shape=[1, 2048]               bf16
mtp.layers.0.mlp.shared_expert.gate_proj.weight       shape=[512, 2048]             bf16
mtp.layers.0.mlp.shared_expert.up_proj.weight         shape=[512, 2048]             bf16
mtp.layers.0.mlp.shared_expert.down_proj.weight       shape=[2048, 512]             bf16
mtp.layers.0.mlp.experts.gate_up_proj                 shape=[256, 1024, 2048]       bf16
mtp.layers.0.mlp.experts.down_proj                    shape=[256, 2048, 512]        bf16

(256 MoE experts; gate_up_proj fuses gate+up into one [256, 2*hidden, dim] tensor.)

Usage — inject into a quant that's missing MTP

Same pattern as the other Qwen MTP repos. Make a sharded-checkpoint directory where every shard from the target quant is symlinked, plus mtp.safetensors is added as the last shard, and model.safetensors.index.json is rewritten to include the MTP keys:

from paroquant.cli.inject_mtp import inject_mtp

inject_mtp(
    target_dir="/path/to/Qwen3.6-35B-A3B-PARO",        # quant missing MTP
    mtp_file="mtp.safetensors",                         # this file
    out_dir="/path/to/Qwen3.6-35B-A3B-PARO-MTP",        # merged result
)

(or use the CLI: paroquant-inject-mtp --target ... --mtp ... --out ...).

Then serve with --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' under vLLM 0.19.1+.

License

Apache 2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B. These weights are bit-identical to what Qwen redistributes; we're just packaging them as a single file for easier re-injection into community quants.