shujunyi/Qwen3.5-4B-MTP-Q4_0

Qwen3.5-4B — MTP — Q4_0 (GGUF)

Q4_0 GGUF quantization of Qwen3.5-4B that retains the Multi-Token Prediction (MTP) tensors, so it can self-speculate with llama.cpp's --spec-type draft-mtp — no separate draft model required.

Architecture	Qwen3.5 dense 4B
Quantization	Q4_0 (4.81 BPW)
File	Qwen3.5-4B-Q4_0.gguf (~2.6 GB, single file)
Speculative decoding	MTP (built in)
Source	requantized from unsloth/Qwen3.5-4B-MTP-GGUF → Qwen3.5-4B-Q8_0.gguf

Requirements

MTP is not yet in mainline llama.cpp. Build the branch that carries the draft-mtp speculative type:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git checkout mtp-exp     # branch with --spec-type draft-mtp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

Run

llama-server \
  -m Qwen3.5-4B-Q4_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -c 262144 -ngl 99

Download the weights:

pip install -U "huggingface_hub[hf_transfer]"
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download shujunyi/Qwen3.5-4B-MTP-Q4_0 \
  Qwen3.5-4B-Q4_0.gguf --local-dir .

Notes

• This quant was produced by requantizing the upstream Q8_0 weights to Q4_0 (llama-quantize --allow-requantize). Requantizing from an already-quantized 8-bit source is slightly lossier than quantizing directly from the BF16 master; if you need the cleanest Q4_0 baseline, quantize from Qwen3.5-4B-BF16.gguf instead.
• Q4_0 is a simple/legacy quant chosen for a clean quality baseline; for the best size/quality tradeoff consider an IQ4_XS / imatrix build instead.