steampunque/Qwen3.6-27B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.6-27B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.6-27B

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the layer quants are as follows:

Extended K (QN_E_H) mixed precision layer quant nomenclature:

QN_K_VOD, Q8_0_VOD
N = {2,3,4,5,6}

VOD = attnV/QKV:attnO:ffnD
V,O,D = {0,2,3,4,5,6,8,f,F}

VOD MAP:
2:Q2_K, 3:Q3_K, 4:Q4_K, 5:Q5_K, 6:Q6_K, 8:Q8_0, f:F:F16, 0:F32, default QN_K

   LAYER_TYPES='[
   [0 ,"Q5_K_658"],[1 ,"Q5_K_655"],[2 ,"Q4_K_546"],[3 ,"Q4_K_646"],[4 ,"Q4_K_546"],[5 ,"Q4_K_545"],[6 ,"Q3_K_335"],[7 ,"Q3_K_434"],
   [8 ,"Q3_K_334"],[9 ,"Q3_K_334"],[10,"Q3_K_334"],[11,"Q3_K_434"],[12,"Q3_K_334"],[13,"Q3_K_334"],[14,"Q3_K_334"],[15,"Q3_K_434"],
   [16,"Q3_K_335"],[17,"Q3_K_334"],[18,"Q3_K_335"],[19,"Q3_K_434"],[20,"Q3_K_335"],[21,"Q3_K_334"],[22,"Q3_K_335"],[23,"Q3_K_434"],
   [24,"Q3_K_335"],[25,"Q3_K_335"],[26,"Q3_K_335"],[27,"Q3_K_535"],[28,"Q4_K_544"],[29,"Q4_K_544"],[30,"Q4_K_544"],[31,"Q4_K_644"],
   [32,"Q4_K_544"],[33,"Q4_K_544"],[34,"Q4_K_544"],[35,"Q4_K_644"],[36,"Q4_K_545"],[37,"Q4_K_544"],[38,"Q4_K_545"],[39,"Q4_K_644"],
   [40,"Q4_K_545"],[41,"Q4_K_545"],[42,"Q4_K_545"],[43,"Q4_K_645"],[44,"Q4_K_546"],[45,"Q4_K_545"],[46,"Q4_K_546"],[47,"Q4_K_645"],
   [48,"Q4_K_546"],[49,"Q4_K_545"],[50,"Q4_K_546"],[51,"Q4_K_645"],[52,"Q4_K_546"],[53,"Q4_K_545"],[54,"Q4_K_546"],[55,"Q4_K_645"],
   [56,"Q4_K_546"],[57,"Q4_K_546"],[58,"Q4_K_546"],[59,"Q5_K_555"],[60,"Q5_K_655"],[61,"Q5_K_656"],[62,"Q5_K_658"],[63,"Q6_K_866"],
   ]'

   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"

This quant shows very strong reasoning, coding and vision performance across a curated set of test prompts. Very rarely it may fall into a short infinite rep loop in its think block.

A second Q4_E_H_MTP quant is available which adds nextn layer for MTP, supported by llama.cpp b9180 and above (versions less than b9180 will not load this quant, however at b9180 and above the model should load with MTP turned on or off) :

   LAYER_TYPES='[
   [64,"Q4_K_654"]
   ]'

This quant has the same layer 0..63 definitions as Q4_E_H with MTP layer quantized to Q4_K,VOD=654 while upsizing about 0.3G size for the MTP layer. If making use of this layer, it should be loaded fully into VRAM.

A larger Q6_K_H quant is also available. This quant was sized to maintain usable context with 24G VRAM with minimum quant of Q4_K_S across layers and strong Q6_K_L output layers. The layer definitions were taken directly from Qwen3.5-27B Q6_K_H, then tested across the set of eval prompts where it scored 100% accuracy.

Q4_K_L : Q4_K_M + attn_o = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
Q6_K_M : attn_v = q8_0 ffn_d = q8_0
Q6_K_L : attn_v = q8_0 attn_o = q8_0 ffn_d = q8_0

   LAYER_TYPES='[
   [0 ,"Q6_K_S"],[1 ,"Q5_K_M"],[2 ,"Q4_K_M"],[3 ,"Q4_K_S"],[4 ,"Q4_K_M"],[5 ,"Q4_K_S"],[6 ,"Q4_K_M"],[7 ,"Q4_K_S"],
   [8 ,"Q4_K_M"],[9 ,"Q4_K_S"],[10,"Q4_K_M"],[11,"Q4_K_S"],[12,"Q4_K_M"],[13,"Q4_K_S"],[14,"Q4_K_M"],[15,"Q4_K_S"],
   [16,"Q4_K_M"],[17,"Q4_K_M"],[18,"Q4_K_M"],[19,"Q4_K_M"],[20,"Q4_K_M"],[21,"Q4_K_M"],[22,"Q4_K_M"],[23,"Q4_K_M"],
   [24,"Q4_K_M"],[25,"Q4_K_M"],[26,"Q4_K_M"],[27,"Q4_K_M"],[28,"Q4_K_M"],[29,"Q4_K_M"],[30,"Q4_K_M"],[31,"Q4_K_M"],
   [32,"Q4_K_L"],[33,"Q4_K_M"],[34,"Q4_K_L"],[35,"Q4_K_M"],[36,"Q4_K_L"],[37,"Q4_K_M"],[38,"Q4_K_L"],[39,"Q4_K_M"],
   [40,"Q4_K_L"],[41,"Q4_K_L"],[42,"Q4_K_L"],[43,"Q4_K_L"],[44,"Q5_K_S"],[45,"Q5_K_S"],[46,"Q5_K_S"],[47,"Q5_K_S"],
   [48,"Q5_K_M"],[49,"Q5_K_M"],[50,"Q5_K_M"],[51,"Q5_K_M"],[52,"Q5_K_L"],[53,"Q5_K_L"],[54,"Q5_K_L"],[55,"Q5_K_L"],
   [56,"Q6_K_S"],[57,"Q6_K_S"],[58,"Q6_K_S"],[59,"Q6_K_S"],[60,"Q6_K_M"],[61,"Q6_K_M"],[62,"Q6_K_L"],[63,"Q6_K_L"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

This quant also shows very strong reasoning, coding and vision performance across a curated set of test prompts, and no infite rep loops were found across the full eval set tested.

Comparison:

Quant	size	PPL	Comment
IQ4_XS	14.8e9	10.6	IQ4_XS with default embedding and output
Q4_K_M	16.5e9	10.6	Q4_K_M with default embedding and output
Q4_E_H	15.8e9	10.1	Hybrid quant with Q4_K embedding Q6_K output
Q4_E_H_MTP	16.1e9	"	", including MTP layer at Q4_K_654
Q6_K	22.1e9	10.0	Q6_K with default embedding and output
Q6_K_H	18.6e9	10.4	Hybrid quant with Q6_K embedding Q6_K output

Usage:

Qwen3.6-27B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Due to the attention scheme used in the model, straightforward speculation approaches cannot be used. Later versions of llama.cpp as of 4/23/2026 attempt to circumvent this problem however efficiency will not be good so use of speculation prior to Qwen3.5 MTP patches is not recommended. As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by using the Q4_E_H_MTP quant.

On a 2x 4070 setup (1 RPC) approx performance is:

Q	QKV	NKV	gen tps
Q4_E_H	F16	100k +	24
Q4_E_H	Q8_0	180k +	24
Q6_K_H	F16	60k	22
Q6_K_H	Q8_0	100k	22

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it can fall into very heavy overthinking but does come up with accurate answers. Over a small set of eval prompts the model did extremely well. To avoid the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think block following the assistant prompt:

"<think>\nHere's a thinking process to solve the problem:"

The model was found to be highly capable on reasoning tasks when skipping think block, with zero overthinking, just accurate direct deductions to final solutions.

Note: preliminary testing shows this model is prone to severe overthinking on some prompts, significantly higher than Qwen 3.5-27B. This version of the model appears to have been fine tuned with stronger forced introspections which make it harder for it to escape the think block if there is even the slightest ambiguity in its reasoning. The Q6_K_H model was slightly better across the eval prompts at escaping the think block.

The Q4_E_H and Q6_K_H quants both went 2 for 2 on a couple tough bird ID images prompts and do very well summarizing a test text image compared.

The Q4_E_H quant was tested across a small set of code gen prompts and found to be very solid in its ability to generate working programs with think mode bypassed, it just directly cranked out working codeblocks with no preliminary reasoning or followup reasoning at all. The Q6_K_H quant does the same thing and appears to generate similar quality code. The quants were not tested in think mode with code gen.

If the model run over RPC it may crash due to a memory leak in RPC: https://github.com/ggml-org/llama.cpp/issues/19892, temp workaround set GGML_CUDA_DISABLE_GRAPHS=1 on rpc server launch. This problem might be addressed in latest upstream of llama.cpp as of 4/23/2026 with various changes made to contain the leak.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.6-27B.Q4_E_H.gguf	Q4_E_H	15.8e9 B	1G bigger than IQ4_XS
Qwen3.6-27B.Q4_E_H_MTP.gguf	Q4_E_H_MTP	16.1e9 B	same as Q4_E_H + MTP layer
Qwen3.6-27B.Q6_K_H.gguf	Q6_K_H	18.6e9 B	3.5G smaller than Q6_K
Qwen3.6-27B.mmproj.gguf	F16	0.93e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040