steampunque/Qwen3.8-27B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.8-27B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.8-27B

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD"],
   [0 ,"Q5_K_6558"],[1 ,"Q5_K_6555"],[2 ,"Q4_K_5456"],[3 ,"Q4_K_6456"],
   [4 ,"Q4_K_5456"],[5 ,"Q4_K_5455"],[6 ,"Q3_K_3345"],[7 ,"Q4_K_4444"],
   [8 ,"Q3_K_3344"],[9 ,"Q3_K_3344"],[10,"Q3_K_3344"],[11,"Q4_K_4444"],
   [12,"Q3_K_3344"],[13,"Q3_K_3344"],[14,"Q3_K_3344"],[15,"Q4_K_4444"],
   [16,"Q3_K_3345"],[17,"Q3_K_3344"],[18,"Q3_K_3345"],[19,"Q4_K_4444"],
   [20,"Q3_K_3345"],[21,"Q3_K_3344"],[22,"Q3_K_3345"],[23,"Q4_K_4444"],
   [24,"Q3_K_3345"],[25,"Q3_K_3345"],[26,"Q3_K_3345"],[27,"Q4_K_5444"],
   [28,"Q4_K_5444"],[29,"Q4_K_5444"],[30,"Q4_K_5444"],[31,"Q4_K_5444"],
   [32,"Q4_K_5454"],[33,"Q4_K_5454"],[34,"Q4_K_5454"],[35,"Q4_K_5554"],
   [36,"Q4_K_5455"],[37,"Q4_K_5454"],[38,"Q4_K_5455"],[39,"Q4_K_6554"],
   [40,"Q4_K_5455"],[41,"Q4_K_5455"],[42,"Q4_K_5455"],[43,"Q4_K_6555"],
   [44,"Q4_K_5456"],[45,"Q4_K_5455"],[46,"Q4_K_5456"],[47,"Q4_K_6555"],
   [48,"Q4_K_5466"],[49,"Q4_K_5465"],[50,"Q4_K_5466"],[51,"Q4_K_6565"],
   [52,"Q4_K_5466"],[53,"Q4_K_5465"],[54,"Q4_K_5466"],[55,"Q4_K_6565"],
   [56,"Q4_K_5466"],[57,"Q4_K_5466"],[58,"Q4_K_5466"],[59,"Q5_K_6565"],
   [60,"Q5_K_6565"],[61,"Q5_K_6566"],[62,"Q5_K_6568"],[63,"Q6_K_8666"],
   [64,"Q4_K_6554"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"

The quant was optimized for approximately Q4_K_M bpw with strong performance across a curated set of reasoning test prompts, scoring close to 100% across the test set, exhibiting good common sense, and not overthinking or infinite rep loops on any of the prompts. The quant includes layer 64 nextn MTP layer. If not using MTP the loader will give warning messages about unused tensors on layer 64 but the model will run correctly.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	16.8e9	7.2	Q4_K_M with default embedding and output
Q4_E_H	16.5e9	7.9	Hybrid quant with Q4_K embedding Q6_K output

of note is a significant decrease in PPL compared to Qwen 3.5/3.6 series, which may imply this version of model was trained heavier on english language processing or just stronger pre training on wikitext so it memorizes it better.

Usage:

Qwen3.8-27B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Due to the attention scheme used in the model, straightforward speculation approaches cannot be used. Later versions of llama.cpp as of 4/23/2026 attempt to circumvent this problem however efficiency will not be good so use of speculation prior to Qwen3.5 MTP patches is not recommended. As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by turning on the MTP functionality if using upstream llama.cpp. MTP functionality was not evaluated.

On a 2x 4070 setup (1 RPC) approx performance with no MTP is:

Q	QKV	NKV	gen tps
Q4_E_H	F16	90k +	24
Q4_E_H	Q8_0	140k +	23

The model appears to be trained to decide itself whether to do a think block or not. If it thinks a prompt is easy it just prefixes its response with an empty think block. When it decided to do a think block it did not overthink or fall into rep loops (common RL model problem) on any of the test prompts. To for the model to skip thinking think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a THINK_START bootstrap following the assistant prompt:

"<think>\n"

On one IQ test prompt the model decided to wing it without a think block and it failed, but by forcing a think block it came up with the right answer.

The model has common sense on the recent dumb washcar prompt in both think and forced nothink mode:

washcar.txt:

I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

lm washcar.txt 
<think>
The user is asking whether they should walk or drive to a car wash that is 50 meters away. Let me think about this practically.

The key point here is: they want to wash their car. The car wash is 50 meters away.

If they walk, they leave the car behind. Then they'd need to go back and drive it to the car wash. That's an extra trip.

If they drive, they take the car directly to the car wash. It's only 50 meters, so it's a very short drive.

The obvious answer is: drive. You need to get the car TO the car wash. Walking there without the car defeats the purpose. You'd have to walk back, get in the car, and drive it there anyway.

This is a pretty straightforward practical question. Let me give a clear, helpful answer.
</think>

**Drive.**

The whole point is to get the car *to* the car wash. If you walk the 50 meters, you'll arrive without your car, then have to walk back, get in, and drive it there anyway. You'd just be adding an unnecessary trip.

Fifty meters is a very short drive — just a few seconds — so there's no real downside to driving it over.

NOTHINK=1 lm washcar.txt 
You should **drive**.

Here’s why:
- The purpose of going to the car wash is to **wash the car**.
- If you walk, you’ll arrive at the car wash without your car, so you can’t wash it.
- Driving ensures your car is present at the car wash, allowing you to get it cleaned.

So, even though 50 meters is a short distance, driving is the logical choice in this context.

VISION:

The model went 0 for 2 on a couple tough bird ID images prompts but did very well summarizing a test text image. Qwen 3.5 and Qwen3.6 both go 2 for 2 on the bird ID images.

CODE:

The quant was tested across a small set of code gen prompts and found to be very solid in its ability to generate working programs with think mode enabled. Code gen was not evaluated with think block forced off.

LONG CONTEXT:

The model passes the long context Qwen3 runescape test prompt running Q8_0 KV: https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt with just a little overthinking:

lm Qwen3_Runescape_Massive_Prompt.txt
<think>
The user is asking what level you are at when you have half the XP of max level (level 99) in RuneScape.

From the Experience Table in the provided text:
- Level 99 requires 13,034,431 XP
- Half of that would be 13,034,431 / 2 = 6,517,215.5 XP

Now I need to find which level corresponds to approximately 6,517,215.5 XP.

Looking at the table:
- Level 92: 6,517,253 XP
- Level 91: 5,902,831 XP

So 6,517,215.5 is very close to level 92 (6,517,253). It's actually just slightly below level 92.

Wait, let me re-read the question. "What level are you at when you get to half the xp of max level?"

Half of max level XP (level 99) = 13,034,431 / 2 = 6,517,215.5

Looking at the table:
- Level 92: 6,517,253 XP

So 6,517,215.5 is just barely below level 92 (which requires 6,517,253). This means you'd be at level 91 (which requires 5,902,831) and very close to reaching level 92.

Actually, let me reconsider. The question asks "what level are you at when you get to half the xp of max level?" This means at what level do you have accumulated half the total XP needed for level 99.

Half of 13,034,431 = 6,517,215.5

Level 92 requires 6,517,253 XP to reach.
Level 91 requires 5,902,831 XP to reach.

So 6,517,215.5 is between level 91 and level 92, but extremely close to level 92 (only 37.5 XP short of level 92).

So the answer would be level 91 (you haven't quite reached level 92 yet), or you could say you're essentially at level 92 since you're so close.

Actually, I think the most accurate answer is that you're at level 91, very close to level 92. But since the question is asking "what level are you at," and you haven't quite reached level 92 yet (you need 6,517,253 but only have 6,517,215.5), you would be at level 91.

Hmm, but this is so close to level 92 that it's essentially level 92. Let me just say level 92 since it's within 37.5 XP of that threshold, or more precisely, you're at level 91 and just barely short of level 92.

I'll go with level 92 as the practical answer since you're essentially there (within 37.5 XP), but I'll note it's technically just barely below level 92.
</think>

Based on the Experience Table from the book, level 99 (max level) requires **13,034,431 XP**. Half of that is **6,517,215.5 XP**.

Looking at the table:
- Level 91 requires 5,902,831 XP
- Level 92 requires 6,517,253 XP

So at half the XP of max level, you would be at **level 91**, just barely short of reaching level 92 (only 37.5 XP away from it).

Prompt processing slows from about 660 tps at start of prompt to 280 tps at end of ~108k token prompt.

The model passes a 75k token needle in haystack test also.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.8-27B.Q4_E_H.gguf	Q4_E_H	16.8e9 B	0.3G smaller than Q4_K_M, includes MTP layer
Qwen3.8-27B.mmproj.gguf	F16	0.93e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040