MTP?
Is MTP included in the model gguf?
Yes, I tried it. Enable it the same way as 3.6
It's already there
It's already there
Listen, could you please tell me how you make bnb 4bit quants? I still haven't found out. But overall, yes, I would like to see Qwen3.8-27B bnb 4bit from you :)
It's just that constantly asking for it wouldn't be very cool – it'll be easier to just do it myself, and it'll be easier for you too. 🤗
The model has become slower; on my setup—an RTX 3060 Ti (16GB) + RTX 4070 Ti Super (16GB)—I used to get around 45–50 tokens per second with Qwen 3.6 27B (including MTP), but now I only get 30 using the same settings.
MTP works for me:
~/llama.cpp/build/bin/llama-server \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
--host 0.0.0.0 --port 8080 \
--fit off --gpu-layers all --gpu-layers-draft all \
--ctx-size 48000 --kv-unified \
--top-p 0.95 --top-k 20 --temp 1.0 --min-p 0.00 --repeat-penalty 1.0 \
--spec-type draft-mtp
getting this in the log:
prompt eval time = 0.15s / 88 tokens ( 572.42 tokens per second)
eval time = 286s / 30247 tokens (, 105.61 tokens per second)
total time = 286s / 30335 tokens
graphs reused = 11498
draft acceptance = 0.54556 (18776 accepted / 34416 generated), mean len = 2.64
I also noticed that the model's quality has dropped; Qwen 3.6 would write a bug-free Minesweeper game in a single HTML file from just one prompt, whereas 3.8 takes about twenty times longer to think and constantly produces bugs.
Im trying to get mtp working but i got the no speedup 18t/s. Could someone point out what I´m doing wrong? Using the Qwen3.8-27B-UD-Q3_K_XL.gguf.
It crashes on startup unless i remove this line: "--spec-draft-n-max 3 ^"
@echo off
start llama-server ^
-m models/Qwen3.8-27B-UD-Q3_K_XL.gguf ^
-fitt 1024 ^
-c 18072 ^
-n 32768 ^
-fa on ^
-np 1 ^
-ctk q8_0 ^
-ctv q8_0 ^
-ctkd q8_0 ^
-ctvd q8_0 ^
-ctxcp 64 ^
--no-mmap ^
--mlock ^
--no-warmup ^
--spec-type mtp ^
--spec-draft-n-max 3 ^
--chat-template-kwargs "{"preserve_thinking": true}" ^
--temp 0.9 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0
It is --spec-type draft-mtp
Im trying to get mtp working but i got the no speedup 18t/s. Could someone point out what I´m doing wrong? Using the Qwen3.8-27B-UD-Q3_K_XL.gguf.
It crashes on startup unless i remove this line: "--spec-draft-n-max 3 ^"
@echo off
start llama-server ^
-m models/Qwen3.8-27B-UD-Q3_K_XL.gguf ^
-fitt 1024 ^
-c 18072 ^
-n 32768 ^
-fa on ^
-np 1 ^
-ctk q8_0 ^
-ctv q8_0 ^
-ctkd q8_0 ^
-ctvd q8_0 ^
-ctxcp 64 ^
--no-mmap ^
--mlock ^
--no-warmup ^
--spec-type mtp ^
--spec-draft-n-max 3 ^
--chat-template-kwargs "{"preserve_thinking": true}" ^
--temp 0.9 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0
Missing the spec-type draft-mtp
I'm getting ~57 tokens/sec using an RTX Pro 4000 Blackwell.
My settings are
~/llama.cpp/build/bin/llama-server
--host 127.0.0.1
--port 8080
--alias Qwen3.8-27B
--model /mnt/ai/models/Qwen3.8-27B/Qwen3.8-27B-Q4_K_M.gguf
--jinja
--metrics
--spec-type draft-mtp
--spec-draft-n-max 3
--n-gpu-layers auto
--n-predict 32768
--ctx-size 131584
--batch-size 2048
--ubatch-size 512
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn auto
--kv-unified
--fit on
--load-mode mlock
--parallel 1
--threads 12
--threads-batch 12
--min-p 0.0
--temperature 0.7
--top-k 20
--top-p 0.95
--repeat-penalty 1.0
--presence-penalty 1.5
Metric | Value | Rate
Prompt eval | 2625 tokens / 3068.30 ms | 855.52 tok/s (1.17 ms/tok)
Generation | 1264 tokens / 22191.39 ms | 56.91 tok/s (17.57 ms/tok)
Total | 3889 tokens / 25259.69 ms | —
CUDA graphs reused | 709 | —
Spec decoding acceptance | 0.8274 (901/1089), mean len 3.48 | —
Yeah guys, meanwhile I found it - but on my 4070 the Q2_K_XL is the only running model and MTP is indeed included, but does not make any sense. Low quants with MTP make the model slower, not faster (would say from Q2 dowawards) - so, it does not make any sense to include it in the base model, a seperate mtp file would be beneficial as people using low quants dont do this because they love low quants - they need to use them to press them into their GPUs :) But apart from this - its by far the best model on Hermes Agent locally. Really, it`s impressive, even in Q2. If we could get rid of the forced MTP ballast, that would be amazing <3
None of these is working, I always get 40 tok/s and no spec accepting rate line in the console. Using latest version, cuda 13
people here are seeing mtp slow llama.cpp down. i had the opposite on memra, 140 tok/s spec on one pro 6000 (plain is 69).
https://github.com/avifenesh/memra
https://inference.tiyuvta.ai/app
Its not about slowing down, MTP is not working at all. It shows no acceptance metrics in the log in the statistics at all, neither does it on Dflash
I use hermes and deepseek-v4-flash to read the first 20MB context of this 27B model, it told me there is no mtp layer at all. Instead I use this MTP-only mold: https://huggingface.co/a4lg/Qwen3.8-27B-MTP-ONLY-GGUF and get 80t/s in RTX4090_48GB VRam card. Qwen3.8-27B-UD-Q6_K_XL.gguf
I use hermes and deepseek-v4-flash to read the first 20MB context of this 27B model, it told me there is no mtp layer at all. Instead I use this MTP-only mold: https://huggingface.co/a4lg/Qwen3.8-27B-MTP-ONLY-GGUF and get 80t/s in RTX4090_48GB VRam card. Qwen3.8-27B-UD-Q6_K_XL.gguf
The unsloth variants have MTP embedded in them. You just need to pass the right parameters
--spec-type draft-mtp
--spec-draft-n-max 3