unsloth

/

Qwen3.8-27B-GGUF

like 2.5k

Follow

Unsloth AI 32.3k

Q6 and 48k only - but it costs 5000Eur ? and before that cost was 10k. Amazing.

5090 with UD-Q4_K_XL, preliminary benches (my own kit), these aren't final, I am still testing

lama-server.exe ^
  -m Qwen3.8-27B-UD-Q4_K_XL.gguf ^
  --spec-type draft-mtp ^
  --alias qwen3.8-27b ^
  -ngl 99 -c 131072 --ctx-checkpoints 64 -np 1 -b 2048 -ub 2048 ^
  --load-mode none --flash-attn on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 ^
  --presence-penalty 0.0 --repeat-penalty 1.0 ^
  --chat-template-kwargs "{\"preserve_thinking\":true,\"reasoning_effort\":\"xhigh\"}"

with UD-Q5_K_XL I can only fit 128k context to RTX 5090.

getting speed:

prompt eval time = 0.98s /  2978 tokens (  3026.68 tokens per second)
       eval time =  418s / 49887 tokens (   119.18 tokens per second)
      total time =  419s / 52865 tokens
   graphs reused = 17949
draft acceptance = 0.59182 (31914 accepted / 53925 generated), mean len =  2.78

@SlavikF , Q8 for KV cache may help - to be honest I don't know the degradation with this option, when model Q5/Q4 with the BF16KV cache...

On RTX 5090 I'm using this config and I get ~100 t/s

  - -m
  - /unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q6_K_XL.gguf

  - --mmproj
  - /unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf

  - -ngl
  - "100"
  - -sm
  - layer
  - -fa
  - on
  - --ctx-size
  - "120000"
  - --threads
  - "16"
  - --threads-batch
  - "16"
  - --cache-type-k
  - q8_0
  - --cache-type-v
  - q8_0

 - -np
  - "1"

  - --image-min-tokens
  - "1024"

  - --jinja

  - --spec-type
  - draft-mtp
  - --spec-draft-n-max
  - "3"

  - --temp
  - "0.7"
  - --top-p
  - "0.80"
  - --top-k
  - "20"
  - --min-p
  - "0.0"

Well, you need to buy dual RDNA 4 cards. Check out my results. Results for qwen 3.8 27b are the same as qwen 3.6 27b.

I am running Q8_0 with 64k context.
Flash attention + cache K q8_0 + no cache V (bf16) = 77k context.
Flash attention + cache K q8_0 + cache V (q8_0) = 93k context.

5090 results: Q6_K_XL context ceiling much higher than reported here (no vision/MTP), plus a real MTP+vision bug worth knowing about

Been running Qwen3.8-27B-GGUF UD-Q6_K_XL on a 5090 today and wanted to share a few findings that either extend or diverge from what's already in this thread.

Context ceiling on Q6 is much higher than reported here, if you drop vision and MTP. @SlavikF mentioned ~48K for Q6_K_XL. Text-only, no MTP, --parallel 1: I bisected up from a working 65K test and landed on 208,896 tokens as a confirmed, real ceiling (262,144 failed on the main KV cache buffer; a few steps down from there kept failing on a small ~1.1-1.2GB "compute pp buffers" allocation before finally clearing at 208,896). So Q6 can go a lot further than 48K if vision/MTP aren't competing for the same VRAM. Might be worth checking whether the 48K number included those.

MTP + vision together triggers a real, known llama.cpp bug (ggml-org/llama.cpp#22867), but it's not actually fatal, at least not in my testing. You'll see find_slot: non-consecutive token position warnings fire repeatedly whenever vision embeddings are in play, MTP-enabled or not (I confirmed it happens even with MTP fully disabled, so it's a vision-processing artifact generally, not MTP-specific). Ran it three separate times with both features on and every single generation completed successfully despite the warning. Worth knowing if you see this and assume it's fatal, in my case it never was.

Watch out for a multi-slot OOM if you're running vision with the default n_slots. Left --parallel unset once (defaults to 4 slots sharing a kv_unified pool), and hit a real crash: a previously completed request's retained context (via --cache-prompt) was still occupying VRAM when a fresh slot needed room for a new image, and the vision encoder OOM'd by ~400MB. --parallel 1 fixed it outright.

reasoning_effort values, small correction for anyone copying configs from this thread: the officially documented values per Qwen's own model card are xhigh (default), medium, low. high isn't one of them, at least not in the official template, I initially set it by mistake and I'm not confident how an unrecognized value gets handled depending on your exact GGUF/template build. Switched to medium and, in a real side-by-side test against xhigh on a 4-part reasoning/code/physics prompt, got equally correct answers, one part even came out slightly more rigorous, for presumably less token spend. Small sample, but worth testing if you're trying to save reasoning tokens without an obvious quality hit.

Setup: llama.cpp server-cuda (prebuilt image), --cache-type-k/v q8_0 (confirmed the only safe KV quant on that image, q5_1 triggered a documented CPU-fallback bug, ggml-org/llama.cpp#24485, 33-45x slower prefill), --spec-draft-n-max 5 for MTP.

Comparison table (RTX 5090, UD-Q6_K_XL, llama.cpp server-cuda)

All three: --cache-type-k/v q8_0 (confirmed the only safe KV quant on the prebuilt server-cuda image — q5_1 triggers a documented CPU-fallback bug, ggml-org/llama.cpp#24485, measured 33-45x slower prefill), --parallel 1, --flash-attn on, temp 1.0 / top-p 0.95 / top-k 20 / min-p 0 / repeat-penalty 1.0.

Context ceilings above are real bisected values (boot succeeds at that number, fails one step higher), not round-number guesses.

Settings — Max context (no vision, no MTP)

llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
  --no-mmproj \
  --ctx-size 208896 \
  --n-gpu-layers 99 \
  --parallel 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cache-ram 16384 \
  --ctx-checkpoints 32 \
  --cache-prompt \
  --batch-size 1024 \
  --ubatch-size 512 \
  --flash-attn on \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --reasoning-preserve \
  --reasoning-budget 8192 \
  --reasoning-budget-message "Time's up, time to finalize the answer based on the reasoning so far." \
  --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}' \
  --jinja

262,144 (native ceiling) fails on the main KV cache buffer (~8.5GB short). 229,376 and 212,992 both fail on a much smaller, roughly-fixed ~1.1-1.2GB "compute pp buffers" allocation, not the KV cache. 208,896 is the confirmed highest working value.

Settings — Vision + MTP

llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
  --image-min-tokens 1024 \
  --spec-type draft-mtp \
  --spec-draft-n-max 5 \
  --ctx-size 65536 \
  --n-gpu-layers 99 \
  --parallel 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cache-ram 16384 \
  --ctx-checkpoints 32 \
  --cache-prompt \
  --batch-size 1024 \
  --ubatch-size 512 \
  --flash-attn on \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --reasoning-preserve \
  --reasoning-budget 8192 \
  --reasoning-budget-message "Time's up, time to finalize the answer based on the reasoning so far." \
  --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}' \
  --jinja

--parallel 1 is load-bearing here, not optional: with the default 4 slots sharing one kv_unified pool, a fresh slot needing room for a new image can OOM against another slot's still-retained context from a finished request (confirmed via a real crash, ~400MB short). Also: find_slot: non-consecutive token position warnings fire repeatedly whenever vision is active — confirmed this is a vision-processing artifact in general, not MTP-specific (it still appears with MTP fully disabled). Never actually failed a generation across three separate tests despite the warning, matches ggml-org/llama.cpp#22867.

Settings — MTP, no vision

llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
  --no-mmproj \
  --spec-type draft-mtp \
  --spec-draft-n-max 5 \
  --ctx-size 131072 \
  --n-gpu-layers 99 \
  --parallel 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cache-ram 16384 \
  --ctx-checkpoints 32 \
  --cache-prompt \
  --batch-size 1024 \
  --ubatch-size 512 \
  --flash-attn on \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --reasoning-preserve \
  --reasoning-budget 8192 \
  --reasoning-budget-message "Time's up, time to finalize the answer based on the reasoning so far." \
  --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}' \
  --jinja

131,072 was specifically the context that failed once with vision also in the picture (MTP's draft context came up short competing with vision's encoder for the same freed room). With vision removed, that exact failure mode doesn't apply, and 131,072 booted clean with zero find_slot warnings, further confirming that warning is vision-specific. Not yet re-bisected upward — real headroom likely exists above 131,072 given how comfortably it fit.

--spec-draft-n-max: table above reflects the confirmed n_max=5 run. A separate n-max sweep posted elsewhere in this thread (UD-Q4_K_XL, so not a direct match) showed peak throughput at n_max=2 with a monotonic decline past that point, n_max=6 actually slower than no speculative decoding at all on prose. Re-testing at n_max=2 on this Q6 quant now based on that finding; no confirmed numbers for it yet.

Looks like there is an option for the vision layer (mmproj) to use RAM & CPU, not VRAM & GPU:

--no-mmproj-offload

My system:

• GPU: RTX 5090
• CPU: AMD Ryzen 9 9950X3D 16-Core Processor
• RAM: 2 channels * 32GB DDR5 6400 MHz

Using this query with image:

write the text from the attached image

using UD-Q5_K_XL quant
using --spec-type draft-mtp
using no KV quantization (default)

Case 1: everything in VRAM:

• I can fit 125k context
• VRAM usage: 31.21GB
• time to complete the task: 13.7 s / 6723 tokens
  • prompt eval time = 1.9s (2589 t/s)
• only 1 CPU core used

Case 2: use --no-mmproj-offload

• I can fit 144k context
• VRAM usage: 31.22GB
• time to complete the task: 76.7 s / 6636 tokens
  • prompt eval time = 66s (75 t/s)
• all 16 CPU cores used 100%

Conclusions:

• if your goal is to maximize the context you can trade image processing speed (~5x slower with 16 CPU cores, ~10x slower with 8 CPU cores) for additional 19k context.
• the text PP, TG speed is not affected. Only affected speed is PP for images.

On my RTX 5090, I did the similar test with MTP:
comparing --spec-type draft-mtp with --spec-type none
using UD-Q5_K_XL quant.
using no KV quantization (default)
using --no-mmproj-offload

With MTP I can fit 144k context
Without MTP I can fit 190k context

Conclusions:

• the "cost" of MTP is 46k context
• when using MTP, PP is ~10% slower (3200 vs 2900 t/s)
• when using MTP, TG is ~100% faster (66 vs 121 t/s)