hampsonw

/

Qwen3.6-27B-AWQ-BF16-INT4-mtp-bf16

Like 20

My system fetch:
OS: Arch Linux x86_64
Kernel: Linux 6.19.10-arch1-1
Uptime: 32 days, 9 hours, 24 mins
Packages: 382 (pacman)
Shell: zsh 5.9
Terminal: /dev/pts/2 10.2p1
CPU: Intel(R) Xeon(R) E5-2680 v4 (28) @ 3.30 GHz
GPU 1: NVIDIA GeForce RTX 3090 [Discrete]
GPU 2: NVIDIA GeForce RTX 3090 [Discrete]
Memory: 11.72 GiB / 62.63 GiB (19%)
Swap: 53.63 MiB / 32.00 GiB (0%)
Disk (/): 103.95 GiB / 205.50 GiB (51%) - btrfs

nvidia-smi result (summary):
NVIDIA-SMI 595.58.03 Driver Version: 595.58.03 CUDA Version: 13.2
390W / 420W | 21873MiB / 24576MiB
410W / 420W | 21879MiB / 24576MiB

Первая GPU в PCIE3 X16, вторая в X8, NVLINK отсутствует.

Настройки docker vllm:
--tensor-parallel-size 2
--dtype auto
--kv-cache-dtype fp8
--max-model-len 210000
--max-num-seqs 4
--max-num-batched-tokens 8192
--gpu-memory-utilization 0.90
--enable-chunked-prefill
--enable-prefix-caching
--load-format safetensors
--trust-remote-code
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder

Раньше я запускал gemma4 31B в квантовании UD_Q4_K_XL от unsloth, на двух 3090, через движок llama-cpp в docker и получал результат около 15-25 t/s.
Сейчас я запускаю через vllm модель с huggingface - hampsonw/Qwen3.6-27B-AWQ-BF16-INT4-mtp-bf16, которая весит даже чуть больше, и получаю такие результаты:

MTP k=1: 39-40 t/s
MTP k=2: 42-47 t/s
MTP k=3: 43-52 t/s
При этом БЕЗ MTP я получаю 50-52 t/s вообще всегда, стабильно, а с MTP - то вверх, то вниз на ~25% от указанных диапазонов.

Так же без MTP при 3 параллельных запросах скорость достигает 160-170 t/s, вместо 150 ожидаемых, почему?

Что это за магия, почему такая огромная скорость, еще и без MTP? Я чего-то не догоняю, или почему?

with nvlink?

I personally run 4x3090's so i have similar hardware. i get 65 t/s across most depths and output sizes with mtp off. i used this for benchmarking prompt processing and token generation speed https://github.com/eugr/llama-benchy .

I see two things i should point out:

1. You said you tried gemma4 31B gguf and got 15-25 t/s. that's a different model and gguf is always slower than vllm in my experience
2. 3090's don't get the same kernel support as newer hardware and so using mtp can force the whole model onto a slower kernel and your results with mtp will be slower, sometimes more than 50% slower than no mtp.

This model was created in one of my attempts to actually get mtp to perform faster on my hardware than no mtp by increasing the acceptance rates through higher quality mtp layers instead of using quantized mtp layers but on my hardware, with current ampere kernel support, i use mtp off because it's faster. i suspect this model will work better for people with blackwell but still using a quantized model.

I personally run 4x3090's so i have similar hardware. i get 65 t/s across most depths and output sizes with mtp off. i used this for benchmarking prompt processing and token generation speed https://github.com/eugr/llama-benchy .

I see two things i should point out:

1. You said you tried gemma4 31B gguf and got 15-25 t/s. that's a different model and gguf is always slower than vllm in my experience
2. 3090's don't get the same kernel support as newer hardware and so using mtp can force the whole model onto a slower kernel and your results with mtp will be slower, sometimes more than 50% slower than no mtp.

This model was created in one of my attempts to actually get mtp to perform faster on my hardware than no mtp by increasing the acceptance rates through higher quality mtp layers instead of using quantized mtp layers but on my hardware, with current ampere kernel support, i use mtp off because it's faster. i suspect this model will work better for people with blackwell but still using a quantized model.

Thx a lot, your model is really very helpful in agent programming, but I turned off MTP and it’s much more comfortable this way. I’d like to ask about your setup: it just so happens that I’m running your 4-bit version on two GPUs, and I have about 12 GB free on each of the two video cards. I feel comfortable in this range—the cache is large and the speed is acceptable—but when the cache gets large, the speed drops significantly. Is this because the cache spills over from the first GPU to the second, or is it just due to the volume? What do you think?

I suspect it's your quantized kvcache. It has a compute burden that is exponential as context length increases. I run without quantizing the kv cache and see only a minor slowdown in decode at large context depths. Like 10% or so.

How large are you talking about? My personal benchmarking has only gone out to 64000 tokens or so for decode speeds.