ubergarm

/

Qwen3.5-35B-A3B-GGUF

Like 10

@vernaut

Sweeeet! Thanks for taking a look! Those are definitely usable numbers for local vibe coding!

I'll be honest, I've only benchmarked Vulkan/ROCm a couple times a while back when I had access to a 7900XTX 24GB like yours. iirc there are different models with different max power configurations as well, but I forget the equivalent of nvidia-smi to show that info.

There are also maybe two Vulkan backends to compile against e.g. (RADV NAVI31) and the other one? If no one else chimes in, I'll have to look for my notes tomorrow.

Are you compiling everything yourself, or is it all pre-built in the docker image?

If you are compiling it yourself, I have a branch of mainline llama.cpp with llama-sweep-bench that can show the speed across the entire context length to see how fast it drops off. Here is that branch: https://github.com/ubergarm/llama.cpp/tree/ug/port-sweep-bench

Anyway, it looks promising at first glance, thanks for some hard data that is encouraging!

Totally, very useable. I'm a bit new to this stuff, had sort of written off LLMs but got exposed to Roo Code at work (SWE) a few weeks ago and it really impressed me, so now I'm on a bit of an enthusiast kick haha, especially because I already had a suitable system for it.

rocm-smi is the equivalent. I've got one of these:

$ rocm-smi
======================================= ROCm System Management Interface =======================================
================================================= Concise Info =================================================
Device  Node  IDs              Temp    Power  Partitions          SCLK  MCLK   Fan  Perf  PwrCap  VRAM%  GPU%
              (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)
================================================================================================================
0       1     0x744c,   22753  43.0°C  9.0W   N/A, N/A, 0         0Mhz  96Mhz  0%   auto  350.0W  3%     0%
================================================================================================================
============================================= End of ROCm SMI Log ==============================================

Right now I'm letting it cap out at the default 350W but I can limit it lower with LACT if that's of any interest.

I'm using the pre-built container images from mainline llama.cpp, but I'll look into compiling and running your branch tomorrow 👍 I took a quick look through the build notes here but I don't see anything specific to RADV/NAVI31 in that Vulkan section. For now I'll plan to use the steps there but let me know if there's a better approach.

@vernaut

Wow you're doing great, even using LACT that is amazing! Keep up the good work! (In my most recent talk I cover LACT from the Linux/CUDA side: https://blog.aifoundry.org/p/adventures-in-model-quantization )

There is an official Vulkan benchmark thread and I see Occ4m (a vulkan dev for llama.cpp) active recently on there: https://github.com/ggml-org/llama.cpp/discussions/10879

I'll go benchmark this quant on my 3090TI FE 450W (tuned with LACT) now and see how it fares. I've heard this model is may be faster on your GPU! 🤞 I'll be happy and sad if that is true! lol

Okay, I am running ik_llama.cpp so a few arguments are different. On CUDA I can inflate the PP speed by increasing batch sizes, but I don't think that works on vulkan. (it does take more VRAM so can't run as much context). its all tradeoffs hah...

#!/usr/bin/env bash
model=/models/ubergarm/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q4_0.gguf
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 98816 \
  --n-predict 128 \
  -ger \
  --merge-qkv \
  -ngl 99 \
  --threads 1 \
  --warmup-batch

#!/usr/bin/env bash
model=/models/ubergarm/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q4_0.gguf
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 36864 \
  --n-predict 128 \
  -ger \
  --merge-qkv \
  -ngl 99 \
  -ub 2048 -b 2048 \
  --threads 1 \
  --warmup-batch

Right on, appreciate the guidance! I think I'm doing something wrong, I can't get llama-sweep-bench to complete...

Here's what I did, basically:

1. Cloned your repo
2. Set up a debian-testing distrobox, downloaded the vulkan libs mentioned here
3. Followed the build instructions from that same section
4. I found I had to remove a few of the flags from your example (as expected) and landed on:

./build/bin/llama-sweep-bench --model "/home/jackson/.cache/llama.cpp/Qwen3.5-35B-A3B-Q4_0.gguf" -c 36864 --n-predict 128 -ngl 99 --threads 1

It runs very briefly but then I hit an error:

sched_reserve: reserve took 122.83 ms, sched copies = 1

main: n_kv_max = 36864, n_batch = 2048, n_ubatch = 512, flash_attn_type = -1, n_gpu_layers = 99, n_threads = 1, n_threads_batch = 1

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|   512 |    128 |      0 |    0.269 |  1905.75 |    1.341 |    95.43 |
init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 639
 - the tokens for sequence 0 in the input batch have a starting position of Y = 512
 for M-RoPE, it is required that the position satisfies: X < Y
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
failed to decode the batch, n_batch = 2048, ret = -1
main: llama_decode() failed

I probably won't have time to look into this more until later this afternoon, but figured I'd post here in case this is something really obvious that you have a pointer for.

Also started watching your talk, very helpful so far, thank you for the link!

So... just a pointer that TG is a bit slower with ROCM, as usual

/home/llama.cpp/build-wmma/bin/llama-bench -fa 1 -m Qwen3.5-35B-A3B-Q4_0.gguf
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32

build: a96a1120b (8149)

I'll try with Vulkan in some time

It's a Sapphire Nitro 7900 XTX

> rocm-smi

======================================= ROCm System Management Interface =======================================
================================================= Concise Info =================================================
Device  Node  IDs              Temp    Power  Partitions          SCLK  MCLK   Fan  Perf  PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                               
================================================================================================================
0       1     0x744c,   30404  41.0°C  65.0W  N/A, N/A, 0         0Mhz  96Mhz  0%   auto  315.0W  1%     0%    
================================================================================================================
============================================= End of ROCm SMI Log ==============================================

Power cap can be raised to 400W with LACT but it doesn't increase inference performance

Mi50 32GB Vulkan comparisons between this and Q4_K_M. Prefer running RoCM but it is apparently broken for Qwen3/3.5 on these cards in llama.cpp atm.

I tried running it on one and two RX 7900 XTXs via ROCm 7.0 in mainline llama.cpp and was surprised that my prompt processing speed was much lower than everyone else's here, maybe because they're running at Gen 4 x8.

edit:
I then noticed that disabling flash attention gives me closer pp2048 to what the others got here with a regular Q8_0 quant and this Q4_0 quant:

$ llama-bench -m models/Qwen/Qwen3.5-35B-A3B-Q8_0.gguf -dev ROCm0/ROCm1 -fa 0,1 -b 2048 -ub 2048 -p 512,2048
ggml_cuda_init: found 3 ROCm devices:
Device 0: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 1: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 2: , gfx1036 (0x1036), VMM: no, Wave Size: 32

$ llama-bench -m models/Qwen/Qwen3.5-35B-A3B-Q4_0.gguf -dev ROCm1 -fa 0,1 -ub 2048 -b 2048 -p 512,2048
ggml_cuda_init: found 3 ROCm devices:
Device 0: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 1: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 2: , gfx1036 (0x1036), VMM: no, Wave Size: 32

@vineetk13 gen4 x8 shouldn't be a factor since it affects model load times, not inference performance itself (once the model has been loaded to VRAM)

Check if you compiled llama.cpp with rocWMMA, otherwise you might be facing the same issue as the poster here: https://github.com/ggml-org/llama.cpp/discussions/15021#discussioncomment-15825763

Thanks all for sharing some benchmarks! Interestingly it seems like vulkan performance is indeed pretty strong with this custom mix optimized for vulkan!

A llama.cpp vulkan dev, 0cc4m, did some testing too showing this quant doing quite well with Vulkan on a newer GPU:

@vernaut

Hey, sorry, you did everything correctly!! but llama-sweep-bench does not work on mainline just for these new qwen arches apparently, i forgot about that as ik mentioned it here how to recreate this kind of graph with llama-bench -d but I don't have a script for that worked out. https://github.com/ikawrakow/ik_llama.cpp/pull/1328

PS

For vulkan folks check out 0cc4m's great recent FOSDEM'26 Vulkan API in llama.cpp talk

@ubergarm ahh got it, appreciate the links! I'll try building+running ik_llama.cpp later today to see if that works. Saw your similar quant for Qwen3-Coder-Next as well, I might try that out and I will post some benches if I do... would also be down to test the Qwen3.5-27B dense model if you were to make a vulkan optimized quant for that one 😁 although I guess the jury's still out on how that model compares to this MoE in general.

I'm definitely curious about why/how this quant format manages to be so much faster, especially if there are aspects of that answer that can be ported to more space-friendly quants in the future. I'll check out 0cc4m's talk!

@vernaut

Super! A person on Beaver AI Discord made a ik_llama.cpp docker build, I haven't tried it yet and it may only have CUDA support: https://github.com/Steel-skull/ik_llama.cpp/pkgs/container/ik_llama.cpp otherwise hit me up if you run into any snags compiling and i'm not 100% how it will go for vulkan though last i checked a few months ago it worked fine with the mainline compatible quant types. (no vulkan kernels for the newer ik_llama.cpp quantization types unfortunately)

hah nice glad you found: https://huggingface.co/ubergarm/Qwen3-Coder-Next-GGUF?show_file_info=Qwen3-Coder-Next-Q4_0.gguf

I'll likely check the 27B dense, but these MoE optimized recipes don't work as well for dense models given all the weights are always active. still it is possible to use q8_0/q4_0/q4_1 and possibly iq4_nl which may be fastest for vulkan even on dense.

I'm definitely curious about why/how this quant format manages to be so much faster

So each quantization type has unique optimized matrix multiplication kernels depending on the backend used to compute it e.g. CPU, CUDA, Vulkan, etc. there has been more vulkan kernel developer time put into optimizing kernels for those quant types I chose to use.

@vineetk13 gen4 x8 shouldn't be a factor since it affects model load times, not inference performance itself (once the model has been loaded to VRAM)

Check if you compiled llama.cpp with rocWMMA, otherwise you might be facing the same issue as the poster here: https://github.com/ggml-org/llama.cpp/discussions/15021#discussioncomment-15825763

I hadn't tried disabling rocWMMA in the compilation in the previous message. With it disabled I do get a marginally higher pp2048 compared to before, which sounds counterintuitive. I also was under the impression that the ROCm backend had higher prompt processing while Vulkan had higher token generation, not Vulkan is better on both (at least for this model family I guess).

root@cfa1ba735865:/app# llama-bench -m models/Qwen/Qwen3.5-35B-A3B-Q4_0.gguf -dev ROCm1,Vulkan0 -fa 0,1 -ub 2048 -b 2048 -p 512,2048
ggml_cuda_init: found 3 ROCm devices:
Device 0: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 1: , gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 2: , gfx1036 (0x1036), VMM: no, Wave Size: 32
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = AMD Unknown (RADV GFX1100) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = AMD Unknown (RADV RAPHAEL_MENDOCINO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 2 = AMD Unknown (RADV GFX1100) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none

@vernaut

Fresh update on https://github.com/ubergarm/llama.cpp/tree/ug/port-sweep-bench now you can run llama-sweep-bench on mainline with the new Qwen models without getting that error.

I have some example commands, but for hybrid inference with 2x GPUs (cuda, honestly not sure if -sm graph works on vulkan, guessing not?) ik's CPU chunked delta net implementation is looking quite performant:

the ik_llama.cpp PR with more details and commands: https://github.com/ikawrakow/ik_llama.cpp/pull/1368#issuecomment-4008379564