raulvidis

/

Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF

like 5

Hi raulvidis,

Thanks for the bailingmoe3 work — the model loads and runs well on my Strix Halo box, and MTP is confirmed active. But I'm measuring decode well below the model-card numbers and would like to understand the gap.

My setup

HW: AMD Strix Halo, Radeon 8060S (gfx1151), 128 GB
Build: charlie12345/ROCmFPX main (b41ce12) + PRs #47, #48, #56, #57 cherry-picked → version 187 (137b40d). Built with build-strix-rocmfp4-mtp.sh (HIP+Vulkan, gfx1151), FORCE_MMQ=ON (default)
Command: -dev ROCm0 -ngl 999 -c 204800 -np 1 -b 4096 -ub 2048 -fa on -ctk q8_0 -ctv q8_0 --no-mmap -fit off --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0.5 --temp 0.6 --top-p 0.95 --top-k 20
Measured vs model card

test measured card
prefill pp512 598 t/s 460–610
prefill ~10k tokens 478 t/s ~250 t/s @64k
decode plain (llama-bench tg128) 32 t/s 37
decode MTP (acceptance 92–100%) ~30 t/s 45–50
What I verified

MTP is active: logs show creating MTP draft context against the target model, adding speculative implementation 'draft-mtp', acceptance 92–100%, mean acc length 2.5.
But MTP gives no net speedup here (30 t/s with vs ~32 t/s without) — the draft-generation cost seems to offset the accepted tokens.
Tried without effect: memlock 8 MB → unlimited, GGML_HIP_FORCE_MMQ ON→OFF, removing --top-k (llama-bench with no sampling is already ~30 t/s).
Questions

What exact build/commit and flags produced the 45–50 t/s (MTP) and 37 t/s (plain)? Was that on the PR branch, or a dev build with decode tweaks not in #57?
Is ~30 t/s decode expected for bailingmoe3 on the released build, or is there a decode optimization (kernel/backend) I'm missing?
Does -np 4 -c 1048576 change per-slot decode, or only total concurrency?
Happy to share full logs if useful. Thanks!

Hi Yohan!
Thanks for the detailed report, and glad the arch + MTP work runs on your box. The model-card numbers are reproducible; let me give you the exact setup and the three deltas I see vs yours.
Exact build behind the card numbers: charlie12345/ROCmFPX @ c190e435 (main as of the moment of testing) with only the #56 + #57 commits on top — not #47/#48, and not current main. HIP+Vulkan, gfx1151, FORCE_MMQ on, NO_VMM on.
Exact serve command (45–50 t/s MTP / 37 plain):
llama-server -m Ling-3.0-flash-ROCmFP4-STRIX-MTP-clamp.gguf
-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0.5
--temp 0.6 --top-p 0.95 --top-k 20

Note what’s absent: no -ctk/-ctv — f16 KV cache. llama-bench for the 40.6 tg128: llama-bench -m -p 512,2048 -n 128 -r 3 -fa 1 -mmp 0.
Suspects for your gap, in order:

1. -ctk q8_0 -ctv q8_0 - this is my top suspect. Ling’s KV is MLA-compressed and tiny (a full 1M context costs ~8 GB at f16), so q8 KV buys you almost nothing here while forcing slower dequant paths in FA on HIP. It also degrades the MTP draft context the same way, which would explain “92–100% acceptance but no net speedup”. The draft pass is paying the same slow-KV tax. Please retest decode with f16 KV; I’d expect most of the gap to close.
2. Base commit - your prefill being faster than mine (598 vs 461) while decode is slower suggests #47/#48 or newer main changed kernel paths. If f16 KV doesn’t fully close the gap, try c190e435 + #56 + #57 only, matching the card.
3. -ub 2048 -b 4096 shouldn’t matter for decode, but for apples-to-apples the card used defaults.
   Your -np/-c question: with unified KV, -np 4 -c 1048576 gives 4×262144 slots; per-slot decode is unchanged (single-stream ≈ same t/s), concurrency raises aggregate throughput (I measured ~54 t/s across two simultaneous streams).

Also for context: 45–50 is the short-context number; my own curve drops to ~43 @16k and ~37 @64k fill (acceptance actually rises with depth). If you’re measuring at 200k context, ~35–40 with MTP would already be on-curve.
If f16 KV + the pinned base doesn’t get you to ~37 plain / ~45 MTP at short context, send the full logs and your amdgpu.gttsize / memory config and I’ll dig in.

Thank you for your Answer, i've got better results with BF16 for Kv cache and when using DIO, (i lost a little decode speed 38 Tk/s but get better PP)

A good release here, thank you for your work, Ling is really better for me than laguna (laguna has better performance, but he is more dumb)