raulvidis/Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF

Ling-3.0-flash — ROCmFP4-STRIX + MTP (Strix Halo optimized)

A Q4_0_ROCMFP4_STRIX quantization of inclusionAI/Ling-3.0-flash (124B MoE, 5.1B active, hybrid KDA+MLA bailingmoe3), tuned for AMD Strix Halo (gfx1151, Radeon 8060S) — with two extras you won't find in stock conversions:

1. The MTP (NextN) head is preserved — surgically restored from the original safetensors at Q8_0 (standard converters drop it), enabling draft-mtp speculative decoding: 90–95% draft acceptance, ~45–50 t/s decode on a 128 GB Strix Halo box (vs ~37 plain).
2. SwiGLU clamp metadata baked in (bailingmoe3.swiglu_clamp_exp/shexp) — Ling is trained with clamped SwiGLU in late layers (vLLM implements it; the public HF modeling code ignores it). Without the clamps, GGUF inference deterministically corrupts occasional tokens (count += 1eville). With them: HumanEval fenced 95.1 / plus 89.0.

Requirements

Runs on the ROCmFPX llama.cpp fork (ROCmFP4 kernels + bailingmoe3 + MTP support):

• bailingmoe3 arch + MTP wiring: https://github.com/charlie12345/ROCmFPX/pull/57
• MTP-on-checkpoint spec fix: https://github.com/charlie12345/ROCmFPX/pull/56
• (converter-side clamp fix upstream: https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant/pull/67)

Stock llama.cpp cannot load ROCmFP4 tensors. For non-ROCm setups use a standard quant (e.g. AtomicChat/Ling-3.0-flash-GGUF) on the feat/bailingmoe3 fork instead.

Serving (measured-optimal on Strix Halo, 128 GB)

ulimit -l unlimited   # LXC default 8 MB memlock cripples GPU registration
llama-server -m Ling-3.0-flash-ROCmFP4-STRIX-MTP-00001-of-00002.gguf \
  -dev ROCm0 -ngl 999 -fa on -c 1048576 -fit off -np 4 --no-mmap \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0.5 \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --temp 0.6 --top-p 0.95 --top-k 20 --jinja

Notes: --no-mmap matters (mmap against a near-full GTT loads at MB/s; no-mmap loads 65 GB in ~17 s); -fit off (the auto-fitter aborts on this size); 4×262k slots = 1M unified context fits in ~75 GiB GTT; thinking toggles per request via chat_template_kwargs.

Measured performance (Strix Halo / Radeon 8060S, ROCm)

metric	value
decode, MTP spec (n-max 2, 95% acceptance)	45–50 t/s
decode, plain	37 t/s
prefill	460–610 t/s (short), ~250 t/s @64k
decode @64k context	36.6 t/s (spec), acceptance rises to 99% at depth
load time (no-mmap, warm)	~17 s
HumanEval fenced / plus (pass@1)	95.1 / 89.0

Sampling per model card: temp 0.6, top-p 0.95, top-k 20.