kingjones777

/

Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF

like 12

Not a stupid question at all — you're on the right fork already, which is the good news. Here's exactly what I did.

Base: https://github.com/charlie12345/ROCmFPX.git at commit d3ca537 ("Merge pull request #59 from charlie12345/fix/spec-replay-livelock"). Same fork Kyuz0's ROCmFPX toolbox is built from.

Donor: upstream ggml-org/llama.cpp, branch pr27742, at 035e227 — "llama: hold the qwen4exp indexer cache in a new llama_memory_hybrid_idx". That commit message is the hint to the thing that trips most people up.

Why naive patching fails: qwen4exp isn't just a new arch file. It needs a whole new memory class for the Qwen Sparse Attention indexer. If you only copy the obvious arch bits you'll get a tree that compiles but can't load. The merge touched 26 files; the ones people miss are the new ones:

src/llama-memory-hybrid-idx.cpp <-- new, the QSA indexer cache
src/llama-memory-hybrid-idx.h <-- new
src/models/qwen4exp.cpp <-- new
conversion/qwen4exp.py <-- new
src/CMakeLists.txt <-- MUST add the new .cpp files or they never build

Plus modifications to llama-arch.{cpp,h}, llama-batch.{cpp,h}, llama-hparams.{cpp,h}, llama-kv-cache.{cpp,h}, llama-model{,-loader,-saver}.{cpp,h}, llama-quant.cpp, models/models.h, gguf-py/gguf/{constants,gguf_writer,tensor_mapping}.py, conversion/init.py, and tests/test-llama-archs.cpp.

I did it as a real merge of pr27742 into d3ca537 and resolved the conflicts by hand — 26 files' worth. Cherry-picking individual files will bite you because the batch/kv-cache changes are interdependent.

Build:

-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON

If you only want to run my GGUFs, that's all you need. Two things if you go further and convert your own:

The entry point is the conversion/ package, not convert_hf_to_gguf.py — the generic converter rejects Qwen4ExpForConditionalGeneration outright.
The 51B-parameter PLE table will OOM a naive conversion. Mine died twice building a 205 GiB F32 intermediate before I made it cast each shard straight to BF16 and write positionally with per-shard flush. On a 128 GB box you cannot materialise that table.

And a runtime tip that'll save you an afternoon: don't use --no-mmap. The PLE table is a sparse lookup that streams from SSD through the page cache — with mmap only ~63 GiB of the 98 GiB file is ever GPU-resident. Force it into anonymous memory and the cgroup OOM-killer takes the process with nothing in the server log.

Hope you enjoy it — and if you get it running, I'd love to know what pp/gen you see at long context.

Thank you for taking the time to write this up. I've spent the day trying to build llama.cpp correctly, and will keep at it. If I can get it up and running, I'll certainly share my stats.

Let me know if I can help further

Ok, up and running. At 100k context (out of 256k), I am getting ~400 t/s prompt processing and an avg of 19.2t/s. This is a fair bit quicker than my experience with Unsloth's Flash Next Q4_K_XL. To get up and running, I started with Kyuz0's toolbox and built your llama.cpp from https://github.com/kingjones30/ROCmFPX within it. This is my launch command:

toolbox run --container llama-rocm-7.14-rocmfpx sh -c "export HIP_VISIBLE_DEVICES=0; export HSA_ENABLE_SDMA=1; export ROCR_VISIBLE_DEVICES=0; export HSA_OVERRIDE_GFX_VERSION=11.5.1; export LLAMA_CHAT_TEMPLATE_KWARGS='{"reasoning_effort":"medium"}'; home/USERNAME/Documents/AI/ROCmFPX/build/bin/llama-server -m " home/USERNAME/ai-models/Qwen3.8/Flash-Next-ROCmFP4/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" --port 8080 --ctx-size 262144 -ngl 999 --fit off -fa on --host 0.0.0.0 --temp 1.0 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --jinja --verbosity 4"