kingjones777

/

Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF

like 17

I can try it again I was getting oom on a few attempts I tried

Yes, what you suggested above, streaming the n-grams from SSD, is the kind of multi-layer computing design I had in mind. If you can somehow make this practical, it would be great for running this as a coding model on Strix. Let me know if you need any help testing it. I’m ready to help on my Strix. Thanks for the models as always ✌️

Appreciate that, and yes — I'll take you up on it.

The n-gram table is the whole story on this model. It's ~51B of the parameters, sitting at Q5_1 in my builds as per_layer_token_embd plus the ple.* tensors — call it ~35 GiB of the file. It's also what made this build fight me at every stage: it needed a ~205 GiB f32 temp during conversion, OOM'd llama-quantize twice until I patched it to chunk, and it's why my Q6 tiers came out at 158 GiB and couldn't fully offload on a 128 GiB Strix.

Good news is you don't have to wait on me to try the idea. It's a sparse lookup — only the bigrams/trigrams actually in your prompt get touched — so it's a much better streaming candidate than dense weights, and the runtime already has the knob:

-ot "per_layer_token_embd|ple.=CPU" -ngl 999

That keeps the n-gram table in host RAM and puts everything else on the GPU. On my box that should take STRIX_LEAN from ~98.5 GiB resident down to ~63, which frees a lot of room for context.

What would help most, if you're willing:

Run it with and without that -ot flag and tell me what happens to tok/s — my worry is the lookup becomes a latency bottleneck once it's off-GPU.
Prompt processing. I haven't published a pp number because I haven't measured one I trust. Use a different long prompt each run — the prompt cache will silently reduce a repeated prompt to prompt_n=4 and hand you a garbage number. Check prompt_n actually reads what you sent.
Long-context behaviour. QSA has a 512-block / 2048-token budget, so I'd expect it to hold up better than most at depth, but nobody's shown that yet.

One thing to flag: these need my merged tree, not stock llama.cpp — PR #27742 grafted into ROCmFPX. Stock builds won't load the files at all.

Thanks for the offer, genuinely. ✌️

I tested it — turns out the streaming already happens, just not the way either of us assumed.

The n-gram table is ~51B of the parameters (per_layer_token_embd + the ple.* tensors, Q5_1 in my builds). It's a sparse lookup: only the bigrams/trigrams actually in your prompt ever get touched. So with mmap, the pages fault in on demand and the rest never leaves disk. Measured on STRIX_LEAN:

63.3 GiB resident on GPU out of a 98.5 GiB file. No flags. That's mmap doing exactly the multi-layer thing you described.

I also forced it explicitly with -ot "per_layer_token_embd|ple.=CPU" to see if there was more to win. Identical 63.3 GiB — nothing left to save — and generation got worse, one run dropping from ~23 to 13.4 tok/s. The lookup is latency-sensitive, so pushing it further out hurts. I'd expect SSD streaming to be worse still, for the same reason.

Numbers on my Ryzen AI MAX+ 395 / gfx1151 / ROCm 7.2.4, full 49/49 offload:

prompt processing: 283–334 tok/s (~3,000-token prompts, warm, different prompt each run)
generation: 22.8–24.1 tok/s

So for coding on Strix it's already practical — pp is in the same range as DeepSeek V4 Flash on this hardware, and you've got ~65 GiB of GTT left over for context.

Two traps that cost me runs, so you don't repeat them:

Don't use --no-mmap. It pulls the whole ~98 GiB into anonymous memory and the cgroup OOM-killer takes it with no error in the log — it just stops at "loading model". I only found it in dmesg.
Use a different prompt every run. Repeat one and llama.cpp's prompt cache reduces it to prompt_n=4 and hands you a garbage pp number. Check prompt_n matches what you actually sent.

Where I'd genuinely like the help: long context. QSA has a 512-block / 2048-token budget and nobody has characterised how it behaves at depth. If you can push it to 32k+ on your Strix and post pp/gen — and whether GTT resident stays near 63 GiB or climbs — that's the number none of us have. Real coding sessions live at that depth, so it's the thing that decides whether this is actually a daily driver.

These need my merged tree (PR #27742 grafted into ROCmFPX) — stock llama.cpp won't load them at all.

Good research! Thank you!

I also found the repo from DGXSpark owners, they also faced the same issue, but they found how to fix it with 2 small patches to llama.cpp codebase.

https://github.com/0xBakeer/qwen38-flash-next-spark/tree/main

I'll probably try it today. Receipe is quite simple:

1. checkout https://github.com/ggml-org/llama.cpp/pull/27742
2. apply patches from https://github.com/0xBakeer/qwen38-flash-next-spark/tree/main over it
3. build for StrixHalo ROCm:

cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

4. Use all required params to start it

Thanks for the research and tests, I'll try this out on my system and check if the perfomance is the same for me.

I also have another idea I’m working on. It’s somewhat similar to MTP, but instead of predicting future output tokens, I’m thinking of using a small model on the ~50 TOPS NPU of Strix Halo to predict future N-gram accesses.

Basically, an MTP-like predictor for memory access. It could look at the model’s current access pattern and predict which N-gram entries it will need next, then proactively prefetch those from SSD into RAM.

That way, instead of inference requesting an N-gram from SSD and waiting for the response, the NPU would continuously predict and stage the likely next data ahead of time. The goal is to turn the SSD → RAM path from a reactive bottleneck into a predictive pipeline.

It’s still a hypothesis and I’m experimenting with how predictable the access pattern actually is, but I think it could be an interesting direction for Flash-Next on Strix Halo.

I think I'll give that a try. Let me know what your findings are. And thank you for trusting my build.