kingjones777/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF

Qwen3.8-Flash-Next — ROCmFP4 FAST GGUF — AMD Ryzen AI Max+ 395 / Strix Halo / gfx1151

⚠️ Read this before comparing any number here to a discrete GPU

Every measurement on this card is from an AMD Ryzen AI MAX+ 395 "Strix Halo" — an integrated GPU with unified memory. There is no discrete VRAM. The Radeon 8060S addresses ordinary system RAM through the GTT aperture, and the box's 128 GB is shared between CPU and GPU.

That is the entire reason a 98 GiB model runs here at all: the GPU can reach system memory, so capacity is enormous. The trade is bandwidth — roughly 215 GB/s measured, against ~1 TB/s on a high-end discrete card. So expect big-model capacity that a 24 GB dGPU cannot touch, and per-token speed well below one.

This is also why the numbers below are reported as GTT resident, not "VRAM used" — on this hardware those are the same pool, and nvidia-smi-style VRAM intuitions do not transfer. FAST is the smallest of the three and the one to take if you are disk-constrained: attention, experts, token embeddings and the PLE table are all ROCmFP4, with only the output head lifted to Q6_K. Quantized directly from my own BF16 conversion of the release weights. general.file_type = 103 (Q4_0_ROCMFP4_FAST). 4.27 bpw, 87.94 GiB.

Read straight out of the GGUF headers of the files in this repo:

tensor group	type
MoE expert weights (ffn_*_exps, 144)	TYPE_101 (ROCmFP4, 4.251 bpw)
shared expert (ffn_*_shexp, 144)	TYPE_101
attention (attn_*, 120)	all TYPE_101
per_layer_token_embd.weight (PLE, 51.2B params)	TYPE_101 — 25.34 GiB
token_embd.weight	TYPE_101
output.weight (lm head)	Q6_K
norms / biases	F32

The Q6_K head

output.weight is Q6_K in every tier, never 4-bit. An unprotected head ruins a 4-bit build: every token you sample passes through the lm head, so its quantization error lands directly in the argmax. On a sparse-MoE model the head is also one of the few dense matrices left, which makes its error stand out more. It is 0.3% of the master weights — pinning it to Q6_K costs under half a GiB and removes that whole error class.

⚠ Patched llama.cpp required

These files need a patched llama.cpp — PR #27742 merged into the ROCmFPX fork. Stock builds will not load them: the qwen4exp architecture support and the Q4_0_ROCMFP4_* tensor types both live in that fork. Build for gfx1151:

-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON

TYPE_100 / TYPE_101 in the tables are the ROCmFP4 tensor types — they have no number in stock ggml, which is precisely why stock builds reject these files.

Where the file actually lives — measured, not inferred

177B parameters, of which 51.2B are per_layer_token_embd — the PLE n-gram lookup table. That table is the only real difference between these three tiers, and it behaves nothing like the rest of the weights.

Solving the three published files for the two ROCmFP4 tensor types (three files, two unknowns, exact fit — residual 0.000 GiB on all three) gives TYPE_101 = 4.251 bpw and TYPE_100 = 4.506 bpw. With those, the mass splits:

tier	file	PLE table	everything else
FAST	87.94 GiB	25.34 GiB	62.60 GiB
STRIX_LEAN	98.49 GiB	35.76 GiB	62.73 GiB
STRIX	113.47 GiB	50.66 GiB	62.81 GiB

Everything that is not the PLE table is the same size in all three tiers — within 0.21 GiB. These are one model plus a differently-quantized lookup table.

That predicts identical GPU memory, and measurement confirms it. GTT resident, identical across all three tiers at every depth:

context	GTT at load	GTT after prompt
8,192	63.3 GiB	63.6 GiB
32,768	64.1 GiB	64.8 GiB
65,536	65.1 GiB	66.2 GiB
131,072	67.2 GiB	69.1 GiB

~63 GiB is the non-PLE mass plus compute buffers. The PLE table never enters GPU memory.

It is streamed off the SSD, through the OS page cache — automatically, with no flags. Measured directly with mincore(2) against the live model file during a 131,072-token run: 40.13 of 113.47 GiB resident in page cache, with system-wide Cached at 40.4 GiB — the page cache was this file. Sampled every 30 s under sustained load it holds 32–35 GiB, rising and falling as the kernel reclaims. Those pages are file-backed and reclaimable, which is exactly why they never cost you real memory.

Three consequences that matter in practice:

1. --no-mmap will get you OOM-killed. It forces the table into anonymous memory, which is neither file-backed nor reclaimable. The cgroup killer takes the process with nothing in the server log — the only evidence is dmesg.
2. Don't force the table to CPU with -ot. I tried -ot "per_layer_token_embd|ple\.=CPU": identical GTT, and generation fell from ~23 to 13.4 tok/s. The kernel already streams it better.
3. A bigger tier costs disk, not GPU memory. If you have the storage, take STRIX.

Long context — the full ladder

Four depths on one Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), full 49/49 offload. Each prompt is a unique, non-overlapping slice of a real 10.2 MB source-code corpus (61,469 distinct words), sized exactly with /tokenize → slice → /detokenize, cache_prompt: false, and prompt_n verified every run.

context	prompt tokens	GTT at load	GTT after prompt	pp tok/s	gen tok/s
8,192	6,963	63.3 GiB	63.6 GiB	296	21.72
32,768	27,852	64.1 GiB	64.7 GiB	278	21.31
65,536	55,705	65.1 GiB	66.1 GiB	250	18.26
131,072	111,411	67.2 GiB	69.1 GiB	190	15.34
262,144	8,000	71.7 GiB	72.0 GiB	307	22.48
262,144	200,000	71.7 GiB	74.9 GiB	128	10.46

The full native 262,144-token context runs on a 128 GB Strix Halo. Not 131,072 — that was simply where I stopped the first ladder, and people rightly asked. At the top rung the box sits at 74.9 GiB GTT with a 200,000-token prompt loaded, leaving real headroom.

The context window is nearly free; depth is what costs. GTT at load grows only ~3.9 GiB from 8k to 128k, and ~8 GiB all the way to the full 256k window — Qwen Sparse Attention's 512-block / 2048-token budget caps KV, where a conventional model would spend tens of GiB. What you actually pay for is how much you put in that window: at 262,144 a short prompt generates at 22.48 tok/s, a 200,000-token prompt at 10.46. It degrades smoothly, no cliff.

(262,144 rows measured on STRIX_LEAN. All three tiers showed identical GTT at every lower rung and agreed on generation to 0.01 tok/s at 131,072, so this behaviour is the model's, not the tier's.)

⚠ Honest limit: the corpus is one source tree. A workload spanning many languages and repositories will touch more of the n-gram table before it saturates. What these numbers do show is that the table does not grow without bound with depth.

For short prompts (~3,300 tokens) this tier measured 348 tok/s prompt processing, 22.4 tok/s generation — median of 3, a different prompt each run.

Every tier, every depth

Generation tok/s. Same box, same flags, prompt sized to 85% of each context window:

context	FAST	STRIX_LEAN	STRIX
8,192	21.72	22.16	22.02
32,768	21.31	20.44	20.52
65,536	18.26	18.15	18.23
131,072	15.34	15.33	15.34

The tiers are within noise of each other at every depth — at 131,072 they agree to 0.01 tok/s. Tier choice changes file size and (presumably) quality. It does not change speed and it does not change memory. Quality is the one thing I have not measured, so I won't claim it.

(The 65,536 FAST figure is the median of 3 re-runs, 18.21–18.34. A single earlier run read 16.34 and did not reproduce — reported here rather than quietly dropped.)

Files

Sharded so no file exceeds HF's 50 GB limit. Point --model at the first shard; llama.cpp loads the rest automatically.

file	size
Qwen3.8-Flash-Next-Q4_0-ROCmFP4-FAST-00001-of-00003.gguf	41.63 GiB
Qwen3.8-Flash-Next-Q4_0-ROCmFP4-FAST-00002-of-00003.gguf	41.60 GiB
Qwen3.8-Flash-Next-Q4_0-ROCmFP4-FAST-00003-of-00003.gguf	4.71 GiB

Usage

llama-server \
  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-FAST-00001-of-00003.gguf \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 999 --flash-attn on --fit off \
  --ctx-size 131072 --threads 16 --jinja

Leave mmap alone — see the consequences above.

Memory — it's shared, not VRAM

"GPU memory" on this box means GTT — the aperture through which the integrated Radeon 8060S addresses ordinary system RAM. There is no separate VRAM pool, so every GiB the model takes is a GiB the OS no longer has. nvidia-smi-style intuitions do not transfer here.

The file is 87.94 GiB but only ~63–75 GiB is ever resident, depending on context depth. It fits a 128 GB Strix Halo at the full native 262,144 context with room to spare. Load it before anything else has taken UMA, and budget from MemAvailable in /proc/meminfo — never from GTT free, which lies on unified-memory parts.

Acknowledgements

This build would not exist without the work below. Please star and follow these projects — the quantisation format used here is their engineering, not mine.

ROCmFPX — the fork that defines the ROCmFP4 / ROCmFPX tensor formats and carries the qwen4exp architecture support merged from PR #27742. Every ROCmFP4 file in this repository was produced with its llama-quantize and runs on its runtime. Licensed MIT, based on upstream llama.cpp.

llama.cpp — ggml-org and contributors The inference engine, GGUF format and conversion tooling everything here is built on.

AMD ROCm — the compute platform these builds target (ROCm 7.2.4 on gfx1151).

Qwen team — the base model. See base_model for the source release; license is qwen-community-1.0.