Instructions to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
# Run inference directly in the terminal:
llama cli -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
# Run inference directly in the terminal:
llama cli -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
# Run inference directly in the terminal:
./llama-cli -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Use Docker

docker model run hf.co/hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

LM Studio
Jan

vLLM

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Ollama
How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with Ollama:
```
ollama run hf.co/hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
```
Unsloth Desktop

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with Docker Model Runner:
```
docker model run hf.co/hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF
```

Lemonade

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Run and chat with the model

lemonade run user.Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen 3.8 Next Flash ROCMFPX MUTANT RAZERFAST

A role-aware, calibration-selected 4-bit + 3-bit build of Qwen3.8-Flash-Next for AMD Strix Halo, using three new GGML weight formats that beat their ROCmFPX / ROCmFP4 predecessors on reconstruction error at exactly the same bits per weight.

125B total parameters, 6B active, 51B n-gram (PLE) embedding table, matched 4B MTP draft head. 90.1 GB target + 2.75 GB MTP sidecar. Loads and generates on a single Ryzen AI MAX+ 395 / Radeon 8060S with the PLE table either resident in shared memory or paged from NVMe with O_DIRECT.

This is not a stock-GGUF release. Stock llama.cpp will refuse these files with unknown type. You need the matching runtime: https://github.com/hstolte11-collab/ROCmFPX/tree/mutant-razerfast-b10752 (upstream llama.cpp b10752 + a 68-path overlay). Build notes are in MUTANT-RAZERFAST.md on that branch.

Files

File	Bytes	SHA-256	Tensors
`Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST.gguf`	90,125,769,984	`89abf38645bfb5bf32bcece6f8cf60701c4bf66d9ac5c46b36c7e843b4d45723`	1,224
`Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-MTP.gguf`	2,752,436,096	`7fd0f48a0e146393a3d9bb8068fab6d948a5de08cd55940dcdb9231033e2212e`	34

SHA256SUMS.txt in this repo carries the same two lines. The MTP file is a draft head, not a model; it only works together with the target and is bound to it by shared token_embd.weight / output.weight payloads.

Why "MUTANT"

The three formats below are mutations of the ROCmFPX family: same block geometry, same bits per weight, same 17-byte or 70-byte wire, but the metadata byte is re-purposed to buy back precision that the incumbent layouts throw away.

`Q4_RZ5_FAST` (GGML type 69) - 4.25 bpw, 770 body matrices

ROCmFP4 FAST stores 32 weights as 16 nibble bytes + 1 metadata byte. The incumbent Codebook10 layout wastes one of its sixteen codes on a duplicate zero (+0 and -0). RZ5 turns that dead code into a +5 or -5 level and uses the top metadata bit to choose the polarity per block, while the lower seven bits keep the UE4M3 scale:

Codebook10 : {0, 1, 2, 3, 4, 6, 8, 10,  0, -1, -2, -3, -4, -6, -8, -10}
RZ5 mode 0 : {0, 1, 2, 3, 4, 6, 8, 10, +5, -1, -2, -3, -4, -6, -8, -10}
RZ5 mode 1 : {0, 1, 2, 3, 4, 6, 8, 10, -5, -1, -2, -3, -4, -6, -8, -10}

Equal wire, one extra usable level, exhaustive per-block scale x mode search.

`Q4_UE4M4_FAST` (GGML type 71) - 4.25 bpw, 38 matrices

Same 17-byte block and Codebook10 codes as ROCmFP4 FAST, but the full 8-bit metadata byte is an unsigned UE4M4 scale: 4 exponent bits, 4 mantissa bits, all 256 encodings finite, exhaustive 256-way scale search. Finer scale resolution, no extra level.

`Q3_PLE_K160_RZ_S3` (GGML type 111) - 3.5 bpw, the 22.4 GB PLE table

per_layer_token_embd.weight is [160, 320001536] - 51.2 billion values. ROCmFP3 packs 3-bit codes {0, +1, +2, +4, -0, -1, -2, -4} with a UE4M3 scale per 16 values. RZ-S3 replaces the duplicate -0 code with +3 or -3 (polarity chosen by the metadata high bit per group). Rows are stored as a row-native 160-value record: 60 code bytes + 10 group metadata bytes = 70 bytes, no padding, so the HIP GET_ROWS kernel can gather one hashed n-gram row in one contiguous read.

Measured: equal-BPW reconstruction precision

These are calibration-set reconstruction metrics on the real BF16 source tensors, computed on CPU before quantization. They are not perplexity and not task scores. They compare candidates at identical bits per weight, so every improvement below is precision gained for free.

Body formats at 4.25 bpw (806 sampled tensor entries, 698,482,688 sampled values per candidate)

Reference = ordinary ROCmFP4 FAST (q4_c10_int4_fast). Lower MSE is better; percentages are error reductions relative to the reference.

Metric	ROCmFP4 FAST	Q4_RZ5_FAST	Q4_UE4M4_FAST
llama-normalized activation MSE	8.319767e-06	7.525537e-06 (-9.55%)	8.335111e-06 (+0.18%)
raw call-energy MSE	1.919011e-05	1.721405e-05 (-10.30%)	1.917726e-05 (-0.07%)
unweighted reconstruction MSE	3.707324e-06	3.411512e-06 (-7.98%)	3.793842e-06 (+2.33%)

RZ5 won all three objectives at the model level and in 22 of 23 tensor roles. The one exception is blk.*.ssm_beta.weight (36 tensors), where UE4M4 wins the two activation-weighted objectives (-6.34% vs -4.16% for RZ5 on normalized activation MSE). So the recipe is role-aware: RZ5 everywhere, UE4M4 on ssm_beta and the two output hyper-connection matrices.

Per-role RZ5 reductions on normalized activation MSE range from -2.99% (ffn_gate_exps) to -10.80% (hc_attn_down).

PLE format at 3.5 bpw (1,024-row stratified diagnostic, 41,962 gathers, 16 heads)

Reference = incumbent ROCmFP3 (q3_ue4m3, 70-byte K160 rows). All five candidates have identical wire size.

Candidate	count-weighted MSE	reduction	count-weighted rel-L2	QSNR dB
ROCmFP3 UE4M3 (incumbent)	2.582816e-06	-	0.17908	14.94
Q3 UE4M4 G16	2.417267e-06	-6.41%	0.17325	15.23
Q3 RZ-S5	2.014808e-06	-21.99%	0.15817	16.02
Q3 RZ-S6	1.947553e-06	-24.60%	0.15551	16.17
Q3 RZ-S3 (shipped)	1.865081e-06	-27.79%	0.15218	16.35

RZ-S3 also wins unweighted MSE (-27.74%) and count-weighted relative L2 (-15.02%), and takes all 16 per-head wins. It was selected under a frozen objective (count-weighted MSE) before any full-tensor conversion.

Caveat carried verbatim from the screen: bounded 1,024-row diagnostic (32 top-frequency + 32 uniformly sampled rows per head); not an unbiased full-table metric.

Composition of the target (1,224 tensors)

Type	Count	Where
`Q4_RZ5_FAST`	770	attention q/k/v/o/qkv/gate, routed and shared experts, hyper-connection up/down/inject, `ssm_alpha`, `ssm_out`, PLE key/value
`Q4_UE4M4_FAST`	38	`blk.*.ssm_beta.weight` x36, `output_hc_up`, `output_hc_down`
`Q3_PLE_K160_RZ_S3`	1	`per_layer_token_embd.weight` (22,400,107,520 bytes)
`Q8_0`	26	`token_embd`, `output`, QSA indexer q/k projections x24
`F32`	388	norms, router gates, SSM `a`/`dt.bias`/`conv1d`, hyper-connection norms
`F16`	1	`ple_conv1d`

MTP sidecar (34 tensors): 3 Q4_RZ5_FAST routed expert matrices, 16 Q4_UE4M4_FAST, 4 Q8_0 (shared embeddings/output/indexer), 11 F32.

Runtime knobs

Everything below is exposed by the mutant-razerfast-b10752 runtime. Axes are independent; combine as needed.

Axis	Choice	Flags
PLE residency	resident in shared memory (~90 GB weights)	default
	on disk, `pread` only gathered rows (~70 GB resident)	`--ngram-on-disk --ngram-io-threads 64 --ngram-cache 256`
PLE I/O	`O_DIRECT` (default on) / buffered	`--ngram-direct-io` / `--no-ngram-direct-io`
PLE source	separate GGUF holding only the PLE tensor	`-mp FILE` (implies on-disk)
Speculative	MTP draft head	`--spec-draft-model <MTP.gguf> --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0`
Draft attention	dense (default) / Qwen Sparse Attention	env `LLAMA_QWEN4EXP_MTP_QSA=1` for QSA
Prompt cache	off	`--cache-ram 0 --no-cache-idle-slots`
	RAM	`--cache-ram <MiB> --cache-idle-slots`
	disk (persistent across restarts)	`--cache-ram 0 --no-cache-idle-slots --cache-disk <dir> --cache-disk-max <MiB> --cache-disk-block 256`
	hybrid	RAM flags + disk flags
Context checkpoints		`--ctx-checkpoints 4 --checkpoint-min-step 2048`
Batch geometry		`-b 512 -ub 512` (only geometry exercised)
KV cache	q8_0 K/V, unified, flash attention	`--cache-type-k q8_0 --cache-type-v q8_0 --kv-unified -fa on`

The disk prompt cache stores target KV, draft KV, and speculative pending_h state together, rejects symlinked components, and on any post-mutation restore failure clears all three before falling back to normal prefill. It never mixes state.

Reference launch (Strix Halo, ROCm 10)

export LD_LIBRARY_PATH=$PWD/build-rocm/bin:/opt/rocm/lib:/opt/rocm/lib/llvm/lib
export HSA_ENABLE_SDMA=0

build-rocm/bin/llama-server \
  -m Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST.gguf \
  --spec-draft-model Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  -c 8192 -b 512 -ub 512 -ngl 999 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --cont-batching \
  --ctx-checkpoints 4 --checkpoint-min-step 2048 \
  --ngram-on-disk --ngram-io-threads 64 --ngram-cache 256 --ngram-direct-io \
  --jinja --no-mmap --metrics --host 127.0.0.1 --port 8080

Drop the --ngram-* line if you have the memory to keep the PLE resident. --no-mmap is mandatory on ROCm shared-memory systems.

Measured: it loads and generates

Single-request acceptance smokes on one Ryzen AI MAX+ 395 (Radeon 8060S, 128 GB), ROCm 10.0 / HIP 7.15, native gfx1151, HSA_ENABLE_SDMA=0, cold cache, 48-token prompt, -c 8192 -b 512 -ub 512, q8_0 KV. These are smokes, not benchmarks. One request each, no warmup, no repeats.

PLE	MTP	Prefill tok/s	Decode tok/s	Drafted / accepted	Status
resident	off	52.43	20.39	-	PASS
resident	n-max 4	50.60	17.82	79 / 43	PASS
on disk, O_DIRECT	off	54.30	17.84	-	PASS
on disk, O_DIRECT	n-max 4	50.04	8.84	79 / 43	PASS

MTP drafted 79 tokens and the target accepted 43 (54%) on this 64-token greedy generation. Wall-clock speedup from MTP was not demonstrated in these smokes; decode tok/s above counts accepted tokens per second of the whole loop and is lower with MTP on. Whether MTP wins depends on prompt, sampling, and depth - measure it on your workload.

RAM prompt cache, exact-hit replay (target only, resident PLE): 1,544-token prime at 196.61 tok/s prefill; replay of the same prefix processed prompt_n=4 with cache_n=1540 - the cache actually reused 1,540 tokens.

Live memory fit for the disk-PLE, target-only, 1 x 8192 configuration: 65.64 GiB weights + 0.25 GiB PLE row cache + ~4 GiB runtime = ~70 GiB resident, on a host with 112.6 GiB available. A separate first-load run with target + MTP

disk PLE peaked at 70.0 GB GTT and loaded in 117.6 s.

What has NOT been measured

Read this before you cite anything above.

No perplexity, no benchmark suite, no task accuracy. The quantization claims are reconstruction error on calibration tensors. They say the shipped formats reproduce the original weights more faithfully than the incumbents at equal size. They do not say how the model scores on anything.
No sustained throughput benchmark, no tail latency, no batch or -ub sweep, no long-context run beyond 8192.
No dense-vs-QSA draft A/B. Dense is the default because two independent published runtimes use it; QSA is exposed as an opt-in experiment.
No hybrid or disk-only prompt-cache PASS receipt for this exact package (RAM mode passed; disk mode was interrupted before its receipt).
Portability: only ROCm 10 on gfx1151 has run these GGUFs. CPU and Vulkan have reference codecs and no tuned kernels. CUDA compiles the same kernel sources but was not tested with this model.

Provenance

Full lineage, hashes of every input, and the calibration policy identifiers are in PROVENANCE.md. Short version:

Source weights: Qwen/Qwen3.8-Flash-Next BF16 safetensors (144 files, 360,023,351,514 bytes, size- and header-verified).
Runtime: ggml-org/llama.cpp@b96806d96061049a5b574269b049bf6241d63d46 (release b10752) + overlay a96bbaf181adaac2d6b652a109ce7bdf01e73c6f70c2940ab50c6e13a5cd359e (404,588 B).
Body-format screen policy a84f7e48...7d8a05; PLE screen policy 936ca2d6...5a8b; final recipe union f85df1cd...181e.
Quantized on CPU only, deterministic; header inventories hashed; target and sidecar shared payloads verified byte-identical.

Credits

Qwen team for Qwen3.8-Flash-Next and the Qwen Community License 1.0.
ggml / llama.cpp authors and contributors, in particular the Qwen3.8-Flash-Next correctness work that landed in b10752: Daniel Han (#27941, and qwen4exp support #27742), Pascal (#28123, #28040, #28023), Jaden_Mach (#27466), itsnotoger (#27991), Sigbjorn Skjaeret (#28159, #28173), JJJYmmm (MTP graph #27739), Ryan Monsurate (NextN draft head #27836).
Charlie (charlie12345) - ROCmFPX, whose FAST block layout and Codebook10 these formats mutate. https://github.com/charlie12345/ROCmFPX
CIRU (ciru-ai) for ROCmFP3, the UE4M3 scale codec our PLE rows stay code-compatible with, and the IU4 / PLE-paging / H121 work we studied. https://github.com/ciru-ai/ROCmFPX
AgentionAI / Laurent Zuijdwijk for the first public ROCmFP4 FAST Qwen3.8-Flash-Next GGUF + MTP draft head, whose sidecar contract this package follows. https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF

Hero image generated locally with Z-Image Turbo.

License

Model weights: Qwen Community License 1.0 (copyright Qwen). If you run a Model-as-a-Service or AI Work Assistant business, read clause 2.

Downloads last month: 441

GGUF

Model size

177B params

Architecture

qwen4exp

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(203)

this model

Qwen 3.8 Next Flash ROCMFPX MUTANT RAZERFAST

Files

Why "MUTANT"

Q4_RZ5_FAST (GGML type 69) - 4.25 bpw, 770 body matrices

Q4_UE4M4_FAST (GGML type 71) - 4.25 bpw, 38 matrices

Q3_PLE_K160_RZ_S3 (GGML type 111) - 3.5 bpw, the 22.4 GB PLE table

Measured: equal-BPW reconstruction precision

Body formats at 4.25 bpw (806 sampled tensor entries, 698,482,688 sampled values per candidate)

PLE format at 3.5 bpw (1,024-row stratified diagnostic, 41,962 gathers, 16 heads)

Composition of the target (1,224 tensors)

Runtime knobs

Reference launch (Strix Halo, ROCm 10)

Measured: it loads and generates

What has NOT been measured

Provenance

Credits

License

Model tree for hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

`Q4_RZ5_FAST` (GGML type 69) - 4.25 bpw, 770 body matrices

`Q4_UE4M4_FAST` (GGML type 71) - 4.25 bpw, 38 matrices

`Q3_PLE_K160_RZ_S3` (GGML type 111) - 3.5 bpw, the 22.4 GB PLE table