raulvidis/Laguna-S-2.1-ROCmFP4-STRIX-GGUF

Laguna-S-2.1 — ROCmFP4_STRIX (GGUF), DFlash-ready

A 4-bit ROCmFP4_STRIX quant of poolside/Laguna-S-2.1 (118B-A8B), built and measured on an AMD Ryzen AI Max+ 395 (Strix Halo, gfx1151, 128 GB unified memory).

58.4 GiB — Laguna-S-2.1-Q4_0_ROCMFP4_STRIX.gguf — 4.27 achieved bpw

Companion to raulvidis/Laguna-S-2.1-ROCmFP4-COHERENT-GGUF. Prefer this one: identical measured quality, 3.4 GiB smaller, slightly faster.

⚠️ Requires a llama.cpp fork — stock llama.cpp cannot read this file

ROCmFP4 uses GGUF tensor types 100/101, unknown to upstream llama.cpp. You need charlie12345/ROCmFPX.

Serving (no speculation)

env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  llama-server -m Laguna-S-2.1-Q4_0_ROCMFP4_STRIX.gguf \
  -ngl 999 -fa on -c 32768 -dev ROCm0 --jinja --metrics \
  --chat-template-file chat_template.jinja \
  --reasoning-budget 0

Use the chat template from the base repo's auxiliary files. Note yarn_attn_factor must be 1.0 (poolside's value) — unsloth-lineage Laguna GGUFs carry 1.4852, which is wrong; this build has the correct value baked in.

Serving with DFlash speculative decoding (~+25%, up to 2.0× at long context)

DFlash needs a separate draft model, which poolside publishes — it is not included here and should not be re-hosted:

poolside/Laguna-S-2.1-GGUF → laguna-s-2.1-DFlash-BF16.gguf   (2.1 GB)

env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  LLAMA_DFLASH_TARGET_LAYER_OFFSET=-1 \
  llama-server -m Laguna-S-2.1-Q4_0_ROCMFP4_STRIX.gguf \
  -md laguna-s-2.1-DFlash-BF16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -ngl 999 -fa on -c 32768 -dev ROCm0 --jinja --metrics \
  --chat-template-file chat_template.jinja --reasoning-budget 0

⚠️ DFlash on the ROCmFPX fork needs two patches that are NOT merged yet

As of 2026-07-31, the fork loads the DFlash draft but produces useless acceptance without these:

• ROCmFPX#47 — laguna.cpp must publish t_layer_inp, or the first request aborts on GGML_ASSERT(t_layer_inp[il] != nullptr).
• ROCmFPX#48 — load and apply enc.aux_norm (+ attn_gate). Without the norm actually applied in the encoder graph, the drafter fuses six unnormalized feature blocks: output is still correct (the target verifies every token) but acceptance collapses and speculation becomes a net slowdown.

Alternatively use poolside's own fork, poolsideai/llama.cpp branch laguna, which implements DFlash properly — but it cannot read ROCmFP4 tensors, so you'd need a standard-quant GGUF instead of this one.

--spec-draft-n-max: use 4, not the 15 from the model card

Measured here (400-token greedy, two samples per arm):

--spec-draft-n-max	decode t/s	mean acceptance length
none	34.8 / 35.5	—
6	37.6 / 38.5	3.75 / 3.72
5	35.8 / 39.0	3.18 / 3.41
4	41.5 / 43.9	3.43 / 3.34

On COHERENT the same sweep put n-max 15 at 15.5 t/s — worse than no draft at all. This target's batch throughput is only ~2–3× its decode, so verifying a 16-token block costs more than sequentially decoding the ~4 tokens that get accepted. Tune n-max just above the observed mean acceptance length, and ignore the aggregate acceptance rate — at n-max 15 it reads 0.198 while position-1 acceptance is a healthy 0.716.

Speedup grows with context (measured on the COHERENT sibling, same drafter): 1.16× at 60 tokens, 1.59× at 7.2k, 2.01× at 21.5k (20.9 → 42.1 t/s) — DFlash largely cancels the KV-bandwidth decay. Prefill cost is −4 to −5% at depth but −32% on a 60-token prompt, since the draft pass is a fixed cost per prefill.

Measured quality

--parallel 1, greedy, thinking off (probe-verified), single runs.

benchmark	STRIX (this)	COHERENT (sibling)
GSM8K (n=500, 5-shot)	0.924 ±0.012	0.924 ±0.012
HumanEval (pass@1, 164)	0.927	not run
HumanEval+ (pass@1, 164)	0.890	not run

The two quant recipes are indistinguishable on quality — the choice is on size and speed, where STRIX wins. (For reference, on a different model, KAT-Coder-V2.5, the same STRIX-vs-COHERENT comparison showed a large gap; that does not generalise to Laguna.)

Caveats

• DFlash makes the lane non-deterministic. Greedy output is not bit-identical to no-draft (batch-shape-dependent FP rounding changes reduction order and can flip near-argmax ties), and repeated runs differ from each other. Measured swing of ±3 points on HumanEval+ across identical runs. Turn DFlash off for any quality measurement; the coding numbers above were taken with it on and carry that noise.
• GSM8K/HumanEval do not measure what this model is for. Laguna is an agentic SWE model whose published strengths are repo-level, tool-using, multi-turn (Terminal-Bench, SWE-Bench, DeepSWE, Toolathlon). Single-function completion understates it; don't rank it against generalist models on these tasks.
• Thinking-off numbers only; reasoning-on was not benchmarked.
• Memory: budget on peak RSS, not file size. With the fleet quiet this loads in ~59 GB, but prompt-cache RAM and context checkpoints add several GB under concurrent load. --cache-ram 1024 --ctx-checkpoints 4 keeps it bounded.

How it was built

Quantized from a local BF16 conversion of the base model using the base model's own imatrix (poolside-aux/laguna-s-2.1.imatrix) with the fork's llama-quantize at Q4_0_ROCMFP4_STRIX. The recipe is selective, not uniform: attn_k/attn_v get the quality layout, attn_q/attn_output/ffn_*_exps get the fast layout, token_embd gets Q6_K.

Credit to poolside for the base model, the imatrix, and the DFlash drafter; and to the ROCmFPX fork for the FP4 kernels.