prometheusAIR/MiniMax-M3-sm120-safe-GGUF

MiniMax-M3 GGUF — sm_120-safe MSA quants

Built with MiniMax M3.

GGUF quantizations of MiniMaxAI/MiniMax-M3 (428B-A23B, MiniMax Sparse Attention) that contain no iq1_s, iq2_s or iq3_s tensors, so they actually run on NVIDIA Blackwell / compute capability 12.0 (RTX PRO 6000, RTX 5090, RTX 5070 Ti …).

Why these exist

iq1_s, iq2_s and iq3_s have broken CUDA matmul kernels on sm_120. Verified with test-backend-ops test -o MUL_MAT -b CUDA0 on llama.cpp master b10148 (2026-07-27): 11/11 FAIL for each of those three types, while iq2_xxs, iq2_xs, iq3_xxs, iq4_xs, iq4_nl, q2_K, q3_K, q4_K, q5_K and q6_K all pass.

Quant names hide this. llama-quantize's own IQ4_XS preset assigns iq3_s to the MoE expert gate/up tensors, so a file labelled IQ4_XS can be mostly a broken type. At the time of writing, every published sub-Q4 M3 GGUF we surveyed was affected — one popular 189 GiB IQ4_XS build carries 275 of its 426 G params in iq3_s.

These builds pin every tensor type explicitly and are verified after quantization, not assumed.

Variants

Measured on a single host: RTX PRO 6000 96 GB + RTX 5070 Ti 16 GB, 125 GiB DDR5, Ryzen 9 9950X3D. 32K context, f16 KV, -ngl 99 -ts 95,5 --threads 12, experts split across both GPUs and CPU.

variant	expert gate/up	expert down	size	bpw	decode tok/s	prefill tok/s	needle @95% depth
Q3_K	q3_K	q4_K	196 GiB	3.95	11.5–12.6	~96	5/5
IQ3_XXS	iq3_xxs	iq4_xs	180 GiB	3.63	13.5–14.4	~116	5/5
Q2_K	q2_K	q4_K	170 GiB	3.39	15.1–16.3	~130	4/5
IQ2_XS	iq2_xs	q4_K	160 GiB	3.22	16.4–17.6	~150	7/11

Everything else is q8_0, and all 228 indexer.* tensors are kept at F32 — MSA's block selection depends on them and they cost under 1 GB.

"needle @95% depth" = exact passphrase retrieval with the needle at 95% depth in a ~29,700-token context (near the 32K ceiling) — the hardest rung we tested, repeated 5–11×. At moderate depths (25/50/75%) every variant scores at or near 5/5; IQ2_XS is 14/15 there. All four degrade somewhat at the 95% extreme; the ordering tracks bpw but only the Q3_K/IQ3_XXS-vs-IQ2_XS gap is even close to significant (p = 0.09), so treat the ranking as suggestive.

Decode is depth-flat in all four — e.g. Q3_K holds 12.55 → 12.76 tok/s from 677 to 20,890 tokens — which is the MSA sparse path working as intended rather than falling back to dense.

Which to pick

• Q3_K (196 GiB) — highest bpw, clean deep retrieval. Slowest.
• IQ3_XXS (180 GiB) — matches Q3_K on retrieval while being faster. Best quality/speed balance if it fits.
• Q2_K (170 GiB) — faster still, one deep-needle miss in five. Not statistically separable from the two above (p = 0.33). Best all-round choice for most setups, and the cheapest to reproduce (pure K-quant, no imatrix required).
• IQ2_XS (160 GiB) — fastest and smallest. Clean at moderate needle depths (14/15 at 25/50/75%), weaker at the 95% extreme (7/11, vs 10/10 for Q3_K and IQ3_XXS). That gap does not clear significance at this sample size (Fisher exact, p = 0.09) — it is suggestive, not established. Reasonable under RAM pressure; prefer Q2_K if you have the headroom and your workload puts critical detail at the very end of a near-full context.

The i-quants were not worth their cost here. They took 241 minutes of quantize time versus 60 for both K-quants, and neither beat the K-quant nearest it — q2_K at 3.39 bpw matched iq3_xxs at 3.63 on every probe. If you are rebuilding these yourself, start with the K-quant recipes.

Required llama.cpp settings

Needs llama.cpp with MSA support — PR #24908, merged 2026-07-26 (b10148 or later). Earlier builds cannot load these files.

-fa on -np 1
--cache-type-k f16 --cache-type-v f16      # MANDATORY, see below
--fit off                                  # MANDATORY, see below
--no-context-shift

KV cache must be f16. A quantized KV cache enables Hadamard attention rotation (attn_rot_k is gated on ggml_is_quantized(type_k) && n_embd_head_k % 64 == 0, and M3's head dim is 128). MSA does not support it and aborts:

GGML_ASSERT(!inp_attn->self_k_rot && !inp_attn->self_v_rot
            && "MSA: attn-rot not supported")  -- src/models/minimax-m3.cpp:337

So -ctk q8_0 crashes on load. f16 KV costs 120 KiB/token. If KV size ever forces your hand, LLAMA_ATTN_ROT_DISABLE=1 bypasses it at the cost of KV accuracy.

--fit off is separately required — the automatic memory-fit probe builds a rope-shift graph and trips the same assert before the model loads.

Two offload gotchas: -ncmoe counts blocks, and M3's first 3 are dense, so N is offset by 3. -ts splits by layer count, not bytes — on a 96/16 GiB pair, -ts 90,10 hands the small card six ~3 GiB layers and fails to allocate; use 95,5.

M3 reasons by default. A tight max_tokens returns empty content, not a short answer — reasoning consumes the budget and nothing is left for the reply. Give it room or set --reasoning-budget.

Set thinking_mode explicitly, or reasoning tags leak into content:

--chat-template-kwargs '{"thinking_mode":"enabled"}'

The chat template accepts enabled / disabled / adaptive, and defaults to adaptive when the field is unset — which llama.cpp does not set. In adaptive mode the model decides per-response whether to think, and when it declines it emits a bare </mm:think> closing tag with no opener. Nothing matches it, so it leaks into message.content. Forcing enabled makes the stream well-formed and reasoning lands in message.reasoning_content as intended.

Tool calling does not work (upstream gap)

Any request containing tools fails:

{"error":{"code":400,"message":"Failed to initialize samplers: failed to parse grammar"}}

MiniMax-M3 uses a bespoke XML tool-call syntax with a namespace token — ]<]minimax[>[<tool_call> wrapping <invoke name="..."> elements — and llama.cpp has no handler for it: common/chat.cpp contains no MiniMax chat format, and its registered formats (CONTENT_ONLY, PEG_SIMPLE, PEG_NATIVE, PEG_GEMMA4) don't match. With tools present it cannot build a grammar and sampler init fails.

This is an upstream llama.cpp gap, not a defect in these quants, and it affects every MiniMax-M3 GGUF equally.

Workaround for clients that always send tool definitions (Hermes, many agent harnesses): send "tool_choice": "none". The grammar is then not built and the model works normally for chat — you just don't get tool calls.

Verify before you trust

Type census is the whole point of these files, so check rather than take our word:

python3 census.py MiniMax-M3-Q2_K-00001-of-00018.gguf

Expect 1176 tensors, 228 indexer, and no iq1_s/iq2_s/iq3_s. The script is included.

Limitations

• Retrieval degrades near the 32K ceiling for all variants at 95% needle depth. IQ2_XS trends worst (7/11 vs 10/10 for Q3_K and IQ3_XXS) but the gap is not statistically established — see the sample-size note below.
• The needle probe uses homogeneous repeated filler, which makes the needle more salient than a realistic long-context task — treat these pass rates as an upper bound.
• n = 5–11 per cell. No pairwise difference between variants reaches statistical significance (closest is Q3_K+IQ3_XXS vs IQ2_XS at the 95% rung, p = 0.09). An earlier revision of this card claimed p = 0.036 for that comparison; that was computed on 6 samples and did not survive 11. The ordering below tracks bpw and is plausible, but it is not established.
• Tool calling is broken — see the section above. Upstream llama.cpp gap, affects all MiniMax-M3 GGUFs.
• Untested: vision (an mmproj is provided but unexercised), multi-turn behaviour, contexts beyond ~29.7K, and any standard benchmark. No perplexity numbers.
• Measured on one host. Expect different throughput with a different GPU/RAM split.

Provenance

Converted from the original BF16 safetensors (795.5 GiB, 59 shards) with convert_hf_to_gguf.py at b10148, then quantized with explicit per-tensor type pinning. imatrix reused from Serpen/Minimax-M3-MSA-GGUF (1524 entries, all 60 layers, no gaps) for the i-quant builds; the K-quant builds do not require it.

Recipes, census script and needle harness are included so the whole thing is reproducible.

License

MiniMax Community License, inherited from the base model. Commercial use requires "Built with MiniMax M3" attribution and a notice email; over $20M annual revenue requires written authorization. See the base model repository.