prometheusAIR/Kimi-K3-REAP55-GGUF

Kimi-K3-REAP55-IQ1_M-GGUF

Kimi-K3 (2.8T-A104B) with 55% of its experts removed, then quantised to ~2.15 bpw — 319 GiB.

REAP expert pruning (896 → 400 experts per layer, all 92 MoE layers) followed by iq1_m gate/up + iq2_xxs down with a self-built imatrix. Dense weights at q8_0.

So far, the smallest Kimi-K3 GGUF that has been shown to work. Highly experimental build mainly for comparison purposes. Built specifically to avoid the iq1_s/iq2_s/iq3_s tensor types, which have broken CUDA matmuls on sm_120 (RTX PRO 6000 Blackwell, RTX 50-series) and silently produce garbage.

size	319 GiB (8 shards)
experts	400 / 896 per layer (55.4% pruned), top-16 unchanged
saliency retained	77.6% of router-weighted expert saliency
expert quant	iq1_m gate/up, iq2_xxs down
dense / embed / output	q8_0
overall	~2.15 bpw
needle test	15/15 exact through 64K, at 15% / 50% / 85% depth
decode	0.48 tok/s warm, 0.32 tok/s cold (see hardware note)
prefill	82–90 tok/s at depth

Requires llama.cpp PR #26185

Kimi-K3 support is not in any llama.cpp release. You need PR #26185 (kimi-k3 arch). Nothing here works without it.

Known chat parser issue

llama.cpp's auto-parser derives delimiters by diffing template renders and picks the shortest distinguishing strings. For XTML that gives it <|sep|> and <|close|> — both of which appear in every channel. So reasoning terminates at the first bare <|close|> and the remaining scaffolding falls into content. The fix is a dedicated handler matching full channel tags, dispatched on <|open|>think<|sep|> + <|open|>response<|sep|>, following the same pattern llama.cpp already uses for GPT-OSS's channel format. It also wires up tool calls — currently untested and almost certainly broken for the same ambiguity reason.

Parser fix for llama.cpp

You must build the small patch included here (kimi-k3-chat-parser.patch). See Building llama.cpp below. If/when the architecture is merged upstream, this requirement goes away.

Running it

llama-server \
  --model Kimi-K3-REAP55-IQ1_M-00001-of-00008.gguf \
  -ngl 99 -ot "ffn_.*_exps=CPU" --no-repack \
  -c 32768 -ctk f16 -ctv f16 \
  -b 8192 -ub 8192 \
  --threads 8 --fit off --jinja

Four of those flags are not obvious and were each measured, not guessed:

• --no-repack is mandatory. Without it the CPU backend claims the expert tensors for its repack buffer and tries to allocate the entire pool in RAM; the load aborts with failed to allocate CPU_REPACK buffer of size ....
• -ngl 99, never a partial value. llama.cpp offloads the last N layers (i_gpu_start = n_layer + 1 - ngl), so any partial -ngl puts layer 0 on CPU — and layer 0 is K3's single dense layer, carrying a fused Gated Delta Net op. The result is a hard abort in ggml-cuda.cu after prompt processing appears to succeed. GGML_CUDA_DISABLE_GRAPHS=1 does not help. This works because q8_0 dense is only ~55 GiB.
• -ub 8192 is ~5x faster than the default on a disk-streamed setup. Every ubatch draws essentially all experts, so each one costs a full sweep of the expert pool — you want as few sweeps as possible. Measured on a 5099-token prompt: -ub 512 = 1383 s, -ub 8192 = 274 s. This is the opposite of the usual VRAM-resident intuition.
• --threads 8, not more. Prefill here is memory-bandwidth-bound, not core-bound. On a 16-core/32-thread part, 8 threads beat 12 by 11.7% and 30 threads were slower than 12. Tune to where your aggregate memory bandwidth peaks.

Keep mmap on (do not pass --no-mmap) — the expert pool streams from disk by design.

Hardware note — the speeds above are storage-bound, not model-bound

Measured on: RTX PRO 6000 Blackwell (96 GB), Ryzen 9 9950X3D (16c/32t), 125 GiB DDR5, 2× Samsung 9100 PRO Gen5 NVMe in RAID0.

Dense weights sit on the GPU; the 262 GiB expert pool streams from NVMe through the page cache. With 125 GiB of RAM only ~42% of the pool stays cached, so decode speed is set by how fast your storage and page cache can supply expert weights, not by the GPU. Consequences:

• Warm ≠ cold. Sustained work keeps the pool warm and reaches ~0.48 tok/s; a first request after load is ~0.32. Both are honest, they measure different things.
• More RAM helps more than a faster GPU. If the pool fits in page cache, decode becomes RAM-bandwidth-bound instead and should be several times faster.
• Decode is flat with context — 0.43–0.51 tok/s from 4K to 64K, no roll-off, because the cost is streaming experts rather than attention.

What was pruned, and what that costs you

REAP (REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression, Cerebras) scores each expert by gate x ||expert_output|| over a calibration set and keeps the highest. Saliency was measured over 264,141 tokens with a purpose-built tool that hooks llama.cpp's graph directly, so no PyTorch pipeline or original safetensors were needed.

The calibration corpus was deliberately narrow — roughly 33% code, 28% technical prose, 20% tool-calling/agentic traces, 20% long-context — and multilingual content was dropped entirely.

Whatever a calibration corpus under-represents gets pruned away silently. Expect this build to be weaker than the full model on multilingual work, and on domains far from code, technical English, reasoning and tool use. If you need breadth, use an unpruned quant.

Every sample was rendered through K3's real chat template with populated thinking channels, so the experts were scored on the chat+reasoning distribution the model actually runs in rather than on raw text.

Prune depth in context: Cerebras validated ~30–50% pruning with near-parity; at 50% on Qwen3-480B-Coder they retained 97.6% of coding ability. This build is at 55%, just past that envelope, which is why it is domain-targeted rather than general-purpose.

Validation

• Census: 2573 tensors, 276 expert tensors at 400 slots, expert params exactly 276 x 11,010,048 x 400, zero sm_120-broken types.
• Imatrix coverage: all 400 kept experts in all 92 layers received activations (minimum 225, mean 10,813) — no expert was quantised blind.
• Needle-in-a-haystack: 15/15 exact retrieval at 4K/8K/16K/32K/64K x depth 0.15/0.5/0.85.
• Reasoning spot-check: correct arithmetic with self-verification, and it terminates naturally rather than running to the token cap.

Not tested: multilingual, tool-calling end-to-end, context beyond 64K (1M would need 28 GiB of KV alone), and any standard benchmark suite. Perplexity is not quoted because the only corpus measured was our own calibration set, which flatters a model pruned toward it.

Known issues

• XTML markers leak into content (<|open|>think<|sep|>, <|close|>response<|sep|>). This is an upstream parser gap, reproducible on the unpruned model — not a defect of this build.
• K3 is always-thinking and there is no reasoning-budget control yet, so allow a generous completion budget or answers get truncated inside the thinking block. Apply included patch for temporary fix

Provenance

Converted from the native MXFP4-QAT release. The routed experts ship as MXFP4, so conversion to GGUF is a lossless byte repack — this build derives from a full-precision master, not from a requantised intermediate. Pruning was done in GGUF space by slicing the expert dimension, which is outermost and therefore contiguous per expert, so surviving experts are byte-exact copies.

Licensed under the Kimi-K3 license, inherited from the base model.

Building llama.cpp

The patch is generated against upstream master cf11c4c05 (2026-07-27), where it is verified to apply cleanly and compile with zero errors. A nearby master should work with minor fuzz.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout cf11c4c05
git apply /path/to/kimi-k3-chat-parser.patch

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120   # 120 = Blackwell; set to your GPU
cmake --build build --target llama-server -j