kwanhee/Kimi-K2.5-REAP50-INT4-W4A16

Kimi-K2.5 · REAP-50% + INT4 W4A16 — a bit-exact pruning slice

Half of Kimi-K2.5's routed experts removed, with no requantization at all. The 192 kept experts carry moonshotai's own INT4 weights, bit-for-bit. It recovers 97.99% of the dense OpenLLM average at 288 GB.

This is a diagnostic, not a competing arm. Because nothing was requantized, the step from the release to this checkpoint is pure expert pruning with zero quantization confound — which is exactly what makes it useful, and exactly why it should not be dropped into a ranking table next to W4A4 arms.

routed experts	192 (from 384), top-8 unchanged
weights	INT4, group size 32, symmetric — inherited, not recomputed
activations	none — W4A16, weight-only
size	288.4 GB · 58 shards · 104,870 tensors
left in bf16	self_attn, shared_experts, the dense MLPs, lm_head, the entire vision tower

How it was built

REAP is pure selection — it removes experts, it never rewrites the survivors. So the kept experts can be lifted straight out of the release:

1. Recover the retained-expert indices. Exact-match the REAP50 bf16 model's router rows against the base router — routers are unquantized in both, so this is exact rather than approximate. Result: 192/192 exact in all 60 MoE layers, unique and order-preserving, zero inexact matches. This was necessary because llm-compressor 0.13.0 emits no retained-expert map.
2. Slice the release checkpoint tensor by tensor — 103,680 expert tensors dropped (60 × 192 × 9), 120 routers sliced.
3. Verify. 25/25 sampled weight_packed tensors bit-identical to the release.

Built by reap_accel/int4arm/; build log reap_accel/logs/build_int4.log. There is no moe-sq compression config for this arm, because no compression ran.

Results

4×B200, ctx 73728, kv-cache-dtype auto. Recovery against the dense Kimi-K2.5 reference on the same stack.

bench	this model	dense	recovery
OpenLLM v1 (6-task avg)	80.77	82.43	97.99%
aime25	0.9100	0.9567	95.1%
gpqa:diamond	0.7879	0.8949	88.0%
math500	0.9784	0.9636	101.5%

Compare against the same prune carried into NVFP4 W4A4 (Kimi-K2.5-REAP50-NVFP4-W4A4-GS16, 80.00 / 97.06%): the difference between the two is the cost of our NVFP4 stage, measured with the prune held fixed.

Two traps in this checkpoint

1. quantization_config is nested under text_config, where the NVFP4 arms put theirs at top level. This is the nesting that makes compressed-tensors' run_compressed detector miss it — the model will load with experts still packed, silently, and you will get wrong numbers rather than an error. Check for it explicitly:

cfg = json.load(open("config.json"))
qc = cfg.get("quantization_config") or cfg["text_config"]["quantization_config"]

2. It ships 34,560 weight_shape tensors and depends on a loader that tolerates them. vLLM registers weight_shape only for embeddings, so stock builds fail on load; our fork skips those keys for MoE experts. This is a load-tolerance issue only — it decides whether the server starts, never what it computes.

Scope of the claim

W4A16 does not share a ranking column with the W4A4 sparse arms. It is weight-only int4 with attention, shared_experts, the dense MLPs, lm_head and the whole vision tower in bf16 — a different point in the design space, not a drop-in comparison. The same rule applies to it as to any A16 arm in our study.

Also worth stating plainly: the INT4 release is the starting point for every Kimi row in our work, this one included. There is no official bf16 Kimi-K2.5, so nothing here is "Kimi at 4 bits from bf16" — relative recovery between arms is apples-to-apples, absolute "bits from full precision" is not a quantity we can report.

No multimodal evaluation has been run; all numbers above are text-only.

Verify your download

sha256sum -c SHARD_HASHES.sha256

58/58 must pass.

Serving

kwanhee-lee/vllm-private, branch paired48-nvfp4-moe. Reference configuration:

--max-model-len 73728 --kv-cache-dtype auto -dp 4   # 4×B200

kv-cache-dtype auto is deliberate — it matches the dense Kimi reference these scores are measured against.

License and attribution

Base model moonshotai/Kimi-K2.5, under a Modified MIT License — see LICENSE, whose terms apply to this checkpoint. THIRD_PARTY_NOTICES.md carries the upstream notices (DeepSeek-V3 modeling code). Moonshot AI's upstream model card is preserved for reference at their repo.

The expert weights in this checkpoint are Moonshot AI's, unmodified. Our contribution is only the expert selection and the slice. REAP is due to Cerebras Research; the implementation is the REAPPruningModifier in llm-compressor.