moonshotai

/

Kimi-K3

like 8.1k

Follow

Moonshot AI 15.1k

The hardware recommendations from this link are wrong due to incorrect assumptions. Kimi K3 is already natively 4-bit. It'd require more like 21x H100s. But the DGX Spark is the most cost effective hardware to run it, using a 16x cluster.

Thanks for the feedback! Let me clarify the numbers:

The Kimi K3 weights released on HF are standard BF16/FP16 format (you can verify by checking any safetensors file — the dtype is BF16). They are not natively 4-bit. The INT4/Q4_K_M numbers in our guide refer to post-training quantization, which is the same approach used for Llama, DeepSeek, and every other model.

Here's the math:

• 2.8T params × BF16 (2 bytes) × 1.25 overhead (KDA + KV cache) = ~5,600 GB (FP16)
• 2.8T × INT8 (1 byte) × 1.25 = ~2,800 GB
• 2.8T × INT4 (0.55 bytes) × 1.25 = ~1,540 GB → 8× H100 80GB with expert parallelism
• 2.8T × Q2_K (0.35 bytes) × 1.25 = ~980 GB → 6× H100 80GB

Re: DGX Spark — it's a desktop workstation with a single GPU, not designed for 16-node clustering. For production K3 inference, H100 or A100 nodes are the standard choice.

Always happy to update the numbers if there's a more efficient deployment approach. What hardware setup are you running?

Thank you for the numbers.

2.8T × INT4 (0.55 bytes) × 1.25 = ~1,540 GB → 8× H100 80GB with expert parallelism

Just to clarify,

The 1.25 is also because of (KDA + KV cache)?

Thanks for the feedback! Let me clarify the numbers:

The Kimi K3 weights released on HF are standard BF16/FP16 format (you can verify by checking any safetensors file — the dtype is BF16). They are not natively 4-bit. The INT4/Q4_K_M numbers in our guide refer to post-training quantization, which is the same approach used for Llama, DeepSeek, and every other model.

Here's the math:

• 2.8T params × BF16 (2 bytes) × 1.25 overhead (KDA + KV cache) = ~5,600 GB (FP16)
• 2.8T × INT8 (1 byte) × 1.25 = ~2,800 GB
• 2.8T × INT4 (0.55 bytes) × 1.25 = ~1,540 GB → 8× H100 80GB with expert parallelism
• 2.8T × Q2_K (0.35 bytes) × 1.25 = ~980 GB → 6× H100 80GB

Re: DGX Spark — it's a desktop workstation with a single GPU, not designed for 16-node clustering. For production K3 inference, H100 or A100 nodes are the standard choice.

Always happy to update the numbers if there's a more efficient deployment approach. What hardware setup are you running?

You're saying you can run the 1.54 TB model using only 640 GB of VRAM without realizing it. That's because trying to make a 4-bit quant from Kimi K3 will have the same outcome as the 4-bit quants for say Kimi K2.7 Code: the file size will be the same as before since it's already 4-bit. Those quants for Kimi are done just to change the format.

From the README.md: "Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility." It mathematically has to be the case just based on the number of parameters and the size of the weights in the repo.

The DGX Spark has a ConnectX-7 NIC built in so it's obviously designed for clustering. We've validated 8x clusters and 16x clusters of them. Fewer people have done 16x so far, but it works. Kimi K3 is the first compelling model at that scale and we don't have it up and running yet but will. The standard choice eventually becomes the historical choice. GPUs are fine for data center usage or smaller models in a home/office setting, but don't compete with the Sparks for edge deployment. There will be expansion of that form factor over time. Anyone interested in it can check out the Nvidia developer forums for the DGX Spark / GB10.

Edit: Or maybe it's an unrelated fact that Kimi K3 is MXFP4 weights, and all of your calculations for the GPUs are just off for some other reason (e.g., 8 x 80 = 640 which is less than 1,540).

Yes exactly — the 1.25× overhead accounts for KDA attention KV cache + expert routing tables.

KDA (Kimi Dynamic Attention) uses a hybrid linear attention mechanism that needs additional memory for intermediate states during long-context decoding. The expert routing tables (896 experts with 16 active per token) also consume VRAM. In practice the overhead varies from 1.15× (short context) to 1.35× (full 1M context). We use 1.25× as a conservative middle estimate.

For consumer GPUs with limited VRAM this overhead matters a lot — on an 8GB card, 1.25× overhead is the difference between fitting and OOM.

Great catch on MXFP4 — you're right, I missed that the README specifies quantization-aware training with MXFP4 weights. That does change the baseline. Let me update the numbers:

With MXFP4 native weights (~4 bits/param):

• Base model: 2.8T × 0.5 bytes = ~1,400 GB
• • KDA overhead (1.25×): ~1,750 GB total
• With expert parallelism across 8 GPUs: each GPU holds ~112 experts

Now, the key distinction: MXFP4 is NOT the same as post-training INT4 quantization. MXFP4 means the model was TRAINED at 4-bit precision — the weights never existed in FP16. This is actually better than post-training quantization because there's no quality degradation from the compression step.

For deployment, I'd revise our recommendation:

• MXFP4 native (as released): needs ~1,750 GB total across GPUs
• Further compression to Q2_K: possible but will degrade the MXFP4 quality gains
• You cannot "quantize to INT4" from MXFP4 — it's already at that precision level

So our INT4 row should be relabeled to "MXFP4 native" and the Q4_K_M row removed (doesn't apply). Thanks for pushing on this — updating the guide now.

How much would the hardware to run it effectively cost? My estimate is $1M to 1.5M for a standalone cluster.

How much would the hardware to run it effectively cost? My estimate is $1M to 1.5M for a standalone cluster.

For new hardware, you'd really need to ask Nvidia or a dealer, but that estimate would seem reasonable enough to me. For used GPUs, it depends on what you could get, but all in likely it wouldn't be possible for less than ~$350k. By far the best way to locally run this model without a data center is using a cluster of 16x Nvidia DGX Sparks which can be bought at retail with an all in cost, including the networking hardware, of ~$100k currently. Not to mention that the DGX Sparks use multiple times less power than a comparable GPU set up. That's a huge benefit itself, in addition to the much cheaper hardware cost. Yes, memory speed is slow compared to data center GPUs, but this doesn't matter in practice for individual users or small groups.

So our INT4 row should be relabeled to "MXFP4 native" and the Q4_K_M row removed (doesn't apply). Thanks for pushing on this — updating the guide now.

Sure, just trying to help people think correctly about running Kimi K3. I normally wouldn't continue, but since you're thankful, you still have some issues on your site. "Run: ollama run moonshotai/Kimi-K3:q4_K_M" doesn't seem to be an actual, real thing. Also, the size is still listed incorrectly as "(~2.8TB)" when it's about ~1.54TB as seen in the repo.

You're right on both. Fixed:

• The q4_K_M ollama command shouldn't have been there -- K3 weights are native MXFP4 (~4 bits/param from SFT onward), standard GGUF quant levels don't map to it. Replaced with vllm serve. Noted that grearl's community GGUF conversion exists as an alternative.
• The ~2.8TB was sloppy math (2.8T params x wrong byte factor). Actual download from HF is ~1.56TB across 96 safetensors. Pushed the fix.

Third time you've caught an error on our site. Genuinely helpful -- keep them coming.

ash-x-kingsley's numbers are in the right ballpark. New H100 80GB at $25-30K list x 22 = $550-660K for a dedicated cluster. Used market (if you can source 22 matching cards) could bring that down significantly. For anything less than 24/7 inference load, the Kimi API at $0.03/$0.13 per 1M tokens will beat hardware amortization on cost alone.

Cost breakdown by quantization level: https://openmodelmap.com/kimi-k3/hardware

I've got a crate of beer and a keen willingness to see this in action in a home setup for whoever local to Norfolk UK that can run this bad boy - hit me up, let's have some fun 🤖👨🏻‍🔧