System requirements?
Will there be an official system requirements report?
minimum 35 - H100 80GB GPUs required to run this model and total cost of GPUs - $875K – $1.4M
The Compute Nodes: NVIDIA GB200 NVL72-class or equivalent DGX/HGX Cluster
GPUs: 8x NVIDIA B200 GPUs (or Blackwell HGX B200 node). Each B200 packs 192 GB of HBM3e VRAM.
8 x 192 GB = 1,536 GB of ultra-fast HBM3e VRAM on a single baseboard, scaling up to a multi-node cluster if you want to push the full 1M context window simultaneously.
Interconnect: NVLink 5 domain with 1.8 TB/s bidirectional bandwidth per GPU. Without high-speed NVLink, the tensor parallelism communication overhead across 2.8 trillion parameters would turn your token generation speed into a cinematic slideshow (measured in minutes per token).The CPU & System Memory Core
CPUs: Dual AMD EPYC 9654 (96 cores each, 192 cores / 384 threads total per node) or the latest EPYC Turin equivalents.
System RAM: 1.5 TB to 2 TB of DDR5-4800 ECC Registered RAM to handle weight staging, OS overhead, and dynamic offloading pipelines.Storage Subsystem (Feeding the Beast)
Model Weight Storage: The raw weights alone require a massive footprint. You need a 30 TB NVMe Gen5 U.2 Enterprise Array configured in RAID 0/10 using drives like Samsung PM1743 or Solidigm P5530.
Read speeds must clear at least 14+ GB/s to load a 1.3TB model file into memory in under two minutes rather than taking an extended coffee break every time you boot vLLM or SGLang.Infrastructure & Power (The Utility Bill Special)
Power Supply: Dual-redundant 10kW to 12kW Titanium-rated enterprise power shelf.
Cooling: Standard air cooling won’t cut it for eight Blackwell enterprise accelerators churning through 2.8T parameters under heavy load. You are looking at a Direct-to-Chip Liquid Cooling Loop with an external CDU (Coolant Distribution Unit).
Chassis: 4U or 6U dense rack-mountable server chassis bolted into a standard 42U rack, complete with 100GbE NICs (ConnectX-7) for cluster networking.
Shouldn't be too crazy, 4 RTX 5090's to fit the active parameters into VRAM and 2TB of DDR5 RAM should be enough to run this model at home!
Shouldn't be too crazy, 4 RTX 5090's to fit the active parameters into VRAM and 2TB of DDR5 RAM should be enough to run this model at home!
祝你好运！
The amount of outdated misinformation people are spouting off about such an advanced topic is crazy.
35x H100 80GB GPUs would be if the model was at 8-bit precision, when it's natively at 4-bit, which is much more efficient and desirable. People are stuck on thinking 8-bit, or 16-bit, is the best because that's all anyone knew how to do well years ago. It's also probably not helpful that this is the inverse of how bit sizes have typically been considered since it's a situation where you ideally want less, not more. But 35x, or even 36x H100s still wouldn't be enough in that case anyway because there wouldn't be any room for context.
8x NVIDIA B200 GPUs would barely not be enough to fit the model, and again, no room for context, so it'd be 9 or 10 in reality, which in practical situations is going to require being split across two workstations with the required networking.
A path like 4x RTX 5090s and 2 TB of system memory would probably be possible, with software support, but it'd be slow, almost certainly unusably slow. This is a technique that's more for smaller models, and even then you'll pay a high cost in slowness.
Most people don't seem to be aware of the current state of hardware, but the best way for individuals and small orgs to run this right now is a cluster of 16x Nvidia DGX Sparks. That would cost a little over $100k currently, but prices continue to go up, and it'd use around up to 2,500 watts of power when properly set up since it's more efficient than conventional GPUs.
AMD has a competitor to the DGX Spark, the Strix Halo, but it lacks proper networking support for real clustering so it's not viable for something like this size, and the situation is similar with a cluster of 4x Mac Studio (M3 Ultra) with 512 GB of memory each, which while much better from a networking perspective because of only 4 nodes instead of 16, aren't even available for sale right now and getting them from a third-party would cost more than going with the DGX Sparks.
Anyway, thanks to Moonshot and the CCP for wanting to throw money at this. I also look forward to all of the other 3-trillion-parameter-class models your sibling orgs will be releasing later this year. Truly great work and hopefully my post is helpful to anyone who wants to run it locally.
Edit: Corrected an error stating 16-bit instead of 8-bit in the first part on 35x H100s.
If you’re unsure, please refer to the serving cookbooks for open-source frameworks such as SGLang. Although they mention that the setup hasn’t been fully validated, they released Docker images with Kimi K3 support as soon as the model was open-sourced. Therefore, it’s worth trying on a 32×H100 setup.
https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3#hw=b300&pdMode=unified&strategy=low-latency&spec=dspark&hicache=off
If you’re unsure, please refer to the serving cookbooks for open-source frameworks such as SGLang. Although they mention that the setup hasn’t been fully validated, they released Docker images with Kimi K3 support as soon as the model was open-sourced. Therefore, it’s worth trying on a 32×H100 setup.
https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3#hw=b300&pdMode=unified&strategy=low-latency&spec=dspark&hicache=off
As stated in my above post, 32x H100 would be massive overkill for running this model since Kimi K3 is native 4-bit. These ideas are misinformation coming from an asumption that it's not 4-bit. It could probably be run with 21x or 22x H100s.
I think 2 B300x8 machine is possible.