Hosting 3T-Class MoEs (Kimi K3) on Recycled Hardware: Dual SP3 + GPU Offload
There is a lot of debate on how to host massive open-weight models like Moonshot's Kimi K3 (2.8T MoE with 896 experts, 16 active per token in native MXFP4). The immediate assumption is usually that you need either a $100k+ enterprise GPU cluster or a pure CPU setup that takes an eternity to load prompts.
Instead of dropping tens of thousands on GPU VRAM, you can take advantage of previous-generation enterprise hardware—specifically a dual-socket AMD SP3 platform (EPYC Rome/Milan) running 16 channels of DDR4 memory.
MoE models don't actually run all their parameters at once. Out of Kimi K3's 896 experts, only 16 are active for any given token (approx. 50B active parameters, or approx. 25GB of data read per token). Because of this, you can split the workload cleanly: load the massive 1,400 GB expert pool into system DDR4 RAM where capacity is cheap, and offload the compute-heavy attention layers and prompt pre-filling to a budget high-VRAM accelerator (like a pair of Tesla V100 32GB SXM2s on an NVLink carrier board or a used RTX 3090). Inference engines designed for CPU/GPU offloading (like KTransformers running alongside SGLang or vLLM) handle this streaming pipeline automatically.
On a dual SP3 board with all 16 DDR4 channels populated, you get approx. 300 to 340 GB/s of real-world memory bandwidth. To put that in perspective, 300 to 340 GB/s is right on par with the unified memory bandwidth of a Mac Studio Max chip (300 to 400 GB/s), which local LLM builders constantly praise. The difference here is that instead of being capped at 36GB or 128GB, a dual SP3 setup lets you scale that exact same bandwidth across a massive 1,536 GB (1.536 TB) memory pool using cheap recycled server RAM.
At 25GB per token read, 300+ GB/s gives you a steady decode speed of 3 to 6 tokens per second—right at natural human reading speed, making it completely practical for interactive generation and background agents.
A common concern with RAM offloading is that long prompts will bloat the KV cache and crash system memory. However, Kimi K3 uses Kimi Delta Attention (KDA), a hybrid linear attention design. Unlike standard Transformer attention that blows up memory usage as context grows, linear attention holds compact recurrent state matrices that cut KV cache memory consumption by up to 75%.
When you build out a 1,536 GB RAM pool (using 24x 64GB DDR4 sticks on a 32-slot server chassis, or 12x 128GB LRDIMMs on a 16-slot board), you get 1,536 GB of physical RAM. After Rocky Linux and engine overhead (approx. 36 GB) plus the 1,400 GB model weights, you're left with approx. 100 GB of free system RAM headroom. Combine that with the 34 GB of free HBM2 VRAM on the GPUs (after loading non-expert layers), and you have more than enough space to run K3's full 1,000,000 token (1M) native context window without overflowing memory.
Cost Breakdown (Secondary / Liquidator Markets):
Dual SP3 Motherboard + 2x EPYC CPUs (Used Pulls): $800 – $1,200 USD
GPU Offload Setup (Dual V100 32GB SXM2 + NVLink Board): $850 – $1,150 USD
Chassis, Server Fans & 1600W Server PSU: $500 – $800 USD
System Memory (Used 64GB DDR4 ECC Sticks @ approx. $170 – $250 each):
- Stage 1 (512GB RAM for 300B–400B MoEs): $1,360 – $2,000 USD
- Stage 2 (Full 1.5TB RAM for 2.8T Kimi K3): $4,080 – $6,000 USD
Total Build Estimates:
- Stage 1 Total (512GB Pool): approx. $3,510 – $5,150 USD
- Stage 2 Total (1.5TB Pool): approx. $6,230 – $9,150 USD
Hardware & Performance Metrics:
System Platform: Dual-Socket AMD SP3 (16-Channel DDR4) + PCIe GPU Offload
Total System RAM: 1,536 GB (1.536 TB) DDR4 ECC RDIMMs / LRDIMMs
Total GPU VRAM: 64 GB HBM2 (Dual Tesla V100 32GB SXM2 via NVLink)
System Memory Bandwidth: 300 – 340 GB/s usable real-world bandwidth
Maximum Native Context: 1,000,000 Tokens (via KDA Linear Attention)
Prompt Pre-fill Speed: 800 – 1,400 tokens/sec (GPU-accelerated)
10,000 Token Prompt Load Delay: approx. 8 – 12 seconds
Decode Speed (Token Stream): 3 – 6 tokens/sec
Idle System Power: approx. 250W
Peak Load Power: approx. 850W – 1,100W
Recommended PSU: 1600W 80+ Platinum / Titanium Server PSU
Note: This post outlines an architectural methodology and concept—it is not physically proof-built for Kimi K3 yet. I am currently in the process of sourcing and assembling this system to test, benchmark, and validate everything in the real world. My goal is to iron out the hardware/software pipeline and release a fully verified, hands-on build guide to give back to the open-source and decentralized AI community. Take whatever math or ideas are useful to you here, and stay tuned for the physical build logs! I'll update this post with the guide TBA.
bro lol
goated
This is a hallucination. First, it's not possible to get that hardware anywhere near that cheaply. Second, it'd be very slow, likely entirely unusable for a model of this size.
In which universe are you living ? Currently 32gb of DDR5 RDIMM costs you around 1k at least. so for 1.5TB ~ 50k.
What are you daydreaming about? Spitting out words one by one is already pretty fast.
keen to see tokens/sec on the hardware you described
I created an account just to mention. 128gb ddr5 in aliexpress second hand is arround ~2k usd. and DDR4 is ~1.8k USD. So Yeah, impossible
I have refined, replaced and improved the information. I'm sure you can get use something from this quick rough info dump.
That's a great idea, and I'm already putting it into practice. Given the exorbitant price of DDR5 server memory, I settled on the final generation DDR4 server platform as a more cost-effective alternative.
- Hardware Selection & Costs
I selected the Intel C621A chipset paired with 3rd Gen Intel® Xeon® Scalable Processors. Thanks to the abundance of ES (Engineering Sample) server CPUs on the Chinese second-hand market—and the fact that 3rd Gen Xeon ES versions are known for having relatively few bugs—I can easily acquire an 8368 ES (38 cores, 76 threads, Ice Lake architecture, Stepping 6—same as the retail version) for just $100–150. Matching DDR4 16GB 3200 memory modules cost about $90 per stick.
Adding a decommissioned 1U server chassis (including the full set of power supplies, coolers, and motherboard) comes to around $400.
All in all, the total cost to assemble a single dual-socket server is approximately $2,200. - Networking & Cluster Architecture
For networking, I'm using Mellanox CX5 series cards with MCP1600 100G cables to attempt an InfiniBand interconnect.
The total investment for building two dual-socket servers with this InfiniBand setup is around $5,000.
The entire cluster possesses approximately 512GB of DDR4 memory, with a theoretical concurrent bandwidth of 800G (effectively around 600G). - Software Stack Comparison
Regarding the software stack, I evaluated two main approaches:
Option 1: Standard, Commercial-Grade Server Frameworks
This involves vLLM and SGLang. There are some enhanced forks on GitHub tailored for personal deployment, such as lvllm, which are extensions of Tsinghua University's KTransformers framework.
Cons: Most of these inference backends do not support GGUF and rely on standard w4a16 computation, leading to excessive memory requirements. Furthermore, they basically don't support pure CPU inference, and the performance is mediocre.
Benchmarks: With a dual-socket 8368 setup and two RTX 3090s, the generation speed (TG) is barely above 10 tokens/s（Dspark Enable will be ≈30toks）.
Option 2: Heavily CPU-Optimized llama.cpp Fork (My Current Focus)
I had Kimi-k3 review the code and discovered that the original version missed many optimization opportunities. So, I rewrote significant portions:
Optimizations: Added repack support for low-quantization operators; fixed the compute path to properly utilize AVX-512; enabled newer acceleration instructions like VNNI and VBMI; implemented operator fusion and barrier optimizations; and added support for weight/KV mirroring as well as CPU EP.
Benchmarks (DeepSeek V4 Flash @ Q2):
Pure CPU: Prefill (PP) speed is approx. 50 tokens/s; Generation (TG) speed is approx. 15 tokens/s.
With Dual 3090s: Generation (TG) speed reaches approx. 30+ tokens/s. - Current Progress & Goals
Currently, I am working hard to implement Tensor Parallelism within llama.cpp.
Right now, only one RTX 3090 is handling the main compute acceleration, while the second card is solely dedicated to MoE computation. If I can successfully offload the Dense layers, Router, and Attention mechanisms of such a massive MoE model across the GPUs, I expect to see a significant boost in actual performance.
- Modified and translate by LLM, because im a Chinese. I promise the data is TRUE and NOT the AI Hallucinations.
Let me know where you're getting hardware at that price 😭
Let me know where you're getting hardware at that price 😭
China's biggist second-hand online market Xianyu& ebay actually
@heiketu Huge thanks to you for your expansion on this theory/methodology with real data! and even photos! I feel like I did something here 😄! I am usually always working in the stealth background on these things, alone. I was just compelled to share something and tell everyone it is not hopeless! personally my first success in the deep planing and iterative process of making local & powerful AI possible at home (risky as it was as I have less money than average). Was with my 4xV100 NVlinked 128GB HBM2 system! I have spend weeks to months, but more and more heavily focused the last few weeks setting up my first first absolutely incredible system that was less than half the price of 2x RTX5090's for double the vram and when patched is way more performant, especially memory bandwidth wise, not to mention possesses real ECC memory and enterprise reliability AND the real potential to TRAIN your own models as you cannot reliably do with RTX cards.
Let's keep this going, I will try to post some photos and plan some more when I have finally got some sleep! it is now 6am here...
@heiketu Huge thanks to you for your expansion on this theory/methodology with real data! and even photos! I feel like I did something here 😄! I am usually always working in the stealth background on these things, alone. I was just compelled to share something and tell everyone it is not hopeless! personally my first success in the deep planing and iterative process of making local & powerful AI possible at home (risky as it was as I have less money than average). Was with my 4xV100 NVlinked 128GB HBM2 system! I have spend weeks to months, but more and more heavily focused the last few weeks setting up my first first absolutely incredible system that was less than half the price of 2x RTX5090's for double the vram and when patched is way more performant, especially memory bandwidth wise, not to mention possesses real ECC memory and enterprise reliability AND the real potential to TRAIN your own models as you cannot reliably do with RTX cards.
Let's keep this going, I will try to post some photos and plan some more when I have finally got some sleep! it is now 6am here...
lets go, i will continue follow your process!
The V100 does seem viable. Maybe you could snag a few more V100s on Xianyu. But honestly, the AI performance of the V100 was never anything to write home about, and on top of that there's no NVLink... Still, just getting it running is already pretty impressive. I think pairing it with something like MTP might make it a bit more usable, though.
V100 performance is stallar. 80B MoE model prefills at over 10,000 tk/s with attention patch in vllm. Code generation speed is over 55 Tk/s.
Read the rough guide above. NVLink is fully supported with extabal backplane. Slow PCI-E is only for uploading weights.
I will be sure to include the quad & dual v100 setup in my ultimate guide on GitHub. Unlike anyone else however, I will bridge the software side of things. I already have my next target in the GPU arena, this new target is 10x better than even a v100 setup. I will post some photos/renders soon on here for my ultimate dual system water cooled AI micro compute server/workstation. Enclosed in a fully CNC'd acrylic box etc.
I will potentially create a new spin/release based on Rocky Linux, pre-built and configured for a perfect turn-key setup.
'Intelligence is not a privilege, it's a gift. Keep it stored up inside and it will make you sick'