Users running this at their house? If so can you sharw your setup?
Any users running this at their house? If so can you share your setup? Perhaps even power consumption if you have it.I bet with 3 GPUs and a system with 1.5tb of system memory could run this with the gguf version primarily with CPU. It would likely be 0.8-2.0 tokens per second depending on cpu memory speeds.
With 3 GPUs you can run a 70b model not a 3T (3000B) model.
what's the point of using it in long-term? ttft's gonna be days with 1.5tb of sysmem
If you got enough fast NVMe storage and about 128gb system RAM you could load just the required experts to RAM for each token (like how colibri works). It would take some serious software setup though, and don't count on getting more than maybe 1tps on average.
I'm using an old gaming laptop with 2tb NVMe storage and 64gb DDR4 RAM connected via ethernet to an ebay server with 128gb DDR4 RAM. Planning on running persistent layers in the laptop, caching whatever extra experts I can there in the overhead, and fitting the active experts into the server's RAM pool.
No clue if it'll work or how slow it would be if it does, but I'm gonna try to get this model working with whatever I got XD
If you got enough fast NVMe storage and about 128gb system RAM you could load just the required experts to RAM for each token (like how colibri works). It would take some serious software setup though, and don't count on getting more than maybe 1tps on average.
Yeah there's basically no point. With NVMe streaming the reasoning overhead will make the model unusable at those agonising speeds and turning off reasoning will make the model significantly less smarter.
As others mentioned TTFT would be horrible due to prompt processing and model load. With local hosting, even when a model fully fits into VRAM many people struggle with prompt processing which takes much longer on a single system compared to over cloud, and for a model of this size streaming over disk your prompt processing would probably hover around the same as your agonising decode speeds basically rendering it unusable for any agentic tasks (which frequently perform bulk reads that would get crippled by the PP speed). Maybe considerable if you dont mind waiting hour(s) for one response
Running this from my lake house, atop my data lake at the data center, hoping they don't find the wires I snuck in and ran to my tent up here on the roof, but so far so good!
Trying to use system memory will be insanely slow. There's only one viable path to running this at home, or an office. 16x Nvidia DGX Sparks. That would cost a little over $100k currently, but prices continue to go up, and it'd use around up to 2,500 watts of power when properly set up since it's more efficient than conventional GPUs. You need the 16 Sparks, ideally the 4 TB of storage ones but the 2 TB would barely work, a suitable enterprise switch for the networking, and suitable DAC cables. There are many options for the switch and the cables depend on the switch being used. Anyone reading this should know, this can be done, but it's significantly harder than building a gaming computer. This uses new architecture, so there will also be various required software changes and performance enhancements being worked on by the open source community over the coming weeks, some required, some all but required. It's very nice to see the model released though, thank you Moonshot and the CCP for your choice to throw money at this, even though the coming bubble pop will have no winners!