DeepSeek V4.1 Flash Lite
This model is much bigger then the orignal DSV4 Flash which makes it much harder to run locally, please release a smaller DSV-4.1-Flash lite or something that's the same size or smaller then the previous DS-V4-Flash.
I’ll second this. An update on if this is the new Flash line architecture (so just much larger models across the board) would be great.
Comparing to the old Flash is kind of disingenuous.
disingenuous
Make your own model with your own naming scheme with your own compute with your own infrastructure with your own research team with your own software engineers etc. then 🙄
So entitled
A part of this model (~203B?) are n-gram tensors and may be able to offload onto system RAM or even nvme drives.
A part of this model (~203B?) are n-gram tensors and may be able to offload onto system RAM or even nvme drives.
Still leaves way too many parameters needed to store in RAM.
I would really like to see a new Deepseek Lite, maybe a 20-40B MoE the average Joe can run.
Yep 200B of 550B are n-grams.
So it would probably work on 128Gb machines in 2bit quantization.
Can we get better community members? These seem to be retarded.
Agreed. At the same time, I look forward to seeing more models that are smaller and cheaper—even if their scores aren't quite as high—so we can use them for less demanding tasks.
Yep 200B of 550B are n-grams.
So it would probably work on 128Gb machines in 2bit quantization.
Did you really just say 2-bit quantization as if that was a valid way of running any of these models? Have you tried 2-bit quantization? It's garbage. The model's too large to run locally. You need to spend $100,000 to run it. It's too expensive. You need 310 gigabytes of VRAM to load the safe tensors, and then you can offload 197 gigabytes to SSD or RAM. That's kind of diabolical considering that DeepSeek Flash before could run on 168 gigabytes of VRAM. This is a Pro model disguised as a Flash model.
DeepSeek v4 Flash was already QAT 4 bit model, so if this is 550 - 200 = 350B QAT 4 bit model, you only need 196Gb of VRAM to run in full precision. One AMD AI MAX 495+ box that costs ~ 4000$ (not even close to 100000$)
Yep 200B of 550B are n-grams.
So it would probably work on 128Gb machines in 2bit quantization.
wrong it's actually 748B total (including ngram)
552B is excluding the ngram
the 200B ngram is added on top of the 550B.
See this reddit post
ok, then no chance if you don't have 512GB VRAM Mac Ultra
A part of this model (~203B?) are n-gram tensors and may be able to offload onto system RAM or even nvme drives.
Still leaves way too many parameters needed to store in RAM.
I would really like to see a new Deepseek Lite, maybe a 20-40B MoE the average Joe can run.
You can also store then on NVME instead of RAM, people tried it and it wasnt much of a problem ?
Also if you have near 300-400GB of VRAM I hope you have a lot of RAM as well.
yes the same techniques for serving work with less vram needed than total total weights published by them which are FP8 with some blending. See the layers where its located:
"Flash" refers to latency, not weight.