RC2 is now too large for RTX 6000
More of a sadness than anything, I get that you need to fix the looping issues, but these updated weights mean that we can't fit the weights in RTX 6000, 2xL40s, 2xA100(40GB) with the updated weights.
Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.
Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.
Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?
Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.
Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?
I used their guff version that is 90GB
Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.
Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?
Try to use this also:
https://huggingface.co/anoane/Laguna-S-2.1-NVFP4-RTX6000PRO-vLLM
i'm trying this, it might be more helpfull
#dgxspark
Too large for Dgx Spark too, I guess it's almost all are in BF16, I tried 160k but got a OOM, I can get 120k in context I think but I didn't try, I know it gonna be too slow to try something...