Having a blast
#1
by chris-hc - opened
I'm having a blast with this model tonight. I quantized the weights to nvfp4 and it seems smart enough, but I can't run full weights on my 16gb 5080.
I did run into an accumulator error, where after so many chunks I would oom because vram reclaim only occurs for new sessions.
Ah! Discovered the sliding window! No need for the memory guard hack.