Run SFT using PEFT on single a10g with 24G mem, got out of mem problem
#35
by
Andcircle
- opened
Trying to run SFT using PEFT as here https://gist.github.com/pacman100/1731b41f7a90a87b457e8c5415ff1c14
If I set device_map to {"":0}, will got out of mem problem, have to set device_map to "auto", which will use all 4 GPU, then it works.
But according to the instruction, it should work with 16G mem, isn't it?
I use a cluster with 4 a10g 24G
CUDA version 12.0
torch version 2.0.1-cu118