CUDA out of memory #5

by duja1 - opened

Trying to train on 15 images for 2400 steps on factory rebooted T4 medium i get this error after Caching latents. Am I doing something wrong?

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 11.40 GiB already allocated; 327.75 MiB free; 13.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

LLEVA ENTRENANDO MAS DE 20 MINUTOS... ES NORMAL?

sreerama Mirage org

@YESO Training for more than 20 minutes is normal, training time is determined by the number of images you upload or you can configure custom training time under Custom Settings

sreerama Mirage org

@duja1 Might have to upgrade to a larger instance, will investigate further if there's another solution

sreerama changed discussion status to closed

Sign up or log in to comment