dreambooth

Runtime error

CUDA out of memory

by duja1 - opened Dec 9, 2022

Dec 9, 2022

Trying to train on 15 images for 2400 steps on factory rebooted T4 medium i get this error after Caching latents. Am I doing something wrong?

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 11.40 GiB already allocated; 327.75 MiB free; 13.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

YESO

Dec 10, 2022

LLEVA ENTRENANDO MAS DE 20 MINUTOS... ES NORMAL?

sreerama

Mirage org Dec 11, 2022

@YESO Training for more than 20 minutes is normal, training time is determined by the number of images you upload or you can configure custom training time under Custom Settings

sreerama

Mirage org Dec 11, 2022

@duja1 Might have to upgrade to a larger instance, will investigate further if there's another solution

kanik155

Dec 12, 2022

edit: Can also confirm that changing git+https://github.com/huggingface/diffusers.git to diffusers==0.10.0 works.

https://huggingface.co/spaces/multimodalart/dreambooth-training/discussions/18

sreerama changed discussion status to closed Dec 13, 2022

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment