fine-tuning stable-diffusion-inpainting model
I saw your comment yesterday, did you succeed on fine-tuning the inpaint model ?
I'm also curious about this. I have been trying for a while with Dreambooth, but can't find how to make it work. Is the architecture the same as Stable Diffusion? In that case, one could simply fine-tune the desired SD model and replace the weights in the respective folder of the inpainter, right? (But Idk which folder it would be)
No I have not but I think its a matter adding the mask channels to the inputs.
In addition to adjusting the Unet (i.e. changing the number of input channels), to get know to finetune the inpainting model, also need to find out how the 'synthetic masks' were created during training - whether the authors did this using segmentation or whether the images were masked randomly.
Has anyone had any luck with this? Very keen to explore.
yes I was able to. See --> https://github.com/huggingface/diffusers/tree/main/examples/research_projects/dreambooth_inpaint
yes I was able to. See --> https://github.com/huggingface/diffusers/tree/main/examples/research_projects/dreambooth_inpaint
Hello,
Have you tested the results of fine-tuning? Are they better than the original version?
yes
Great. How much data did you use for fine-tuning? Is 50000 images enough?
yes
I used Dreambooth so it was a small sample size (< 50)
I'd love to be able to write images every N steps to Tensorboard or Weights and Biases to see if the model is getting better or not (so I can iterate faster). Does anyone have docs on how to do this?
Maybe this Notebook could help: https://github.com/davide97l/stable_diffusion_dreambooth_inpainting/blob/main/stable_diffusion_dreambooth_inpaint.ipynb