fine-tuning stable-diffusion-inpainting model #9

by ncchadwi - opened
This comment has been hidden
ncchadwi changed discussion status to closed

I saw your comment yesterday, did you succeed on fine-tuning the inpaint model ?

I'm also curious about this. I have been trying for a while with Dreambooth, but can't find how to make it work. Is the architecture the same as Stable Diffusion? In that case, one could simply fine-tune the desired SD model and replace the weights in the respective folder of the inpainter, right? (But Idk which folder it would be)

No I have not but I think its a matter adding the mask channels to the inputs.

In addition to adjusting the Unet (i.e. changing the number of input channels), to get know to finetune the inpainting model, also need to find out how the 'synthetic masks' were created during training - whether the authors did this using segmentation or whether the images were masked randomly.

Has anyone had any luck with this? Very keen to explore.

yes I was able to. See -->

Have you tested the results of fine-tuning? Are they better than the original version?

Great. How much data did you use for fine-tuning? Is 50000 images enough?


I used Dreambooth so it was a small sample size (< 50)

I'd love to be able to write images every N steps to Tensorboard or Weights and Biases to see if the model is getting better or not (so I can iterate faster). Does anyone have docs on how to do this?

Sign up or log in to comment