Training inpainting discussion #33

by minhtri171997 - opened

As I know, may be it not correct yet. I currently having an issue as:

  • I want to train some specific style of image ( my custom style ) using DreamBooth on stable-diffusion model runwayml/stable-diffusion-v1-5, it worked, but the issue here is i want to use this model "stabilityai/stable-diffusion-2-inpainting" to train custom styles as stable-diffusion-v1-5. But I could not find the way to fine-tune that inpainting model. I have try to merged these models for my use case but there was difference in the param shape of text_model from inpainting is different from normal stable diffusion model.

image.png

Is there any solution to train new styles using inpainting model or is there a method to merge these two model for my purpose ?
If there is anything not clear yet, please feel free to leave a message i will clarify that
Thank you,

I made a quick LORA inpainting finetuning script for this https://github.com/sshh12/terrain-diffusion/blob/main/scripts/train_text_to_image_lora_sd2_inpaint.py, hope its helpful!

Sign up or log in to comment