RePaint
RePaint: Inpainting using Denoising Diffusion Probabilistic Models is by Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, Luc Van Gool.
The abstract from the paper is:
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior. To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image information. Since this technique does not modify or condition the original DDPM network itself, the model produces high-quality and diverse output images for any inpainting form. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks. RePaint outperforms state-of-the-art Autoregressive, and GAN approaches for at least five out of six mask distributions.
The original codebase can be found at andreas128/RePaint.
Make sure to check out the Schedulers guide to learn how to explore the tradeoff between scheduler speed and quality, and see the reuse components across pipelines section to learn how to efficiently load the same components into multiple pipelines.
RePaintPipeline
class diffusers.RePaintPipeline
< source >( unet scheduler )
Parameters
-
unet (UNet2DModel) —
A
UNet2DModelto denoise the encoded image latents. -
scheduler (RePaintScheduler) —
A
RePaintSchedulerto be used in combination withunetto denoise the encoded image.
Pipeline for image inpainting using RePaint.
This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).
__call__
< source >(
image: typing.Union[torch.Tensor, PIL.Image.Image]
mask_image: typing.Union[torch.Tensor, PIL.Image.Image]
num_inference_steps: int = 250
eta: float = 0.0
jump_length: int = 10
jump_n_sample: int = 10
generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None
output_type: typing.Optional[str] = 'pil'
return_dict: bool = True
)
→
ImagePipelineOutput or tuple
Parameters
-
image (
torch.FloatTensororPIL.Image.Image) — The original image to inpaint on. -
mask_image (
torch.FloatTensororPIL.Image.Image) — The mask_image where 0.0 define which part of the original image to inpaint. -
num_inference_steps (
int, optional, defaults to 1000) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
eta (
float) — The weight of the added noise in a diffusion step. Its value is between 0.0 and 1.0; 0.0 corresponds to DDIM and 1.0 is the DDPM scheduler. -
jump_length (
int, optional, defaults to 10) — The number of steps taken forward in time before going backward in time for a single jump (“j” in RePaint paper). Take a look at Figure 9 and 10 in the paper. -
jump_n_sample (
int, optional, defaults to 10) — The number of times to make a forward time jump for a given chosen time sample. Take a look at Figure 9 and 10 in the paper. -
generator (
torch.Generator, optional) — Atorch.Generatorto make generation deterministic. -
output_type (
str,optional, defaults to"pil") — The output format of the generated image. Choose betweenPIL.Imageornp.array. -
return_dict (
bool, optional, defaults toTrue) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
Returns
ImagePipelineOutput or tuple
If return_dict is True, ImagePipelineOutput is returned, otherwise a tuple is
returned where the first element is a list with the generated images.
The call function to the pipeline for generation.
Example:
>>> from io import BytesIO
>>> import torch
>>> import PIL
>>> import requests
>>> from diffusers import RePaintPipeline, RePaintScheduler
>>> def download_image(url):
... response = requests.get(url)
... return PIL.Image.open(BytesIO(response.content)).convert("RGB")
>>> img_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/repaint/celeba_hq_256.png"
>>> mask_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/repaint/mask_256.png"
>>> # Load the original image and the mask as PIL images
>>> original_image = download_image(img_url).resize((256, 256))
>>> mask_image = download_image(mask_url).resize((256, 256))
>>> # Load the RePaint scheduler and pipeline based on a pretrained DDPM model
>>> scheduler = RePaintScheduler.from_pretrained("google/ddpm-ema-celebahq-256")
>>> pipe = RePaintPipeline.from_pretrained("google/ddpm-ema-celebahq-256", scheduler=scheduler)
>>> pipe = pipe.to("cuda")
>>> generator = torch.Generator(device="cuda").manual_seed(0)
>>> output = pipe(
... image=original_image,
... mask_image=mask_image,
... num_inference_steps=250,
... eta=0.0,
... jump_length=10,
... jump_n_sample=10,
... generator=generator,
... )
>>> inpainted_image = output.images[0]ImagePipelineOutput
class diffusers.ImagePipelineOutput
< source >( images: typing.Union[typing.List[PIL.Image.Image], numpy.ndarray] )
Output class for image pipelines.