|
--- |
|
language: |
|
- en |
|
pipeline_tag: image-to-image |
|
tags: |
|
- Diffusion Models |
|
- Stable Diffusion |
|
- Perturbed-Attention Guidance |
|
- PAG |
|
--- |
|
|
|
# Inpainting with Perturbed-Attention Guidance |
|
|
|
[Project](https://ku-cvlab.github.io/Perturbed-Attention-Guidance/) / [arXiv](https://arxiv.org/abs/2403.17377) / [GitHub](https://github.com/KU-CVLAB/Perturbed-Attention-Guidance) |
|
|
|
This repository is based on [Diffusers](https://huggingface.co/docs/diffusers/index). |
|
|
|
The pipeline is a modification of StableDiffusionPipeline to support inpainting with Perturbed-Attention Guidance (PAG). Please refer to "Inpainting" section of an [official document](https://huggingface.co/docs/diffusers/using-diffusers/inpaint) for details. |
|
|
|
## Quickstart |
|
|
|
Loading Custom Piepline: |
|
|
|
``` |
|
from diffusers import StableDiffusionPipeline |
|
|
|
pipe = StableDiffusionPipeline.from_pretrained( |
|
"runwayml/stable-diffusion-inpainting", |
|
custom_pipeline="hyoungwoncho/sd_perturbed_attention_guidance_inpaint", |
|
torch_dtype=torch.float16, |
|
safety_checker=None |
|
) |
|
|
|
device="cuda" |
|
pipe = pipe.to(device) |
|
``` |
|
|
|
Inpainting with PAG: |
|
|
|
``` |
|
output = pipe( |
|
prompts, |
|
image=init_image, |
|
mask_image=mask_image, |
|
num_inference_steps=50, |
|
guidance_scale=0.0, |
|
pag_scale=3.0, |
|
pag_applied_layers_index=['u0'] |
|
).images[0] |
|
``` |
|
|
|
## Parameters |
|
|
|
guidance_scale : gudiance scale of CFG (ex: 7.5) |
|
|
|
pag_scale : gudiance scale of PAG (ex: 3.0) |
|
|
|
pag_applied_layers_index : index of the layer to apply perturbation (ex: ['u0']) |