hyoungwoncho/sd_perturbed_attention_guidance_controlnet

Super-Resolution with Perturbed-Attention Guidance

Project / arXiv / GitHub

This repository is based on Diffusers.

ControlNet is a neural network structure to control diffusion models by adding extra conditions. The pipeline is a modification of StableDiffusionControlNetPipeline to support image generation with ControlNet and Perturbed-Attention Guidance (PAG).

In addition to the examples provided with Openpose, you can also generate using various conditions. Please refer to "ControlNet" section of an official document for details.

Quickstart

Loading ControlNet and Custom Piepline:

from diffusers import ControlNetModel, StableDiffusionControlNetPipeline

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-openpose",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    custom_pipeline="hyoungwoncho/sd_perturbed_attention_guidance_controlnet",
    controlnet=controlnet,
    torch_dtype=torch.float16
)

device="cuda"
pipe = pipe.to(device)

Prepare Conditional Images:

from controlnet_aux import OpenposeDetector

openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
original_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png"
)
openpose_image = openpose(original_image)

prompts=""

Conditional Generation with ControlNet and PAG:

output = pipe(
    prompts,
    image=openpose_image,
    num_inference_steps=50,
    guidance_scale=0.0,
    pag_scale=4.0,
    pag_applied_layers_index=["m0"]
).images[0]

Parameters

guidance_scale : gudiance scale of CFG (ex: 7.5)

pag_scale : gudiance scale of PAG (ex: 4.0)

pag_applied_layers_index : index of the layer to apply perturbation (ex: ['m0'])