File size: 1,910 Bytes
b27d22d
 
 
 
 
b8f0ab3
b27d22d
 
 
 
 
bc0a907
9085f30
bc0a907
b27d22d
56e9f19
c27a688
f549903
b27d22d
 
 
f549903
b27d22d
 
70d427f
 
b27d22d
 
 
5193ae7
 
b27d22d
70d427f
 
 
5193ae7
 
b27d22d
 
f3e3fe8
b27d22d
 
70d427f
b27d22d
 
 
 
 
c9b62a3
a81fa9d
b27d22d
 
 
f3e3fe8
 
 
 
 
 
 
 
 
 
 
 
 
 
bc0a907
b27d22d
8ac8c9e
1bb7127
aabeddd
58d4985
866f866
80e27cc
 
 
3790c8f
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
---
language:
- en
pipeline_tag: unconditional-image-generation
tags:
- Diffusion Models
- Stable Diffusion
- Perturbed-Attention Guidance
- PAG
---

# Perturbed-Attention Guidance

![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/6601282b569b30694e67b886/27Lmuol8anwd6L6BLzyWf.jpeg)

[Project](https://ku-cvlab.github.io/Perturbed-Attention-Guidance/) / [arXiv](https://arxiv.org/abs/2403.17377) / [GitHub](https://github.com/KU-CVLAB/Perturbed-Attention-Guidance)

This repository is based on [Diffusers](https://huggingface.co/docs/diffusers/index). The pipeline is a modification of StableDiffusionPipeline to support Perturbed-Attention Guidance (PAG).

## Quickstart

Loading Custom Piepline:

```
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    custom_pipeline="hyoungwoncho/sd_perturbed_attention_guidance",
    torch_dtype=torch.float16,
    safety_checker=None
)

device="cuda"
pipe = pipe.to(device)

prompts = ["a corgi"]
```

Sampling with PAG:

```
output = pipe(
        prompts,
        width=512,
        height=512,
        num_inference_steps=50,
        guidance_scale=0.0,
        pag_scale=5.0,
        pag_applied_layers_index=['m0']
    ).images
```

Sampling with PAG and CFG:

```
output = pipe(
        prompts,
        width=512,
        height=512,
        num_inference_steps=50,
        guidance_scale=4.0,
        pag_scale=3.0,
        pag_applied_layers_index=['m0']
    ).images
```

## Parameters

guidance_scale : gudiance scale of CFG (ex: 7.5)

pag_scale : gudiance scale of PAG (ex: 5.0)

pag_applied_layers_index : index of the layer to apply perturbation (ex: ['m0'])

## Stable Diffusion Demo

To join a demo of PAG on Stable Diffusion, run [sd_pag_demo.ipynb](https://huggingface.co/hyoungwoncho/sd_perturbed_attention_guidance/blob/main/sd_pag_demo.ipynb).