chengjiaxiang commited on
Commit
e8a6647
1 Parent(s): 08a5fa2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +129 -1
README.md CHANGED
@@ -6,4 +6,132 @@ library_name: diffusers
6
  pipeline_tag: text-to-image
7
  cite: arxiv.org/abs/2403.02084
8
  ---
9
- Github repository: https://github.com/bytedance/res-adapter
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  pipeline_tag: text-to-image
7
  cite: arxiv.org/abs/2403.02084
8
  ---
9
+ # ResAdapter Model Card
10
+
11
+ <div align="center">
12
+
13
+ [**Project Page**](https://res-adapter.github.io/) **|** [**Paper**](https://arxiv.org/abs/2403.02084) **|** [**Code**](https://github.com/bytedance/res-adapter) **|** [ **Gradio demo**](https://huggingface.co/spaces/ameerazam08/Res-Adapter-GPU-Demo) **|** [**ComfyUI Extension**](https://github.com/jiaxiangc/ComfyUI-ResAdapter)
14
+
15
+
16
+ </div>
17
+
18
+ ## Introduction
19
+
20
+ We propose ResAdapter, a plug-and-play resolution adapter for enabling any diffusion model generate resolution-free images: no additional training, no additional inference and no style transfer.
21
+
22
+
23
+ <div align="center">
24
+ <img src='https://github.com/bytedance/res-adapter/blob/main/assets/misc/dreamlike1.png?raw=true' width='49.9%'><img src='https://github.com/bytedance/res-adapter/blob/main/assets/misc/dreamlike2.png?raw=true' width='49.9%'>
25
+ </div>
26
+
27
+
28
+ ## Usage
29
+
30
+ We provide a standalone [example code](https://github.com/bytedance/res-adapter/blob/main/quicktour.py) to help you quickly use resadapter with diffusion models.
31
+
32
+ <img src="https://github.com/bytedance/res-adapter/blob/main/assets/misc/dreamshaper_resadapter.png?raw=true" width="100%">
33
+ <img src="https://github.com/bytedance/res-adapter/blob/main/assets/misc/dreamshaper_baseline.png?raw=true" width="100%">
34
+
35
+ Comparison examples (640x384) between resadapter and [dreamshaper-xl-1.0](https://huggingface.co/Lykon/dreamshaper-xl-1-0). Top: with resadapter. Bottom: without resadapter.
36
+
37
+
38
+ ```python
39
+ # pip install diffusers, transformers, accelerate, safetensors, huggingface_hub
40
+ import torch
41
+ from torchvision.utils import save_image
42
+ from safetensors.torch import load_file
43
+ from huggingface_hub import hf_hub_download
44
+ from diffusers import AutoPipelineForText2Image, DPMSolverMultistepScheduler
45
+
46
+ generator = torch.manual_seed(0)
47
+ prompt = "portrait photo of muscular bearded guy in a worn mech suit, light bokeh, intricate, steel metal, elegant, sharp focus, soft lighting, vibrant colors"
48
+ width, height = 640, 384
49
+
50
+ # Load baseline pipe
51
+ model_name = "lykon-models/dreamshaper-xl-1-0"
52
+ pipe = AutoPipelineForText2Image.from_pretrained(model_name, torch_dtype=torch.float16, variant="fp16").to("cuda")
53
+ pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True, algorithm_type="sde-dpmsolver++")
54
+
55
+ # Inference baseline pipe
56
+ image = pipe(prompt, width=width, height=height, num_inference_steps=25, num_images_per_prompt=4, output_type="pt").images
57
+ save_image(image, f"image_baseline.png", normalize=True, padding=0)
58
+
59
+ # Load resadapter for baseline
60
+ resadapter_model_name = "resadapter_v1_sdxl"
61
+ pipe.load_lora_weights(
62
+ hf_hub_download(repo_id="jiaxiangc/res-adapter", subfolder=resadapter_model_name, filename="pytorch_lora_weights.safetensors"),
63
+ adapter_name="res_adapter",
64
+ ) # load lora weights
65
+ pipe.set_adapters(["res_adapter"], adapter_weights=[1.0])
66
+ pipe.unet.load_state_dict(
67
+ load_file(hf_hub_download(repo_id="jiaxiangc/res-adapter", subfolder=resadapter_model_name, filename="diffusion_pytorch_model.safetensors")),
68
+ strict=False,
69
+ ) # load norm weights
70
+
71
+ # Inference resadapter pipe
72
+ image = pipe(prompt, width=width, height=height, num_inference_steps=25, num_images_per_prompt=4, output_type="pt").images
73
+ save_image(image, f"image_resadapter.png", normalize=True, padding=0)
74
+ ```
75
+
76
+ For more details, please follow the instructions in our [GitHub repository](https://github.com/bytedance/res-adapter).
77
+
78
+ ## Models
79
+ |Models | Parameters | Resolution Range | Ratio Range | Links |
80
+ | --- | --- |--- | --- | --- |
81
+ |resadapter_v2_sd1.5| 0.9M | 128 <= x <= 1024 | 0.28 <= r <= 3.5 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v2_sd1.5)|
82
+ |resadapter_v2_sdxl| 0.5M | 256 <= x <= 1536 | 0.28 <= r <= 3.5 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v2_sdxl)|
83
+ |resadapter_v1_sd1.5| 0.9M | 128 <= x <= 1024 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sd1.5)|
84
+ |resadapter_v1_sd1.5_extrapolation| 0.9M | 512 <= x <= 1024 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sd1.5_extrapolation)|
85
+ |resadapter_v1_sd1.5_interpolation| 0.9M | 128 <= x <= 512 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sd1.5_interpolation)|
86
+ |resadapter_v1_sdxl| 0.5M | 256 <= x <= 1536 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sdxl) |
87
+ |resadapter_v1_sdxl_extrapolation| 0.5M | 1024 <= x <= 1536 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sdxl_extrapolation) |
88
+ |resadapter_v1_sdxl_interpolation| 0.5M | 256 <= x <= 1024 | 0.5 <= r <= 2 | [Download](https://huggingface.co/jiaxiangc/res-adapter/tree/main/resadapter_v1_sdxl_interpolation) |
89
+
90
+ ## ComfyUI Extension
91
+
92
+ ### Text-to-Image
93
+
94
+ - workflow: [resadapter_text_to_image_workflow](https://github.com/jiaxiangc/ComfyUI-ResAdapter/blob/main/examples/resadapter_text_to_image_workflow.json).
95
+ - models: [dreamlike-diffusion-1.0](https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0)
96
+
97
+ https://github.com/jiaxiangc/ComfyUI-ResAdapter/assets/162297627/82453931-23de-4f72-8a9c-1053c4c8d81a
98
+
99
+ ### ControlNet
100
+
101
+ - workflow: [resadapter_controlnet_workflow](https://github.com/jiaxiangc/ComfyUI-ResAdapter/blob/main/examples/resadapter_controlnet_workflow.json).
102
+ - models: [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [sd-controlnet-canny](https://huggingface.co/lllyasviel/sd-controlnet-canny)
103
+
104
+ https://github.com/jiaxiangc/ComfyUI-ResAdapter/assets/162297627/aab14d91-f68f-4415-b947-fd99662ca2ca
105
+
106
+ ### IPAdapter
107
+
108
+ - workflow: [resadapter_ipadapter_workflow](https://github.com/jiaxiangc/ComfyUI-ResAdapter/blob/main/examples/resadapter_ipadapter_workflow.json).
109
+ - models: [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [IP-Adapter](https://huggingface.co/h94/IP-Adapter)
110
+
111
+ https://github.com/jiaxiangc/ComfyUI-ResAdapter/assets/162297627/92bcfc11-b1d5-4909-9e6d-7e1978c948c2
112
+
113
+ ### Accelerate LoRA
114
+
115
+ - workflow: [resadapter_accelerate_lora_workflow](https://github.com/jiaxiangc/ComfyUI-ResAdapter/blob/main/examples/resadapter_accelerate_lora_workflow.json).
116
+ - models: [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [lcm-lora-sdv1-5](https://huggingface.co/latent-consistency/lcm-lora-sdv1-5)
117
+
118
+ https://github.com/jiaxiangc/ComfyUI-ResAdapter/assets/162297627/46ce4224-ab4d-4890-abc8-00b716dcb29a
119
+
120
+ ## Usage Tips
121
+
122
+ 1. If you are not satisfied with interpolation images, try to increase the alpha of resadapter to 1.0.
123
+ 2. If you are not satisfied with extrapolate images, try to choose the alpha of resadapter in 0.3 ~ 0.7.
124
+ 3. If you find the images with style conflicts, try to decrease the alpha of resadapter.
125
+ 4. If you find resadapter is not compatible with other accelerate lora, try to decrease the alpha of resadapter to 0.5 ~ 0.7.
126
+
127
+ ## Citation
128
+ If you find ResAdapter useful for your research and applications, please cite us using this BibTeX:
129
+ ```
130
+ @article{cheng2024resadapter,
131
+ title={ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models},
132
+ author={Cheng, Jiaxiang and Xie, Pan and Xia, Xin and Li, Jiashi and Wu, Jie and Ren, Yuxi and Li, Huixia and Xiao, Xuefeng and Zheng, Min and Fu, Lean},
133
+ booktitle={arXiv preprint arxiv:2403.02084},
134
+ year={2024}
135
+ }
136
+ ```
137
+ For any question, please feel free to contact us via chengjiaxiang@bytedance.com or xiepan.01@bytedance.com.