⏬[**Download Models**](#-download-models) **|** 💻[**How to Test**](#-how-to-test)
Official implementation of T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.
#### [Paper](https://arxiv.org/abs/2302.08453)
We propose T2I-Adapter, a **simple and small (~70M parameters, ~300M storage space)** network that can provide extra guidance to pre-trained text-to-image models while **freezing** the original large text-to-image models.
T2I-Adapter aligns internal knowledge in T2I models with external control signals.
We can train various adapters according to different conditions, and achieve rich control and editing effects.
### ⏬ Download Models
Put the downloaded models in the `T2I-Adapter/models` folder.
1. The **T2I-Adapters** can be download from .
2. The pretrained **Stable Diffusion v1.4** models can be download from . You need to download the `sd-v1-4.ckpt
` file.
3. [Optional] If you want to use **Anything v4.0** models, you can download the pretrained models from . You need to download the `anything-v4.0-pruned.ckpt` file.
4. The pretrained **clip-vit-large-patch14** folder can be download from . Remember to download the whole folder!
5. The pretrained keypose detection models include FasterRCNN (human detection) from and HRNet (pose detection) from .
After downloading, the folder structure should be like this:
### 🔧 Dependencies and Installation
- Python >= 3.6 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
- [PyTorch >= 1.4](https://pytorch.org/)
```bash
pip install -r requirements.txt
```
- If you want to use the full function of keypose-guided generation, you need to install MMPose. For details please refer to .
### 💻 How to Test
- The results are in the `experiments` folder.
- If you want to use the `Anything v4.0`, please add `--ckpt models/anything-v4.0-pruned.ckpt` in the following commands.
#### **For Simple Experience**
> python app.py
#### **Sketch Adapter**
- Sketch to Image Generation
> python test_sketch.py --plms --auto_resume --prompt "A car with flying wings" --path_cond examples/sketch/car.png --ckpt models/sd-v1-4.ckpt --type_in sketch
- Image to Image Generation
> python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/sd-v1-4.ckpt --type_in image
- Generation with **Anything** setting
> python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image
##### Gradio Demo
You can use gradio to experience all these three functions at once. CPU is also supported by setting device to 'cpu'.
```bash
python gradio_sketch.py
```
#### **Keypose Adapter**
- Keypose to Image Generation
> python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/keypose/iron.png --type_in pose
- Image to Image Generation
> python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --type_in image
- Generation with **Anything** setting
> python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image
##### Gradio Demo
You can use gradio to experience all these three functions at once. CPU is also supported by setting device to 'cpu'.
```bash
python gradio_keypose.py
```
#### **Segmentation Adapter**
> python test_seg.py --plms --auto_resume --prompt "A black Honda motorcycle parked in front of a garage" --path_cond examples/seg/motor.png
#### **Two adapters: Segmentation and Sketch Adapters**
> python test_seg_sketch.py --plms --auto_resume --prompt "An all white kitchen with an electric stovetop" --path_cond examples/seg_sketch/mask.png --path_cond2 examples/seg_sketch/edge.png
#### **Local editing with adapters**
> python test_sketch_edit.py --plms --auto_resume --prompt "A white cat" --path_cond examples/edit_cat/edge_2.png --path_x0 examples/edit_cat/im.png --path_mask examples/edit_cat/mask.png
## Stable Diffusion + T2I-Adapters (only ~70M parameters, ~300M storage space)
The following is the detailed structure of a **Stable Diffusion** model with the **T2I-Adapter**.
## 🚀 Interesting Applications
### Stable Diffusion results guided with the sketch T2I-Adapter
The corresponding edge maps are predicted by PiDiNet. The sketch T2I-Adapter can well generalize to other similar sketch types, for example, sketches from the Internet and user scribbles.
### Stable Diffusion results guided with the keypose T2I-Adapter
The keypose results predicted by the [MMPose](https://github.com/open-mmlab/mmpose).
With the keypose guidance, the keypose T2I-Adapter can also help to generate animals with the same keypose, for example, pandas and tigers.
### T2I-Adapter with Anything-v4.0
Once the T2I-Adapter is trained, it can act as a **plug-and-play module** and can be seamlessly integrated into the finetuned diffusion models **without re-training**, for example, Anything-4.0.
#### ✨ Anything results with the plug-and-play sketch T2I-Adapter (no extra training)
#### Anything results with the plug-and-play keypose T2I-Adapter (no extra training)
### Local editing with the sketch adapter
When combined with the inpaiting mode of Stable Diffusion, we can realize local editing with user specific guidance.
#### ✨ Change the head direction of the cat
#### ✨ Add rabbit ears on the head of the Iron Man.
### Combine different concepts with adapter
Adapter can be used to enhance the SD ability to combine different concepts.
#### ✨ A car with flying wings. / A doll in the shape of letter ‘A’.
### Sequential editing with the sketch adapter
We can realize the sequential editing with the adapter guidance.
### Composable Guidance with multiple adapters
Stable Diffusion results guided with the segmentation and sketch adapters together.
Logo materials: [adapter](https://www.flaticon.com/free-icon/adapter_4777242), [lightbulb](https://www.flaticon.com/free-icon/lightbulb_3176369)