DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing
CVPR 2024
Kaiwen Zhang
Yifan Zhou
Xudong Xu
Xingang Pan✉
Bo Dai
✉Corresponding Author
## Web Demos
[](https://openxlab.org.cn/apps/detail/KaiwenZhang/DiffMorpher)
## Requirements
To install the requirements, run the following in your environment first:
```bash
pip install -r requirements.txt
```
To run the code with CUDA properly, you can comment out `torch` and `torchvision` in `requirement.txt`, and install the appropriate version of `torch` and `torchvision` according to the instructions on [PyTorch](https://pytorch.org/get-started/locally/).
You can also download the pretrained model *Stable Diffusion v2.1-base* from [Huggingface](https://huggingface.co/stabilityai/stable-diffusion-2-1-base), and specify the `model_path` to your local directory.
## Run Gradio UI
To start the Gradio UI of DiffMorpher, run the following in your environment:
```bash
python app.py
```
Then, by default, you can access the UI at [http://127.0.0.1:7860](http://127.0.0.1:7860).
## Run the code
You can also run the code with the following command:
```bash
python main.py \
--image_path_0 [image_path_0] --image_path_1 [image_path_1] \
--prompt_0 [prompt_0] --prompt_1 [prompt_1] \
--output_path [output_path] \
--use_adain --use_reschedule --save_inter
```
The script also supports the following options:
- `--image_path_0`: Path of the first image (default: "")
- `--prompt_0`: Prompt of the first image (default: "")
- `--image_path_1`: Path of the second image (default: "")
- `--prompt_1`: Prompt of the second image (default: "")
- `--model_path`: Pretrained model path (default: "stabilityai/stable-diffusion-2-1-base")
- `--output_path`: Path of the output image (default: "")
- `--save_lora_dir`: Path of the output lora directory (default: "./lora")
- `--load_lora_path_0`: Path of the lora directory of the first image (default: "")
- `--load_lora_path_1`: Path of the lora directory of the second image (default: "")
- `--use_adain`: Use AdaIN (default: False)
- `--use_reschedule`: Use reschedule sampling (default: False)
- `--lamb`: Hyperparameter $\lambda \in [0,1]$ for self-attention replacement, where a larger $\lambda$ indicates more replacements (default: 0.6)
- `--fix_lora_value`: Fix lora value (default: LoRA Interpolation, not fixed)
- `--save_inter`: Save intermediate results (default: False)
- `--num_frames`: Number of frames to generate (default: 50)
- `--duration`: Duration of each frame (default: 50)
Examples:
```bash
python main.py \
--image_path_0 ./assets/Trump.jpg --image_path_1 ./assets/Biden.jpg \
--prompt_0 "A photo of an American man" --prompt_1 "A photo of an American man" \
--output_path "./results/Trump_Biden" \
--use_adain --use_reschedule --save_inter
```
```bash
python main.py \
--image_path_0 ./assets/vangogh.jpg --image_path_1 ./assets/pearlgirl.jpg \
--prompt_0 "An oil painting of a man" --prompt_1 "An oil painting of a woman" \
--output_path "./results/vangogh_pearlgirl" \
--use_adain --use_reschedule --save_inter
```
```bash
python main.py \
--image_path_0 ./assets/lion.png --image_path_1 ./assets/tiger.png \
--prompt_0 "A photo of a lion" --prompt_1 "A photo of a tiger" \
--output_path "./results/lion_tiger" \
--use_adain --use_reschedule --save_inter
```
## MorphBench
To evaluate the effectiveness of our methods, we present *MorphBench*, the first benchmark dataset for assessing image morphing of general objects. You can download the dataset from [Google Drive](https://drive.google.com/file/d/1NWPzJhOgP-udP_wYbd0selRG4cu8xsu4/view?usp=sharing) or [Baidu Netdisk](https://pan.baidu.com/s/1J3xE3OJdEhKyoc1QObyYaA?pwd=putk).
## License
The code related to the DiffMorpher algorithm is licensed under [LICENSE](LICENSE.txt).
However, this project is mostly built on the open-sourse library [diffusers](https://github.com/huggingface/diffusers), which is under a separate license terms [Apache License 2.0](https://github.com/huggingface/diffusers/blob/main/LICENSE). (Cheers to the community as well!)
## Citation
```bibtex
@article{zhang2023diffmorpher,
title={DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing},
author={Zhang, Kaiwen and Zhou, Yifan and Xu, Xudong and Pan, Xingang and Dai, Bo},
journal={arXiv preprint arXiv:2312.07409},
year={2023}
}
```
