title: Liveportrait Vid2Vid
emoji: 😻
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.38.1
app_file: app.py
pinned: false
license: apache-2.0
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
This is the modification of LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control for allowing video as a source
🔥 For more results, visit LivePortrait homepage 🔥
🔥 Getting Started
1. Clone the code and prepare the environment
git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait
# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt
2. Download pretrained weights
Download the pretrained weights from HuggingFace:
# you may need to run `git lfs install` first
git clone https://huggingface.co/KwaiVGI/liveportrait pretrained_weights
Or, download all pretrained weights from Google Drive or Baidu Yun. We have packed all weights in one directory 😊. Unzip and place them in ./pretrained_weights ensuring the directory structure is as follows:
pretrained_weights
├── insightface
│ └── models
│ └── buffalo_l
│ ├── 2d106det.onnx
│ └── det_10g.onnx
└── liveportrait
├── base_models
│ ├── appearance_feature_extractor.pth
│ ├── motion_extractor.pth
│ ├── spade_generator.pth
│ └── warping_module.pth
├── landmark.onnx
└── retargeting_models
└── stitching_retargeting_module.pth
3. Inference 🚀
Fast hands-on
python inference.py
If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image, and generated result.
Or, you can change the input by specifying the -s and -d arguments:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4
# disable pasting back to run faster
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback
# more options to see
python inference.py -h
For video: you can change the input by specifying the -sd and -d arguments:
python inference.py -sd assets/examples/driving/d3.mp4 -d assets/examples/driving/d0.mp4 -vd True
# disable pasting back to run faster
python inference.py -sd assets/examples/driving/d3.mp4 -d assets/examples/driving/d0.mp4 -vd True --no_flag_pasteback
Driving video auto-cropping
📕 To use your own driving video, we recommend:
- Crop it to a 1:1 aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by
--flag_crop_driving_video. - Focus on the head area, similar to the example videos.
- Minimize shoulder movement.
- Make sure the first frame of driving video is a frontal face with neutral expression.
Below is a auto-cropping case by --flag_crop_driving_video:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video
If you find the results of auto-cropping is not well, you can modify the --scale_crop_video, --vy_ratio_crop_video options to adjust the scale and offset, or do it manually.
Template making
You can also use the .pkl file auto-generated to speed up the inference, and protect privacy, such as:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl
Discover more interesting results on our Homepage 😊
4. Gradio interface 🤗
We also provide a Gradio interface for a better experience, just run by:
python app.py
You can specify the --server_port, --share, --server_name arguments to satisfy your needs!
Acknowledgements
We would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace, LivePortrait repositories, for their open research and contributions.
Citation 💖
@article{guo2024liveportrait,
title = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
author = {Guo, Jianzhu and Zhang, Dingyun and Liu, Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},
journal = {arXiv preprint arXiv:2407.03168},
year = {2024}
}