--- license: mit ---

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Jianzhu Guo 1†Dingyun Zhang 1,2Xiaoqiang Liu 1Zhizhou Zhong 1,3Yuan Zhang 1
Pengfei Wan 1Di Zhang 1
1 Kuaishou Technology  2 University of Science and Technology of China  3 Fudan University 


showcase
🔥 For more results, visit our homepage 🔥

## 🔥 Updates - **`2024/07/04`**: 🔥 We released the initial version of the inference code and models. Continuous updates, stay tuned! - **`2024/07/04`**: 😊 We released the [homepage](https://liveportrait.github.io) and technical report on [arXiv](https://arxiv.org/pdf/2407.03168). ## Introduction This repo, named **LivePortrait**, contains the official PyTorch implementation of our paper [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://arxiv.org/pdf/2407.03168). We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖. ## 🔥 Getting Started ### 1. Clone the code and prepare the environment ```bash git clone https://github.com/KwaiVGI/LivePortrait cd LivePortrait # create env using conda conda create -n LivePortrait python==3.9.18 conda activate LivePortrait # install dependencies with pip pip install -r requirements.txt ``` ### 2. Download pretrained weights Download our pretrained LivePortrait weights and face detection models of InsightFace from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). We have packed all weights in one directory 😊. Unzip and place them in `./pretrained_weights` ensuring the directory structure is as follows: ```text pretrained_weights ├── insightface │ └── models │ └── buffalo_l │ ├── 2d106det.onnx │ └── det_10g.onnx └── liveportrait ├── base_models │ ├── appearance_feature_extractor.pth │ ├── motion_extractor.pth │ ├── spade_generator.pth │ └── warping_module.pth ├── landmark.onnx └── retargeting_models └── stitching_retargeting_module.pth ``` ### 3. Inference 🚀 ```bash python inference.py ``` If the script runs successfully, you will get an output mp4 file named `animations/s6--d0_concat.mp4`. This file includes the following results: driving video, input image, and generated result.

image

Or, you can change the input by specifying the `-s` and `-d` arguments: ```bash python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 # or disable pasting back python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback # more options to see python inference.py -h ``` **More interesting results can be found in our [Homepage](https://liveportrait.github.io)** 😊 ### 4. Gradio interface We also provide a Gradio interface for a better experience, just run by: ```bash python app.py ``` ### 5. Inference speed evaluation 🚀🚀🚀 We have also provided a script to evaluate the inference speed of each module: ```bash python speed.py ``` Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with `torch.compile`: | Model | Parameters(M) | Model Size(MB) | Inference(ms) | |-----------------------------------|:-------------:|:--------------:|:-------------:| | Appearance Feature Extractor | 0.84 | 3.3 | 0.82 | | Motion Extractor | 28.12 | 108 | 0.84 | | Spade Generator | 55.37 | 212 | 7.59 | | Warping Module | 45.53 | 174 | 5.21 | | Stitching and Retargeting Modules| 0.23 | 2.3 | 0.31 | *Note: the listed values of Stitching and Retargeting Modules represent the combined parameter counts and the total sequential inference time of three MLP networks.* ## Acknowledgements We would like to thank the contributors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model), [Open Facevid2vid](https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis), [SPADE](https://github.com/NVlabs/SPADE), [InsightFace](https://github.com/deepinsight/insightface) repositories, for their open research and contributions. ## Citation 💖 If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX: ```bibtex @article{guo2024live, title = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control}, author = {Jianzhu Guo and Dingyun Zhang and Xiaoqiang Liu and Zhizhou Zhong and Yuan Zhang and Pengfei Wan and Di Zhang}, year = {2024}, journal = {arXiv preprint:2407.03168}, } ```