Spaces:
Running
on
A10G
title: Liveportrait_video
emoji: π»
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.38.1
app_file: app.py
pinned: false
license: apache-2.0
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
This is the modification of LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control for allowing video as a source
π₯ For more results, visit LivePortrait homepage π₯
π₯ Getting Started
1. Clone the code and prepare the environment
git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait
# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt
2. Download pretrained weights
Download the pretrained weights from HuggingFace:
# you may need to run `git lfs install` first
git clone https://huggingface.co/KwaiVGI/liveportrait pretrained_weights
Or, download all pretrained weights from Google Drive or Baidu Yun. We have packed all weights in one directory π. Unzip and place them in ./pretrained_weights
ensuring the directory structure is as follows:
pretrained_weights
βββ insightface
β βββ models
β βββ buffalo_l
β βββ 2d106det.onnx
β βββ det_10g.onnx
βββ liveportrait
βββ base_models
β βββ appearance_feature_extractor.pth
β βββ motion_extractor.pth
β βββ spade_generator.pth
β βββ warping_module.pth
βββ landmark.onnx
βββ retargeting_models
βββ stitching_retargeting_module.pth
3. Inference π
Fast hands-on
python inference.py
If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4
. This file includes the following results: driving video, input image, and generated result.
Or, you can change the input by specifying the -s
and -d
arguments:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4
# disable pasting back to run faster
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback
# more options to see
python inference.py -h
For video: you can change the input by specifying the -sd
and -d
arguments:
python inference.py -sd assets/examples/driving/d3.mp4 -d assets/examples/driving/d0.mp4 -vd True
# disable pasting back to run faster
python inference.py -sd assets/examples/driving/d3.mp4 -d assets/examples/driving/d0.mp4 -vd True --no_flag_pasteback
Driving video auto-cropping
π To use your own driving video, we recommend:
- Crop it to a 1:1 aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by
--flag_crop_driving_video
. - Focus on the head area, similar to the example videos.
- Minimize shoulder movement.
- Make sure the first frame of driving video is a frontal face with neutral expression.
Below is a auto-cropping case by --flag_crop_driving_video
:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video
If you find the results of auto-cropping is not well, you can modify the --scale_crop_video
, --vy_ratio_crop_video
options to adjust the scale and offset, or do it manually.
Template making
You can also use the .pkl
file auto-generated to speed up the inference, and protect privacy, such as:
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl
Discover more interesting results on our Homepage π
4. Gradio interface π€
We also provide a Gradio interface for a better experience, just run by:
python app.py
You can specify the --server_port
, --share
, --server_name
arguments to satisfy your needs!
Acknowledgements
We would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace, LivePortrait repositories, for their open research and contributions.
Citation π
@article{guo2024liveportrait,
title = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
author = {Guo, Jianzhu and Zhang, Dingyun and Liu, Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},
journal = {arXiv preprint arXiv:2407.03168},
year = {2024}
}