Spaces:
Runtime error
ECON: Explicit Clothed humans Obtained from Normals
Yuliang Xiu 路 Jinlong Yang 路 Xu Cao 路 Dimitrios Tzionas 路 Michael J. Black
arXiv 2022
ECON is designed for "Human digitization from a color image", which combines the best properties of implicit and explicit representations, to infer high-fidelity 3D clothed humans from in-the-wild images, even with loose clothing or in challenging poses. ECON also supports multi-person reconstruction and SMPL-X based animation.
News :triangular_flag_on_post:
TODO
- Blender add-on for FBX export
- Full RGB texture generation
Table of Contents
Instructions
- See docs/installation.md to install all the required packages and setup the models
Demo
# For single-person image-based reconstruction
python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results
# For multi-person image-based reconstruction (see config/econ.yaml)
python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results -multi
# To generate the demo video of reconstruction results
python -m apps.multi_render -n {filename}
# To animate the reconstruction with SMPL-X pose parameters
python -m apps.avatarizer -n {filename}
Tricks
Some adjustable parameters in config/econ.yaml
use_ifnet: True- True: use IF-Nets+ for mesh completion ( $\text{ECON}_\text{IF}$ - Better quality)
- False: use SMPL-X for mesh completion ( $\text{ECON}_\text{EX}$ - Faster speed)
use_smpl: ["hand", "face"]- [ ]: don't use either hands or face parts from SMPL-X
- ["hand"]: only use the visible hands from SMPL-X
- ["hand", "face"]: use both visible hands and face from SMPL-X
thickness: 2cm- could be increased accordingly in case final reconstruction xx_full.obj looks flat
hps_type: PIXIE- "pixie": more accurate for face and hands
- "pymafx": more robust for challenging poses
k: 4- could be reduced accordingly in case the surface of xx_full.obj has discontinous artifacts
More Qualitative Results
 |
|---|
| Challenging Poses |
 |
| Loose Clothes |
Applications
 |
 |
|---|---|
| ECON could provide pseudo 3D GT for SHHQ Dataset | ECON supports multi-person reconstruction |
Citation
@article{xiu2022econ,
title={{ECON: Explicit Clothed humans Obtained from Normals}},
author={Xiu, Yuliang and Yang, Jinlong and Cao, Xu and Tzionas, Dimitrios and Black, Michael J.},
year={2022}
journal={{arXiv}:2212.07422},
}
Acknowledgments
We thank Lea Hering and Radek Dan臎膷ek for proof reading, Yao Feng, Haven Feng, and Weiyang Liu for their feedback and discussions, Tsvetelina Alexiadis for her help with the AMT perceptual study.
Here are some great resources we benefit from:
- ICON for SMPL-X Body Fitting
- BiNI for Bilateral Normal Integration
- MonoPortDataset for Data Processing, MonoPort for fast implicit surface query
- rembg for Human Segmentation
- pypoisson for poisson reconstruction
- MediaPipe for full-body landmark estimation
- PyTorch-NICP for non-rigid registration
- smplx, PyMAF-X, PIXIE for Human Pose & Shape Estimation
- CAPE and THuman for Dataset
- PyTorch3D for Differential Rendering
Some images used in the qualitative examples come from pinterest.com.
This project has received funding from the European Union鈥檚 Horizon 2020 research and innovation programme under the Marie Sk艂odowska-Curie grant agreement No.860768 (CLIPE Project).
License
This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.
Disclosure
MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society.
Contact
For technical questions, please contact yuliang.xiu@tue.mpg.de
For commercial licensing, please contact ps-licensing@tue.mpg.de