ECON: Explicit Clothed humans Obtained from Normals
Yuliang Xiu
·
Jinlong Yang
·
Xu Cao
·
Dimitrios Tzionas
·
Michael J. Black
arXiv 2022
ECON is designed for "Human digitization from a color image", which combines the best properties of implicit and explicit representations, to infer high-fidelity 3D clothed humans from in-the-wild images, even with **loose clothing** or in **challenging poses**. ECON also supports **multi-person reconstruction** and **SMPL-X based animation**.
## News :triangular_flag_on_post:
- [2022/12/22] is now available, created by [AroArz](https://github.com/AroArz)!
- [2022/12/15] Both demo and arXiv are available.
## TODO
- [ ] Blender add-on for FBX export
- [ ] Full RGB texture generation
Table of Contents
-
Instructions
-
Demo
-
Applications
-
Tricks
-
Citation
## Instructions
- See [docs/installation.md](docs/installation.md) to install all the required packages and setup the models
## Demo
```bash
# For single-person image-based reconstruction
python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results
# For multi-person image-based reconstruction (see config/econ.yaml)
python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results -multi
# To generate the demo video of reconstruction results
python -m apps.multi_render -n {filename}
# To animate the reconstruction with SMPL-X pose parameters
python -m apps.avatarizer -n {filename}
```
## Tricks
### Some adjustable parameters in _config/econ.yaml_
- `use_ifnet: False`
- True: use IF-Nets+ for mesh completion ( $\text{ECON}_\text{IF}$ - Better quality, **~2min / img**)
- False: use SMPL-X for mesh completion ( $\text{ECON}_\text{EX}$ - Faster speed, **~1.5min / img**)
- `use_smpl: ["hand", "face"]`
- [ ]: don't use either hands or face parts from SMPL-X
- ["hand"]: only use the **visible** hands from SMPL-X
- ["hand", "face"]: use both **visible** hands and face from SMPL-X
- `thickness: 2cm`
- could be increased accordingly in case final reconstruction **xx_full.obj** looks flat
- `k: 4`
- could be reduced accordingly in case the surface of **xx_full.obj** has discontinous artifacts
- `hps_type: PIXIE`
- "pixie": more accurate for face and hands
- "pymafx": more robust for challenging poses
- `texture_src: image`
- "image": direct mapping the aligned pixels to final mesh
- "SD": use Stable Diffusion to generate full texture (TODO)
## More Qualitative Results
|  |
| :------------------------------------: |
| _Challenging Poses_ |
|  |
| _Loose Clothes_ |
## Applications
|  |  |
| :----------------------------------------------------------------------------------------------------: | :-----------------------------------------: |
| _ECON could provide pseudo 3D GT for [SHHQ Dataset](https://github.com/stylegan-human/StyleGAN-Human)_ | _ECON supports multi-person reconstruction_ |
## Citation
```bibtex
@article{xiu2022econ,
title={{ECON: Explicit Clothed humans Obtained from Normals}},
author={Xiu, Yuliang and Yang, Jinlong and Cao, Xu and Tzionas, Dimitrios and Black, Michael J.},
year={2022}
journal={{arXiv}:2212.07422},
}
```
## Acknowledgments
We thank [Lea Hering](https://is.mpg.de/person/lhering) and [Radek Daněček](https://is.mpg.de/person/rdanecek) for proof reading, [Yao Feng](https://ps.is.mpg.de/person/yfeng), [Haven Feng](https://is.mpg.de/person/hfeng), and [Weiyang Liu](https://wyliu.com/) for their feedback and discussions, [Tsvetelina Alexiadis](https://ps.is.mpg.de/person/talexiadis) for her help with the AMT perceptual study.
Here are some great resources we benefit from:
- [ICON](https://github.com/YuliangXiu/ICON) for SMPL-X Body Fitting
- [BiNI](https://github.com/hoshino042/bilateral_normal_integration) for Bilateral Normal Integration
- [MonoPortDataset](https://github.com/Project-Splinter/MonoPortDataset) for Data Processing, [MonoPort](https://github.com/Project-Splinter/MonoPort) for fast implicit surface query
- [rembg](https://github.com/danielgatis/rembg) for Human Segmentation
- [MediaPipe](https://google.github.io/mediapipe/getting_started/python.html) for full-body landmark estimation
- [PyTorch-NICP](https://github.com/wuhaozhe/pytorch-nicp) for non-rigid registration
- [smplx](https://github.com/vchoutas/smplx), [PyMAF-X](https://www.liuyebin.com/pymaf-x/), [PIXIE](https://github.com/YadiraF/PIXIE) for Human Pose & Shape Estimation
- [CAPE](https://github.com/qianlim/CAPE) and [THuman](https://github.com/ZhengZerong/DeepHuman/tree/master/THUmanDataset) for Dataset
- [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Rendering
Some images used in the qualitative examples come from [pinterest.com](https://www.pinterest.com/).
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 ([CLIPE Project](https://www.clipe-itn.eu)).
---
## License
This code and model are available for non-commercial scientific research purposes as defined in the [LICENSE](LICENSE) file. By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE).
## Disclosure
MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society.
## Contact
For technical questions, please contact yuliang.xiu@tue.mpg.de
For commercial licensing, please contact ps-licensing@tue.mpg.de
