ECON

Runtime error

App Files Files Community

Yuliang commited on Apr 15, 2023

Commit

df6cc56

•

1 Parent(s): f4e5ac5

gradio init

Browse files

Files changed (23) hide show

.gitignore +2 -1
README.md +10 -211
app.py +352 -0
apps/__init__.py +0 -0
apps/infer.py +472 -474
apps/multi_render.py +0 -25
docker-compose.yaml +0 -19
docs/installation-docker.md +0 -80
docs/installation-ubuntu.md +0 -80
docs/installation-windows.md +0 -100
docs/testing.md +0 -71
docs/tricks.md +0 -29
environment-windows.yaml +0 -18
environment.yaml +0 -21
fetch_data.sh +0 -60
lib/dataset/TestDataset.py +2 -7
lib/dataset/mesh_util.py +60 -19
lib/pixielib/utils/config.py +36 -24
lib/smplx/body_models.py +6 -6
loose.txt +0 -100
packages.txt +13 -0
pose.txt +0 -100
requirements.txt +14 -0

.gitignore CHANGED Viewed

@@ -17,4 +17,5 @@ dist
 *egg-info
 *.so
 run.sh
-*.log

 *egg-info
 *.so
 run.sh
+*.log
+gradio_cached_examples/

README.md CHANGED Viewed

@@ -1,211 +1,10 @@
-<!-- PROJECT LOGO -->
-<p align="center">
-  <h1 align="center">ECON: Explicit Clothed humans Optimized via Normal integration</h1>
-  <p align="center">
-    <a href="http://xiuyuliang.cn/"><strong>Yuliang Xiu</strong></a>
-    ·
-    <a href="https://ps.is.tuebingen.mpg.de/person/jyang"><strong>Jinlong Yang</strong></a>
-    ·
-    <a href="https://hoshino042.github.io/homepage/"><strong>Xu Cao</strong></a>
-    ·
-    <a href="https://ps.is.mpg.de/~dtzionas"><strong>Dimitrios Tzionas</strong></a>
-    ·
-    <a href="https://ps.is.tuebingen.mpg.de/person/black"><strong>Michael J. Black</strong></a>
-  </p>
-  <h2 align="center">CVPR 2023 (Highlight)</h2>
-  <div align="center">
-    <img src="./assets/teaser.gif" alt="Logo" width="100%">
-  </div>
-  <p align="center">
-  <br>
-    <a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white"></a>
-    <a href="https://pytorchlightning.ai/"><img alt="Lightning" src="https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning&logoColor=white"></a>
-    <a href="https://cupy.dev/"><img alt="cupy" src="https://img.shields.io/badge/-Cupy-46C02B?logo=numpy&logoColor=white"></a>
-    <a href="https://twitter.com/yuliangxiu"><img alt='Twitter' src="https://img.shields.io/twitter/follow/yuliangxiu?label=%40yuliangxiu"></a>
-    <a href="https://discord.gg/Vqa7KBGRyk"><img alt="discord invitation link" src="https://dcbadge.vercel.app/api/server/Vqa7KBGRyk?style=flat"></a>
-    <br></br>
-    <a href='https://colab.research.google.com/drive/1YRgwoRCZIrSB2e7auEWFyG10Xzjbrbno?usp=sharing'><img src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'></a>
-    <a href='https://github.com/YuliangXiu/ECON/blob/master/docs/installation-docker.md'><img src='https://img.shields.io/badge/Docker-9cf.svg?logo=Docker' alt='Docker'></a>
-    <a href='https://carlosedubarreto.gumroad.com/l/CEB_ECON'><img src='https://img.shields.io/badge/Blender-F6DDCC.svg?logo=Blender' alt='Blender'></a>
-    <br></br>
-    <a href="https://arxiv.org/abs/2212.07422">
-      <img src='https://img.shields.io/badge/Paper-PDF-green?style=for-the-badge&logo=adobeacrobatreader&logoWidth=20&logoColor=white&labelColor=66cc00&color=94DD15' alt='Paper PDF'>
-    </a>
-    <a href='https://xiuyuliang.cn/econ/'>
-      <img src='https://img.shields.io/badge/ECON-Page-orange?style=for-the-badge&logo=Google%20chrome&logoColor=white&labelColor=D35400' alt='Project Page'></a>
-    <a href="https://youtu.be/j5hw4tsWpoY"><img alt="youtube views" title="Subscribe to my YouTube channel" src="https://img.shields.io/youtube/views/j5hw4tsWpoY?logo=youtube&labelColor=ce4630&style=for-the-badge"/></a>
-  </p>
-</p>
-<br/>
-ECON is designed for "Human digitization from a color image", which combines the best properties of implicit and explicit representations, to infer high-fidelity 3D clothed humans from in-the-wild images, even with **loose clothing** or in **challenging poses**. ECON also supports **multi-person reconstruction** and **SMPL-X based animation**.
-<br/>
-<br/>
-## News :triangular_flag_on_post:
-- [2023/02/27] ECON got accepted by CVPR 2023 as Highlight (top 10%)!
-- [2023/01/12] [Carlos Barreto](https://twitter.com/carlosedubarret/status/1613252471035494403) creates a Blender Addon ([Download](https://carlosedubarreto.gumroad.com/l/CEB_ECON), [Tutorial](https://youtu.be/sbWZbTf6ZYk)).
-- [2023/01/08] [Teddy Huang](https://github.com/Teddy12155555) creates [install-with-docker](docs/installation-docker.md) for ECON .
-- [2023/01/06] [Justin John](https://github.com/justinjohn0306) and [Carlos Barreto](https://github.com/carlosedubarreto) creates [install-on-windows](docs/installation-windows.md) for ECON .
-- [2022/12/22] <a href='https://colab.research.google.com/drive/1YRgwoRCZIrSB2e7auEWFyG10Xzjbrbno?usp=sharing' style='padding-left: 0.5rem;'><img src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'></a> is now available, created by [Aron Arzoomand](https://github.com/AroArz).
-- [2022/12/15] Both <a href="#demo">demo</a> and <a href="https://arxiv.org/abs/2212.07422">arXiv</a> are available.
-## TODO
-- [ ] Blender add-on for FBX export
-- [ ] Full RGB texture generation
-## Key idea: d-BiNI
-d-BiNI jointly optimizes front-back 2.5D surfaces such that: (1) high-frequency surface details agree with normal maps, (2) low-frequency surface variations, including discontinuities, align with SMPL-X surfaces, and (3) front-back 2.5D surface silhouettes are coherent with each other.
-|Front-view|Back-view|Side-view|
-|:--:|:--:|:---:|
-|![](assets/front-45.gif)|![](assets/back-45.gif)|![](assets/double-90.gif)||
-<details><summary>Please consider cite <strong>BiNI</strong> if it also helps on your project</summary>
-```bibtex
-@inproceedings{cao2022bilateral,
-  title={Bilateral normal integration},
-  author={Cao, Xu and Santo, Hiroaki and Shi, Boxin and Okura, Fumio and Matsushita, Yasuyuki},
-  booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part I},
-  pages={552--567},
-  year={2022},
-  organization={Springer}
-}
-```
-</details>
-<br>
-<!-- TABLE OF CONTENTS -->
-<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'>
-  <summary>Table of Contents</summary>
-  <ol>
-    <li>
-      <a href="#instructions">Instructions</a>
-    </li>
-    <li>
-      <a href="#demo">Demo</a>
-    </li>
-    <li>
-      <a href="#applications">Applications</a>
-    </li>
-    <li>
-      <a href="#citation">Citation</a>
-    </li>
-  </ol>
-</details>
-<br/>
-## Instructions
-- See [installion doc for Docker](docs/installation-docker.md) to run a docker container with pre-built image for ECON demo
-- See [installion doc for Windows](docs/installation-windows.md) to install all the required packages and setup the models on _Windows_
-- See [installion doc for Ubuntu](docs/installation-ubuntu.md) to install all the required packages and setup the models on _Ubuntu_
-- See [magic tricks](docs/tricks.md) to know a few technical tricks to further improve and accelerate ECON
-- See [testing](docs/testing.md) to prepare the testing data and evaluate ECON
-## Demo
-```bash
-# For single-person image-based reconstruction (w/ l visualization steps, 1.8min)
-python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results
-# For multi-person image-based reconstruction (see config/econ.yaml)
-python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results -multi
-# To generate the demo video of reconstruction results
-python -m apps.multi_render -n <filename>
-# To animate the reconstruction with SMPL-X pose parameters
-python -m apps.avatarizer -n <filename>
-```
-<br/>
-## More Qualitative Results
-|   ![OOD Poses](assets/OOD-poses.jpg)   |
-| :------------------------------------: |
-|          _Challenging Poses_           |
-| ![OOD Clothes](assets/OOD-outfits.jpg) |
-|            _Loose Clothes_             |
-## Applications
-|                                        ![SHHQ](assets/SHHQ.gif)                                        |         ![crowd](assets/crowd.gif)          |
-| :----------------------------------------------------------------------------------------------------: | :-----------------------------------------: |
-| _ECON could provide pseudo 3D GT for [SHHQ Dataset](https://github.com/stylegan-human/StyleGAN-Human)_ | _ECON supports multi-person reconstruction_ |
-<br/>
-<br/>
-## Citation
-```bibtex
-@inproceedings{xiu2023econ,
-  title     = {{ECON: Explicit Clothed humans Optimized via Normal integration}},
-  author    = {Xiu, Yuliang and Yang, Jinlong and Cao, Xu and Tzionas, Dimitrios and Black, Michael J.},
-  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
-  month     = {June},
-  year      = {2023},
-}
-```
-<br/>
-## Acknowledgments
-We thank [Lea Hering](https://is.mpg.de/person/lhering) and [Radek Daněček](https://is.mpg.de/person/rdanecek) for proof reading, [Yao Feng](https://ps.is.mpg.de/person/yfeng), [Haven Feng](https://is.mpg.de/person/hfeng), and [Weiyang Liu](https://wyliu.com/) for their feedback and discussions, [Tsvetelina Alexiadis](https://ps.is.mpg.de/person/talexiadis) for her help with the AMT perceptual study.
-Here are some great resources we benefit from:
-- [ICON](https://github.com/YuliangXiu/ICON) for SMPL-X Body Fitting
-- [BiNI](https://github.com/hoshino042/bilateral_normal_integration) for Bilateral Normal Integration
-- [MonoPortDataset](https://github.com/Project-Splinter/MonoPortDataset) for Data Processing, [MonoPort](https://github.com/Project-Splinter/MonoPort) for fast implicit surface query
-- [rembg](https://github.com/danielgatis/rembg) for Human Segmentation
-- [MediaPipe](https://google.github.io/mediapipe/getting_started/python.html) for full-body landmark estimation
-- [PyTorch-NICP](https://github.com/wuhaozhe/pytorch-nicp) for non-rigid registration
-- [smplx](https://github.com/vchoutas/smplx), [PyMAF-X](https://www.liuyebin.com/pymaf-x/), [PIXIE](https://github.com/YadiraF/PIXIE) for Human Pose & Shape Estimation
-- [CAPE](https://github.com/qianlim/CAPE) and [THuman](https://github.com/ZhengZerong/DeepHuman/tree/master/THUmanDataset) for Dataset
-- [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Rendering
-Some images used in the qualitative examples come from [pinterest.com](https://www.pinterest.com/).
-This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 ([CLIPE Project](https://www.clipe-itn.eu)).
-## Contributors
-Kudos to all of our amazing contributors! ECON thrives through open-source. In that spirit, we welcome all kinds of contributions from the community.
-<a href="https://github.com/yuliangxiu/ECON/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=yuliangxiu/ECON" />
-</a>
-_Contributor avatars are randomly shuffled._
----
-<br>
-## License
-This code and model are available for non-commercial scientific research purposes as defined in the [LICENSE](LICENSE) file. By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE).
-## Disclosure
-MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society.
-## Contact
-For technical questions, please contact yuliang.xiu@tue.mpg.de
-For commercial licensing, please contact ps-licensing@tue.mpg.de

+title: Fully-textured Clothed Human Digitization (ECON + TEXTure)
+metaTitle: Avatarify yourself from single image, by Yuliang Xiu
+emoji: 🤼
+colorFrom: green
+colorTo: pink
+sdk: gradio
+sdk_version: 3.27.0
+app_file: app.py
+pinned: true
+python_version: 3.8.15

app.py ADDED Viewed

	@@ -0,0 +1,352 @@

+# install
+import glob
+import gradio as gr
+import os
+import subprocess
+if os.getenv('SYSTEM') == 'spaces':
+    # subprocess.run('pip install pyembree'.split())
+    subprocess.run(
+        'pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu116_pyt1130/download.html'
+        .split()
+    )
+from apps.infer import generate_model, generate_video
+# running
+description = '''
+# Fully-textured Clothed Human Digitization (ECON + ControlNet)
+### ECON: Explicit Clothed humans Optimized via Normal integration (CVPR 2023, Highlight)
+<table>
+<th width="20%">
+<ul>
+<li><strong>Homepage</strong> <a href="https://econ.is.tue.mpg.de/">econ.is.tue.mpg.de</a></li>
+<li><strong>Code</strong> <a href="https://github.com/YuliangXiu/ECON">YuliangXiu/ECON</a></li>
+<li><strong>Paper</strong> <a href="https://arxiv.org/abs/2212.07422">arXiv</a>, <a href="https://readpaper.com/paper/4736821012688027649">ReadPaper</a></li>
+<li><strong>Chatroom</strong> <a href="https://discord.gg/Vqa7KBGRyk">Discord</a></li>
+</ul>
+<br>
+<ul>
+<li><strong>Colab Notebook</strong> <a href='https://colab.research.google.com/drive/1YRgwoRCZIrSB2e7auEWFyG10Xzjbrbno?usp=sharing'><img style="display: inline-block;" src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'></a></li>
+<li><strong>Blender Plugin</strong> <a href='https://carlosedubarreto.gumroad.com/l/CEB_ECON'><img style="display: inline-block;" src='https://img.shields.io/badge/Blender-F6DDCC.svg?logo=Blender' alt='Blender'></a></li>
+<li><strong>Docker Image</strong> <a href='https://github.com/YuliangXiu/ECON/blob/master/docs/installation-docker.md'><img style="display: inline-block;" src='https://img.shields.io/badge/Docker-9cf.svg?logo=Docker' alt='Docker'></a></li>
+<li><strong>Windows Setup</strong> <a href="https://github.com/YuliangXiu/ECON/blob/master/docs/installation-windows.md"><img style="display: inline-block;" src='https://img.shields.io/badge/Windows-00a2ed.svg?logo=Windows' akt='Windows'></a></li>
+</ul>
+<br>
+<a href="https://twitter.com/yuliangxiu"><img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/yuliangxiu?style=social"></a><br>
+<iframe src="https://ghbtns.com/github-btn.html?user=yuliangxiu&repo=ECON&type=star&count=true&v=2&size=small" frameborder="0" scrolling="0" width="100" height="20"></iframe>
+</th>
+<th width="40%">
+<iframe width="560" height="315" src="https://www.youtube.com/embed/j5hw4tsWpoY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</th>
+<th width="40%">
+<iframe width="560" height="315" src="https://www.youtube.com/embed/sbWZbTf6ZYk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</th>
+</table>
+#### Citation
+```
+@inproceedings{xiu2023econ,
+  title     = {{ECON: Explicit Clothed humans Optimized via Normal integration}},
+  author    = {Xiu, Yuliang and Yang, Jinlong and Cao, Xu and Tzionas, Dimitrios and Black, Michael J.},
+  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  month     = {June},
+  year      = {2023},
+}
+```
+<details>
+<summary>More</summary>
+#### Acknowledgments:
+- [controlnet-openpose](https://huggingface.co/spaces/diffusers/controlnet-openpose)
+- [TEXTure](https://huggingface.co/spaces/TEXTurePaper/TEXTure)
+#### Image Credits
+* [Pinterest](https://www.pinterest.com/search/pins/?q=parkour&rs=sitelinks_searchbox)
+#### Related works
+* [ICON @ MPI-IS](https://icon.is.tue.mpg.de/)
+* [MonoPort @ USC](https://xiuyuliang.cn/monoport)
+* [Phorhum @ Google](https://phorhum.github.io/)
+* [PIFuHD @ Meta](https://shunsukesaito.github.io/PIFuHD/)
+* [PaMIR @ Tsinghua](http://www.liuyebin.com/pamir/pamir.html)
+</details>
+<center>
+<h2> Generate pose & prompt-guided images / Upload photos /  Use examples &rarr; Submit Image (~2min) &rarr; Generate Video (~2min) </h2>
+</center>
+'''
+from controlnet_aux import OpenposeDetector
+from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
+from diffusers import UniPCMultistepScheduler
+import gradio as gr
+import torch
+import base64
+from io import BytesIO
+from PIL import Image
+# live conditioning
+canvas_html = "<pose-canvas id='canvas-root' style='display:flex;max-width: 500px;margin: 0 auto;'></pose-canvas>"
+load_js = """
+async () => {
+  const url = "https://huggingface.co/datasets/radames/gradio-components/raw/main/pose-gradio.js"
+  fetch(url)
+    .then(res => res.text())
+    .then(text => {
+      const script = document.createElement('script');
+      script.type = "module"
+      script.src = URL.createObjectURL(new Blob([text], { type: 'application/javascript' }));
+      document.head.appendChild(script);
+    });
+}
+"""
+get_js_image = """
+async (image_in_img, prompt, image_file_live_opt, live_conditioning) => {
+  const canvasEl = document.getElementById("canvas-root");
+  const data = canvasEl? canvasEl._data : null;
+  return [image_in_img, prompt, image_file_live_opt, data]
+}
+"""
+# Constants
+low_threshold = 100
+high_threshold = 200
+# Models
+pose_model = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
+controlnet = ControlNetModel.from_pretrained(
+    "lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16
+)
+pipe = StableDiffusionControlNetPipeline.from_pretrained(
+    "runwayml/stable-diffusion-v1-5",
+    controlnet=controlnet,
+    safety_checker=None,
+    torch_dtype=torch.float16
+)
+pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
+# This command loads the individual model components on GPU on-demand. So, we don't
+# need to explicitly call pipe.to("cuda").
+pipe.enable_model_cpu_offload()
+# xformers
+pipe.enable_xformers_memory_efficient_attention()
+# Generator seed,
+generator = torch.manual_seed(0)
+hint_prompts = '''
+<strong>Hints</strong>: <br>
+best quality, extremely detailed, solid color background,
+super detail, high detail, edge lighting, soft focus,
+light and dark contrast, 8k, high detail, edge lighting,
+3d, c4d, blender, oc renderer, ultra high definition, 3d rendering
+'''
+def get_pose(image):
+    return pose_model(image)
+# def generate_texture(input_shape, text, seed, guidance_scale):
+#     iface = gr.Interface.load("spaces/TEXTurePaper/TEXTure")
+#     output_shape = iface(input_shape, text, seed, guidance_scale)
+#     return output_shape
+def generate_images(image, prompt, image_file_live_opt='file', live_conditioning=None):
+    if image is None and 'image' not in live_conditioning:
+        raise gr.Error("Please provide an image")
+    try:
+        if image_file_live_opt == 'file':
+            pose = get_pose(image)
+        elif image_file_live_opt == 'webcam':
+            base64_img = live_conditioning['image']
+            image_data = base64.b64decode(base64_img.split(',')[1])
+            pose = Image.open(BytesIO(image_data)).convert('RGB').resize((512, 512))
+        output = pipe(
+            prompt,
+            pose,
+            generator=generator,
+            num_images_per_prompt=3,
+            num_inference_steps=20,
+        )
+        all_outputs = []
+        all_outputs.append(pose)
+        for image in output.images:
+            all_outputs.append(image)
+        return all_outputs, all_outputs
+    except Exception as e:
+        raise gr.Error(str(e))
+def toggle(choice):
+    if choice == "file":
+        return gr.update(visible=True, value=None), gr.update(visible=False, value=None)
+    elif choice == "webcam":
+        return gr.update(visible=False, value=None), gr.update(visible=True, value=canvas_html)
+examples_pose = glob.glob('examples/pose/*')
+examples_cloth = glob.glob('examples/cloth/*')
+default_step = 50
+with gr.Blocks() as demo:
+    gr.Markdown(description)
+    out_lst = []
+    with gr.Row():
+        with gr.Column():
+            with gr.Row():
+                live_conditioning = gr.JSON(value={}, visible=False)
+                with gr.Column():
+                    image_file_live_opt = gr.Radio(["file", "webcam"],
+                                                   value="file",
+                                                   label="How would you like to upload your image?")
+                    with gr.Row():
+                        image_in_img = gr.Image(source="upload", visible=True, type="pil", label="Image for Pose")
+                        canvas = gr.HTML(None, elem_id="canvas_html", visible=False)
+                    image_file_live_opt.change(
+                        fn=toggle,
+                        inputs=[image_file_live_opt],
+                        outputs=[image_in_img, canvas],
+                        queue=False
+                    )
+                    prompt = gr.Textbox(
+                        label="Enter your prompt to synthesise the image",
+                        max_lines=10,
+                        placeholder=
+                        "best quality, extremely detailed",
+                    )
+                    gr.Markdown(hint_prompts)
+                with gr.Column():
+                    gallery = gr.Gallery().style(grid=[2], height="auto")
+                    gallery_cache = gr.State()
+                    inp = gr.Image(type="filepath", label="Input Image for ECON")
+                    fitting_step = gr.inputs.Slider(
+                        10, 100, step=10, label='Fitting steps', default=default_step
+                    )
+            with gr.Row():
+                btn_sample = gr.Button("Generate Image")
+                btn_submit = gr.Button("Submit Image (~2min)")
+            btn_sample.click(
+                fn=generate_images,
+                inputs=[image_in_img, prompt, image_file_live_opt, live_conditioning],
+                outputs=[gallery, gallery_cache],
+                _js=get_js_image
+            )
+            def get_select_index(cache, evt: gr.SelectData):
+                return cache[evt.index]
+            gallery.select(
+                fn=get_select_index,
+                inputs=[gallery_cache],
+                outputs=[inp],
+            )
+            with gr.Row():
+                gr.Examples(
+                    examples=list(examples_pose),
+                    inputs=[inp],
+                    cache_examples=False,
+                    fn=generate_model,
+                    outputs=out_lst,
+                    label="Hard Pose Exampels"
+                )
+                gr.Examples(
+                    examples=list(examples_cloth),
+                    inputs=[inp],
+                    cache_examples=False,
+                    fn=generate_model,
+                    outputs=out_lst,
+                    label="Loose Cloth Exampels"
+                )
+        with gr.Column():
+            overlap_inp = gr.Image(type="filepath", label="Image Normal Overlap")
+            with gr.Row():
+                out_final = gr.Model3D(clear_color=[0.0, 0.0, 0.0, 0.0], label="Clothed human")
+                out_smpl = gr.Model3D(clear_color=[0.0, 0.0, 0.0, 0.0], label="SMPL-X body")
+            out_final_obj = gr.State()
+            vis_tensor_path = gr.State()
+            with gr.Row():
+                btn_video = gr.Button("Generate Video (~2min)")
+            with gr.Row():
+                out_vid = gr.Video(label="Shared on Twitter with #ECON")
+    # with gr.Row():
+    #     btn_texture = gr.Button("Generate Full-texture")
+    #     with gr.Row():
+    #         prompt = gr.Textbox(
+    #             label="Enter your prompt to texture the mesh",
+    #             max_lines=10,
+    #             placeholder=
+    #             "best quality, extremely detailed, solid color background, super detail, high detail, edge lighting, soft focus, light and dark contrast, 8k, high detail, edge lighting, 3d, c4d, blender, oc renderer, ultra high definition, 3d rendering",
+    #         )
+    #         seed = gr.Slider(label='Seed', minimum=0, maximum=100000, value=3, step=1)
+    #         guidance_scale = gr.Slider(
+    #             label='Guidance scale', minimum=0, maximum=50, value=7.5, step=0.1
+    #         )
+    #     progress_text = gr.Text(label='Progress')
+    #     with gr.Tabs():
+    #         with gr.TabItem(label='Images from each viewpoint'):
+    #             viewpoint_images = gr.Gallery(show_label=False)
+    #         with gr.TabItem(label='Result video'):
+    #             result_video = gr.Video(show_label=False)
+    #         with gr.TabItem(label='Output mesh file'):
+    #             output_file = gr.File(show_label=False)
+    out_lst = [out_smpl, out_final, out_final_obj, overlap_inp, vis_tensor_path]
+    btn_video.click(
+        fn=generate_video,
+        inputs=[vis_tensor_path],
+        outputs=[out_vid],
+    )
+    btn_submit.click(fn=generate_model, inputs=[inp, fitting_step], outputs=out_lst)
+    # btn_texture.click(
+    #     fn=generate_texture,
+    #     inputs=[out_final_obj, prompt, seed, guidance_scale],
+    #     outputs=[viewpoint_images, result_video, output_file, progress_text]
+    # )
+    demo.load(None, None, None, _js=load_js)
+if __name__ == "__main__":
+    # demo.launch(debug=False, enable_queue=False,
+    #             auth=(os.environ['USER'], os.environ['PASSWORD']),
+    #             auth_message="Register at icon.is.tue.mpg.de to get HuggingFace username and password.")
+    demo.launch(debug=True, enable_queue=True)

apps/__init__.py ADDED Viewed

File without changes

apps/infer.py CHANGED Viewed

@@ -21,7 +21,6 @@ warnings.filterwarnings("ignore")
 logging.getLogger("lightning").setLevel(logging.ERROR)
 logging.getLogger("trimesh").setLevel(logging.ERROR)
-import argparse
 import os
 import numpy as np
@@ -39,7 +38,7 @@ from lib.common.BNI_utils import save_normal_tensor
 from lib.common.config import cfg
 from lib.common.imutils import blend_rgb_norm
 from lib.common.local_affine import register
-from lib.common.render import query_color
 from lib.common.train_util import Format, init_loss
 from lib.common.voxelize import VoxelGrid
 from lib.dataset.mesh_util import *
@@ -48,32 +47,39 @@ from lib.net.geometry import rot6d_to_rotmat, rotation_matrix_to_angle_axis
 torch.backends.cudnn.benchmark = True
-if __name__ == "__main__":
-    # loading cfg file
-    parser = argparse.ArgumentParser()
-    parser.add_argument("-gpu", "--gpu_device", type=int, default=0)
-    parser.add_argument("-loop_smpl", "--loop_smpl", type=int, default=50)
-    parser.add_argument("-patience", "--patience", type=int, default=5)
-    parser.add_argument("-in_dir", "--in_dir", type=str, default="./examples")
-    parser.add_argument("-out_dir", "--out_dir", type=str, default="./results")
-    parser.add_argument("-seg_dir", "--seg_dir", type=str, default=None)
-    parser.add_argument("-cfg", "--config", type=str, default="./configs/econ.yaml")
-    parser.add_argument("-multi", action="store_false")
-    parser.add_argument("-novis", action="store_true")
-    args = parser.parse_args()
     # cfg read and merge
-    cfg.merge_from_file(args.config)
     cfg.merge_from_file("./lib/pymafx/configs/pymafx_config.yaml")
-    device = torch.device(f"cuda:{args.gpu_device}")
     # setting for testing on in-the-wild images
     cfg_show_list = [
-        "test_gpus", [args.gpu_device], "mcube_res", 512, "clean_mesh", True, "test_mode", True,
-        "batch_size", 1
     ]
     cfg.merge_from_list(cfg_show_list)
@@ -95,12 +101,11 @@ if __name__ == "__main__":
     SMPLX_object = SMPLX()
     dataset_param = {
-        "image_dir": args.in_dir,
-        "seg_dir": args.seg_dir,
         "use_seg": True,    # w/ or w/o segmentation
         "hps_type": cfg.bni.hps_type,    # pymafx/pixie
         "vol_res": cfg.vol_res,
-        "single": args.multi,
     }
     if cfg.bni.use_ifnet:
@@ -120,541 +125,534 @@ if __name__ == "__main__":
     print(colored(f"Dataset Size: {len(dataset)}", "green"))
-    pbar = tqdm(dataset)
-    for data in pbar:
-        losses = init_loss()
-        pbar.set_description(f"{data['name']}")
-        # final results rendered as image (PNG)
-        # 1. Render the final fitted SMPL (xxx_smpl.png)
-        # 2. Render the final reconstructed clothed human (xxx_cloth.png)
-        # 3. Blend the original image with predicted cloth normal (xxx_overlap.png)
-        # 4. Blend the cropped image with predicted cloth normal (xxx_crop.png)
-        os.makedirs(osp.join(args.out_dir, cfg.name, "png"), exist_ok=True)
-        # final reconstruction meshes (OBJ)
-        # 1. SMPL mesh (xxx_smpl_xx.obj)
-        # 2. SMPL params (xxx_smpl.npy)
-        # 3. d-BiNI surfaces (xxx_BNI.obj)
-        # 4. seperate face/hand mesh (xxx_hand/face.obj)
-        # 5. full shape impainted by IF-Nets+ after remeshing (xxx_IF.obj)
-        # 6. sideded or occluded parts (xxx_side.obj)
-        # 7. final reconstructed clothed human (xxx_full.obj)
-        os.makedirs(osp.join(args.out_dir, cfg.name, "obj"), exist_ok=True)
-        in_tensor = {
-            "smpl_faces": data["smpl_faces"], "image": data["img_icon"].to(device), "mask":
-            data["img_mask"].to(device)
-        }
-        # The optimizer and variables
-        optimed_pose = data["body_pose"].requires_grad_(True)
-        optimed_trans = data["trans"].requires_grad_(True)
-        optimed_betas = data["betas"].requires_grad_(True)
-        optimed_orient = data["global_orient"].requires_grad_(True)
-        optimizer_smpl = torch.optim.Adam([
-            optimed_pose, optimed_trans, optimed_betas, optimed_orient
-        ],
-                                          lr=1e-2,
-                                          amsgrad=True)
-        scheduler_smpl = torch.optim.lr_scheduler.ReduceLROnPlateau(
-            optimizer_smpl,
-            mode="min",
-            factor=0.5,
-            verbose=0,
-            min_lr=1e-5,
-            patience=args.patience,
         )
-        # [result_loop_1, result_loop_2, ...]
-        per_data_lst = []
-        N_body, N_pose = optimed_pose.shape[:2]
-        smpl_path = f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl_00.obj"
-        # remove this line if you change the loop_smpl and obtain different SMPL-X fits
-        if osp.exists(smpl_path):
-            smpl_verts_lst = []
-            smpl_faces_lst = []
-            for idx in range(N_body):
-                smpl_obj = f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl_{idx:02d}.obj"
-                smpl_mesh = trimesh.load(smpl_obj)
-                smpl_verts = torch.tensor(smpl_mesh.vertices).to(device).float()
-                smpl_faces = torch.tensor(smpl_mesh.faces).to(device).long()
-                smpl_verts_lst.append(smpl_verts)
-                smpl_faces_lst.append(smpl_faces)
-            batch_smpl_verts = torch.stack(smpl_verts_lst)
-            batch_smpl_faces = torch.stack(smpl_faces_lst)
             # render optimized mesh as normal [-1,1]
             in_tensor["T_normal_F"], in_tensor["T_normal_B"] = dataset.render_normal(
-                batch_smpl_verts, batch_smpl_faces
             )
             with torch.no_grad():
                 in_tensor["normal_F"], in_tensor["normal_B"] = normal_net.netG(in_tensor)
-            in_tensor["smpl_verts"] = batch_smpl_verts * torch.tensor([1., -1., 1.]).to(device)
-            in_tensor["smpl_faces"] = batch_smpl_faces[:, :, [0, 2, 1]]
-        else:
-            # smpl optimization
-            loop_smpl = tqdm(range(args.loop_smpl))
-            for i in loop_smpl:
-                per_loop_lst = []
-                optimizer_smpl.zero_grad()
-                N_body, N_pose = optimed_pose.shape[:2]
-                # 6d_rot to rot_mat
-                optimed_orient_mat = rot6d_to_rotmat(optimed_orient.view(-1,
-                                                                         6)).view(N_body, 1, 3, 3)
-                optimed_pose_mat = rot6d_to_rotmat(optimed_pose.view(-1,
-                                                                     6)).view(N_body, N_pose, 3, 3)
-                smpl_verts, smpl_landmarks, smpl_joints = dataset.smpl_model(
-                    shape_params=optimed_betas,
-                    expression_params=tensor2variable(data["exp"], device),
-                    body_pose=optimed_pose_mat,
-                    global_pose=optimed_orient_mat,
-                    jaw_pose=tensor2variable(data["jaw_pose"], device),
-                    left_hand_pose=tensor2variable(data["left_hand_pose"], device),
-                    right_hand_pose=tensor2variable(data["right_hand_pose"], device),
-                )
-                smpl_verts = (smpl_verts + optimed_trans) * data["scale"]
-                smpl_joints = (smpl_joints + optimed_trans) * data["scale"] * torch.tensor([
-                    1.0, 1.0, -1.0
-                ]).to(device)
-                # landmark errors
-                smpl_joints_3d = (
-                    smpl_joints[:, dataset.smpl_data.smpl_joint_ids_45_pixie, :] + 1.0
-                ) * 0.5
-                in_tensor["smpl_joint"] = smpl_joints[:,
-                                                      dataset.smpl_data.smpl_joint_ids_24_pixie, :]
-                ghum_lmks = data["landmark"][:, SMPLX_object.ghum_smpl_pairs[:, 0], :2].to(device)
-                ghum_conf = data["landmark"][:, SMPLX_object.ghum_smpl_pairs[:, 0], -1].to(device)
-                smpl_lmks = smpl_joints_3d[:, SMPLX_object.ghum_smpl_pairs[:, 1], :2]
-                # render optimized mesh as normal [-1,1]
-                in_tensor["T_normal_F"], in_tensor["T_normal_B"] = dataset.render_normal(
-                    smpl_verts * torch.tensor([1.0, -1.0, -1.0]).to(device),
-                    in_tensor["smpl_faces"],
-                )
-                T_mask_F, T_mask_B = dataset.render.get_image(type="mask")
-                with torch.no_grad():
-                    in_tensor["normal_F"], in_tensor["normal_B"] = normal_net.netG(in_tensor)
-                diff_F_smpl = torch.abs(in_tensor["T_normal_F"] - in_tensor["normal_F"])
-                diff_B_smpl = torch.abs(in_tensor["T_normal_B"] - in_tensor["normal_B"])
-                # silhouette loss
-                smpl_arr = torch.cat([T_mask_F, T_mask_B], dim=-1)
-                gt_arr = in_tensor["mask"].repeat(1, 1, 2)
-                diff_S = torch.abs(smpl_arr - gt_arr)
-                losses["silhouette"]["value"] = diff_S.mean()
-                # large cloth_overlap --> big difference between body and cloth mask
-                # for loose clothing, reply more on landmarks instead of silhouette+normal loss
-                cloth_overlap = diff_S.sum(dim=[1, 2]) / gt_arr.sum(dim=[1, 2])
-                cloth_overlap_flag = cloth_overlap > cfg.cloth_overlap_thres
-                losses["joint"]["weight"] = [50.0 if flag else 5.0 for flag in cloth_overlap_flag]
-                # small body_overlap --> large occlusion or out-of-frame
-                # for highly occluded body, reply only on high-confidence landmarks, no silhouette+normal loss
-                # BUG: PyTorch3D silhouette renderer generates dilated mask
-                bg_value = in_tensor["T_normal_F"][0, 0, 0, 0]
-                smpl_arr_fake = torch.cat([
-                    in_tensor["T_normal_F"][:, 0].ne(bg_value).float(),
-                    in_tensor["T_normal_B"][:, 0].ne(bg_value).float()
-                ],
-                                          dim=-1)
-                body_overlap = (gt_arr * smpl_arr_fake.gt(0.0)
-                               ).sum(dim=[1, 2]) / smpl_arr_fake.gt(0.0).sum(dim=[1, 2])
-                body_overlap_mask = (gt_arr * smpl_arr_fake).unsqueeze(1)
-                body_overlap_flag = body_overlap < cfg.body_overlap_thres
-                losses["normal"]["value"] = (
-                    diff_F_smpl * body_overlap_mask[..., :512] +
-                    diff_B_smpl * body_overlap_mask[..., 512:]
-                ).mean() / 2.0
-                losses["silhouette"]["weight"] = [0 if flag else 1.0 for flag in body_overlap_flag]
-                occluded_idx = torch.where(body_overlap_flag)[0]
-                ghum_conf[occluded_idx] *= ghum_conf[occluded_idx] > 0.95
-                losses["joint"]["value"] = (torch.norm(ghum_lmks - smpl_lmks, dim=2) *
-                                            ghum_conf).mean(dim=1)
-                # Weighted sum of the losses
-                smpl_loss = 0.0
-                pbar_desc = "Body Fitting -- "
-                for k in ["normal", "silhouette", "joint"]:
-                    per_loop_loss = (
-                        losses[k]["value"] * torch.tensor(losses[k]["weight"]).to(device)
-                    ).mean()
-                    pbar_desc += f"{k}: {per_loop_loss:.3f} | "
-                    smpl_loss += per_loop_loss
-                pbar_desc += f"Total: {smpl_loss:.3f}"
-                loose_str = ''.join([str(j) for j in cloth_overlap_flag.int().tolist()])
-                occlude_str = ''.join([str(j) for j in body_overlap_flag.int().tolist()])
-                pbar_desc += colored(f"| loose:{loose_str}, occluded:{occlude_str}", "yellow")
-                loop_smpl.set_description(pbar_desc)
-                # save intermediate results
-                if (i == args.loop_smpl - 1) and (not args.novis):
-                    per_loop_lst.extend([
-                        in_tensor["image"],
-                        in_tensor["T_normal_F"],
-                        in_tensor["normal_F"],
-                        diff_S[:, :, :512].unsqueeze(1).repeat(1, 3, 1, 1),
-                    ])
-                    per_loop_lst.extend([
-                        in_tensor["image"],
-                        in_tensor["T_normal_B"],
-                        in_tensor["normal_B"],
-                        diff_S[:, :, 512:].unsqueeze(1).repeat(1, 3, 1, 1),
-                    ])
-                    per_data_lst.append(
-                        get_optim_grid_image(per_loop_lst, None, nrow=N_body * 2, type="smpl")
-                    )
-                smpl_loss.backward()
-                optimizer_smpl.step()
-                scheduler_smpl.step(smpl_loss)
-            in_tensor["smpl_verts"] = smpl_verts * torch.tensor([1.0, 1.0, -1.0]).to(device)
-            in_tensor["smpl_faces"] = in_tensor["smpl_faces"][:, :, [0, 2, 1]]
-            if not args.novis:
-                per_data_lst[-1].save(
-                    osp.join(args.out_dir, cfg.name, f"png/{data['name']}_smpl.png")
-                )
-        if not args.novis:
-            img_crop_path = osp.join(args.out_dir, cfg.name, "png", f"{data['name']}_crop.png")
-            torchvision.utils.save_image(
-                torch.cat([
-                    data["img_crop"][:, :3], (in_tensor['normal_F'].detach().cpu() + 1.0) * 0.5,
-                    (in_tensor['normal_B'].detach().cpu() + 1.0) * 0.5
-                ],
-                          dim=3), img_crop_path
             )
-            rgb_norm_F = blend_rgb_norm(in_tensor["normal_F"], data)
-            rgb_norm_B = blend_rgb_norm(in_tensor["normal_B"], data)
-            img_overlap_path = osp.join(args.out_dir, cfg.name, f"png/{data['name']}_overlap.png")
-            torchvision.utils.save_image(
-                torch.cat([data["img_raw"], rgb_norm_F, rgb_norm_B], dim=-1) / 255.,
-                img_overlap_path
-            )
-        smpl_obj_lst = []
-        for idx in range(N_body):
-            smpl_obj = trimesh.Trimesh(
-                in_tensor["smpl_verts"].detach().cpu()[idx] * torch.tensor([1.0, -1.0, 1.0]),
-                in_tensor["smpl_faces"].detach().cpu()[0][:, [0, 2, 1]],
-                process=False,
-                maintains_order=True,
-            )
-            smpl_obj_path = f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl_{idx:02d}.obj"
-            if not osp.exists(smpl_obj_path):
-                smpl_obj.export(smpl_obj_path)
-                smpl_info = {
-                    "betas":
-                    optimed_betas[idx].detach().cpu().unsqueeze(0),
-                    "body_pose":
-                    rotation_matrix_to_angle_axis(optimed_pose_mat[idx].detach()
-                                                 ).cpu().unsqueeze(0),
-                    "global_orient":
-                    rotation_matrix_to_angle_axis(optimed_orient_mat[idx].detach()
-                                                 ).cpu().unsqueeze(0),
-                    "transl":
-                    optimed_trans[idx].detach().cpu(),
-                    "expression":
-                    data["exp"][idx].cpu().unsqueeze(0),
-                    "jaw_pose":
-                    rotation_matrix_to_angle_axis(data["jaw_pose"][idx]).cpu().unsqueeze(0),
-                    "left_hand_pose":
-                    rotation_matrix_to_angle_axis(data["left_hand_pose"][idx]).cpu().unsqueeze(0),
-                    "right_hand_pose":
-                    rotation_matrix_to_angle_axis(data["right_hand_pose"][idx]).cpu().unsqueeze(0),
-                    "scale":
-                    data["scale"][idx].cpu(),
-                }
-                np.save(
-                    smpl_obj_path.replace(".obj", ".npy"),
-                    smpl_info,
-                    allow_pickle=True,
-                )
-            smpl_obj_lst.append(smpl_obj)
-        del optimizer_smpl
-        del optimed_betas
-        del optimed_orient
-        del optimed_pose
-        del optimed_trans
-        torch.cuda.empty_cache()
-        # ------------------------------------------------------------------------------------------------------------------
-        # clothing refinement
-        per_data_lst = []
-        batch_smpl_verts = in_tensor["smpl_verts"].detach() * torch.tensor([1.0, -1.0, 1.0],
-                                                                           device=device)
-        batch_smpl_faces = in_tensor["smpl_faces"].detach()[:, :, [0, 2, 1]]
-        in_tensor["depth_F"], in_tensor["depth_B"] = dataset.render_depth(
-            batch_smpl_verts, batch_smpl_faces
         )
-        per_loop_lst = []
-        in_tensor["BNI_verts"] = []
-        in_tensor["BNI_faces"] = []
-        in_tensor["body_verts"] = []
-        in_tensor["body_faces"] = []
-        for idx in range(N_body):
-            final_path = f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_full.obj"
-            side_mesh = smpl_obj_lst[idx].copy()
-            face_mesh = smpl_obj_lst[idx].copy()
-            hand_mesh = smpl_obj_lst[idx].copy()
-            smplx_mesh = smpl_obj_lst[idx].copy()
-            # save normals, depths and masks
-            BNI_dict = save_normal_tensor(
-                in_tensor,
-                idx,
-                osp.join(args.out_dir, cfg.name, f"BNI/{data['name']}_{idx}"),
-                cfg.bni.thickness,
-            )
-            # BNI process
-            BNI_object = BNI(
-                dir_path=osp.join(args.out_dir, cfg.name, "BNI"),
-                name=data["name"],
-                BNI_dict=BNI_dict,
-                cfg=cfg.bni,
-                device=device
             )
-            BNI_object.extract_surface(False)
-            in_tensor["body_verts"].append(torch.tensor(smpl_obj_lst[idx].vertices).float())
-            in_tensor["body_faces"].append(torch.tensor(smpl_obj_lst[idx].faces).long())
-            # requires shape completion when low overlap
-            # replace SMPL by completed mesh as side_mesh
-            if cfg.bni.use_ifnet:
-                side_mesh_path = f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_IF.obj"
-                side_mesh = apply_face_mask(side_mesh, ~SMPLX_object.smplx_eyeball_fid_mask)
-                # mesh completion via IF-net
-                in_tensor.update(
-                    dataset.depth_to_voxel({
-                        "depth_F": BNI_object.F_depth.unsqueeze(0), "depth_B":
-                        BNI_object.B_depth.unsqueeze(0)
-                    })
                 )
-                occupancies = VoxelGrid.from_mesh(side_mesh, cfg.vol_res, loc=[
-                    0,
-                ] * 3, scale=2.0).data.transpose(2, 1, 0)
-                occupancies = np.flip(occupancies, axis=1)
-                in_tensor["body_voxels"] = torch.tensor(occupancies.copy()
-                                                       ).float().unsqueeze(0).to(device)
-                with torch.no_grad():
-                    sdf = ifnet.reconEngine(netG=ifnet.netG, batch=in_tensor)
-                    verts_IF, faces_IF = ifnet.reconEngine.export_mesh(sdf)
-                if ifnet.clean_mesh_flag:
-                    verts_IF, faces_IF = clean_mesh(verts_IF, faces_IF)
-                side_mesh = trimesh.Trimesh(verts_IF, faces_IF)
-                side_mesh = remesh_laplacian(side_mesh, side_mesh_path)
-            else:
-                side_mesh = apply_vertex_mask(
-                    side_mesh,
-                    (
-                        SMPLX_object.front_flame_vertex_mask + SMPLX_object.smplx_mano_vertex_mask +
-                        SMPLX_object.eyeball_vertex_mask
-                    ).eq(0).float(),
-                )
-                #register side_mesh to BNI surfaces
-                side_mesh = Meshes(
-                    verts=[torch.tensor(side_mesh.vertices).float()],
-                    faces=[torch.tensor(side_mesh.faces).long()],
-                ).to(device)
-                sm = SubdivideMeshes(side_mesh)
-                side_mesh = register(BNI_object.F_B_trimesh, sm(side_mesh), device)
-            side_verts = torch.tensor(side_mesh.vertices).float().to(device)
-            side_faces = torch.tensor(side_mesh.faces).long().to(device)
-            # Possion Fusion between SMPLX and BNI
-            # 1. keep the faces invisible to front+back cameras
-            # 2. keep the front-FLAME+MANO faces
-            # 3. remove eyeball faces
-            # export intermediate meshes
-            BNI_object.F_B_trimesh.export(
-                f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_BNI.obj"
-            )
-            full_lst = []
-            if "face" in cfg.bni.use_smpl:
-                # only face
-                face_mesh = apply_vertex_mask(face_mesh, SMPLX_object.front_flame_vertex_mask)
-                face_mesh.vertices = face_mesh.vertices - np.array([0, 0, cfg.bni.thickness])
-                # remove face neighbor triangles
-                BNI_object.F_B_trimesh = part_removal(
-                    BNI_object.F_B_trimesh,
-                    face_mesh,
-                    cfg.bni.face_thres,
-                    device,
-                    smplx_mesh,
-                    region="face"
-                )
-                side_mesh = part_removal(
-                    side_mesh, face_mesh, cfg.bni.face_thres, device, smplx_mesh, region="face"
-                )
-                face_mesh.export(f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_face.obj")
-                full_lst += [face_mesh]
-            if "hand" in cfg.bni.use_smpl and (True in data['hands_visibility'][idx]):
-                hand_mask = torch.zeros(SMPLX_object.smplx_verts.shape[0], )
-                if data['hands_visibility'][idx][0]:
-                    hand_mask.index_fill_(
-                        0, torch.tensor(SMPLX_object.smplx_mano_vid_dict["left_hand"]), 1.0
-                    )
-                if data['hands_visibility'][idx][1]:
-                    hand_mask.index_fill_(
-                        0, torch.tensor(SMPLX_object.smplx_mano_vid_dict["right_hand"]), 1.0
-                    )
-                # only hands
-                hand_mesh = apply_vertex_mask(hand_mesh, hand_mask)
-                # remove hand neighbor triangles
-                BNI_object.F_B_trimesh = part_removal(
-                    BNI_object.F_B_trimesh,
-                    hand_mesh,
-                    cfg.bni.hand_thres,
-                    device,
-                    smplx_mesh,
-                    region="hand"
-                )
-                side_mesh = part_removal(
-                    side_mesh, hand_mesh, cfg.bni.hand_thres, device, smplx_mesh, region="hand"
-                )
-                hand_mesh.export(f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_hand.obj")
-                full_lst += [hand_mesh]
-            full_lst += [BNI_object.F_B_trimesh]
-            # initial side_mesh could be SMPLX or IF-net
-            side_mesh = part_removal(
-                side_mesh, sum(full_lst), 2e-2, device, smplx_mesh, region="", clean=False
             )
-            full_lst += [side_mesh]
-            # # export intermediate meshes
-            BNI_object.F_B_trimesh.export(
-                f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_BNI.obj"
-            )
-            side_mesh.export(f"{args.out_dir}/{cfg.name}/obj/{data['name']}_{idx}_side.obj")
-            if cfg.bni.use_poisson:
-                final_mesh = poisson(
-                    sum(full_lst),
-                    final_path,
-                    cfg.bni.poisson_depth,
-                )
-                print(
-                    colored(
-                        f"\n Poisson completion to {Format.start} {final_path} {Format.end}",
-                        "yellow"
-                    )
-                )
-            else:
-                final_mesh = sum(full_lst)
-                final_mesh.export(final_path)
-            if not args.novis:
-                dataset.render.load_meshes(final_mesh.vertices, final_mesh.faces)
-                rotate_recon_lst = dataset.render.get_image(cam_type="four")
-                per_loop_lst.extend([in_tensor['image'][idx:idx + 1]] + rotate_recon_lst)
-            if cfg.bni.texture_src == 'image':
-                # coloring the final mesh (front: RGB pixels, back: normal colors)
-                final_colors = query_color(
-                    torch.tensor(final_mesh.vertices).float(),
-                    torch.tensor(final_mesh.faces).long(),
-                    in_tensor["image"][idx:idx + 1],
-                    device=device,
-                )
-                final_mesh.visual.vertex_colors = final_colors
-                final_mesh.export(final_path)
-            elif cfg.bni.texture_src == 'SD':
-                # !TODO: add texture from Stable Diffusion
-                pass
-        if len(per_loop_lst) > 0 and (not args.novis):
-            per_data_lst.append(get_optim_grid_image(per_loop_lst, None, nrow=5, type="cloth"))
-            per_data_lst[-1].save(osp.join(args.out_dir, cfg.name, f"png/{data['name']}_cloth.png"))
-            # for video rendering
-            in_tensor["BNI_verts"].append(torch.tensor(final_mesh.vertices).float())
-            in_tensor["BNI_faces"].append(torch.tensor(final_mesh.faces).long())
-            os.makedirs(osp.join(args.out_dir, cfg.name, "vid"), exist_ok=True)
-            in_tensor["uncrop_param"] = data["uncrop_param"]
-            in_tensor["img_raw"] = data["img_raw"]
-            torch.save(
-                in_tensor, osp.join(args.out_dir, cfg.name, f"vid/{data['name']}_in_tensor.pt")
-            )

 logging.getLogger("lightning").setLevel(logging.ERROR)
 logging.getLogger("trimesh").setLevel(logging.ERROR)
 import os
 import numpy as np
 from lib.common.config import cfg
 from lib.common.imutils import blend_rgb_norm
 from lib.common.local_affine import register
+from lib.common.render import query_color, Render
 from lib.common.train_util import Format, init_loss
 from lib.common.voxelize import VoxelGrid
 from lib.dataset.mesh_util import *
 torch.backends.cudnn.benchmark = True
+def generate_video(vis_tensor_path):
+    in_tensor = torch.load(vis_tensor_path)
+    render = Render(size=512, device=torch.device("cuda:0"))
+    # visualize the final results in self-rotation mode
+    verts_lst = in_tensor["body_verts"] + in_tensor["BNI_verts"]
+    faces_lst = in_tensor["body_faces"] + in_tensor["BNI_faces"]
+    # self-rotated video
+    tmp_path = vis_tensor_path.replace("_in_tensor.pt", "_tmp.mp4")
+    out_path = vis_tensor_path.replace("_in_tensor.pt", ".mp4")
+    render.load_meshes(verts_lst, faces_lst)
+    render.get_rendered_video_multi(in_tensor, tmp_path)
+    os.system(f'ffmpeg -y -loglevel quiet -stats -i {tmp_path} -c:v libx264 {out_path}')
+    return out_path, out_path
+def generate_model(in_path, fitting_step=50):
+    out_dir = "./results"
     # cfg read and merge
+    cfg.merge_from_file("./configs/econ.yaml")
     cfg.merge_from_file("./lib/pymafx/configs/pymafx_config.yaml")
+    device = torch.device(f"cuda:0")
     # setting for testing on in-the-wild images
     cfg_show_list = [
+        "test_gpus", [0], "mcube_res", 512, "clean_mesh", True, "test_mode", True, "batch_size", 1
     ]
     cfg.merge_from_list(cfg_show_list)
     SMPLX_object = SMPLX()
     dataset_param = {
+        "image_path": in_path,
         "use_seg": True,    # w/ or w/o segmentation
         "hps_type": cfg.bni.hps_type,    # pymafx/pixie
         "vol_res": cfg.vol_res,
+        "single": True,
     }
     if cfg.bni.use_ifnet:
     print(colored(f"Dataset Size: {len(dataset)}", "green"))
+    data = dataset[0]
+    losses = init_loss()
+    print(f"{data['name']}")
+    # final results rendered as image (PNG)
+    # 1. Render the final fitted SMPL (xxx_smpl.png)
+    # 2. Render the final reconstructed clothed human (xxx_cloth.png)
+    # 3. Blend the original image with predicted cloth normal (xxx_overlap.png)
+    # 4. Blend the cropped image with predicted cloth normal (xxx_crop.png)
+    os.makedirs(osp.join(out_dir, cfg.name, "png"), exist_ok=True)
+    # final reconstruction meshes (OBJ)
+    # 1. SMPL mesh (xxx_smpl_xx.obj)
+    # 2. SMPL params (xxx_smpl.npy)
+    # 3. d-BiNI surfaces (xxx_BNI.obj)
+    # 4. seperate face/hand mesh (xxx_hand/face.obj)
+    # 5. full shape impainted by IF-Nets+ after remeshing (xxx_IF.obj)
+    # 6. sideded or occluded parts (xxx_side.obj)
+    # 7. final reconstructed clothed human (xxx_full.obj)
+    os.makedirs(osp.join(out_dir, cfg.name, "obj"), exist_ok=True)
+    in_tensor = {
+        "smpl_faces": data["smpl_faces"], "image": data["img_icon"].to(device), "mask":
+        data["img_mask"].to(device)
+    }
+    # The optimizer and variables
+    optimed_pose = data["body_pose"].requires_grad_(True)
+    optimed_trans = data["trans"].requires_grad_(True)
+    optimed_betas = data["betas"].requires_grad_(True)
+    optimed_orient = data["global_orient"].requires_grad_(True)
+    optimizer_smpl = torch.optim.Adam([optimed_pose, optimed_trans, optimed_betas, optimed_orient],
+                                      lr=1e-2,
+                                      amsgrad=True)
+    scheduler_smpl = torch.optim.lr_scheduler.ReduceLROnPlateau(
+        optimizer_smpl,
+        mode="min",
+        factor=0.5,
+        verbose=0,
+        min_lr=1e-5,
+        patience=5,
+    )
+    # [result_loop_1, result_loop_2, ...]
+    per_data_lst = []
+    N_body, N_pose = optimed_pose.shape[:2]
+    smpl_path = f"{out_dir}/{cfg.name}/obj/{data['name']}_smpl_00.obj"
+    # remove this line if you change the loop_smpl and obtain different SMPL-X fits
+    if osp.exists(smpl_path):
+        smpl_verts_lst = []
+        smpl_faces_lst = []
+        for idx in range(N_body):
+            smpl_obj = f"{out_dir}/{cfg.name}/obj/{data['name']}_smpl_{idx:02d}.obj"
+            smpl_mesh = trimesh.load(smpl_obj)
+            smpl_verts = torch.tensor(smpl_mesh.vertices).to(device).float()
+            smpl_faces = torch.tensor(smpl_mesh.faces).to(device).long()
+            smpl_verts_lst.append(smpl_verts)
+            smpl_faces_lst.append(smpl_faces)
+        batch_smpl_verts = torch.stack(smpl_verts_lst)
+        batch_smpl_faces = torch.stack(smpl_faces_lst)
+        # render optimized mesh as normal [-1,1]
+        in_tensor["T_normal_F"], in_tensor["T_normal_B"] = dataset.render_normal(
+            batch_smpl_verts, batch_smpl_faces
         )
+        with torch.no_grad():
+            in_tensor["normal_F"], in_tensor["normal_B"] = normal_net.netG(in_tensor)
+        in_tensor["smpl_verts"] = batch_smpl_verts * torch.tensor([1., -1., 1.]).to(device)
+        in_tensor["smpl_faces"] = batch_smpl_faces[:, :, [0, 2, 1]]
+    else:
+        # smpl optimization
+        loop_smpl = tqdm(range(fitting_step))
+        for i in loop_smpl:
+            per_loop_lst = []
+            optimizer_smpl.zero_grad()
+            N_body, N_pose = optimed_pose.shape[:2]
+            # 6d_rot to rot_mat
+            optimed_orient_mat = rot6d_to_rotmat(optimed_orient.view(-1, 6)).view(N_body, 1, 3, 3)
+            optimed_pose_mat = rot6d_to_rotmat(optimed_pose.view(-1, 6)).view(N_body, N_pose, 3, 3)
+            smpl_verts, smpl_landmarks, smpl_joints = dataset.smpl_model(
+                shape_params=optimed_betas,
+                expression_params=tensor2variable(data["exp"], device),
+                body_pose=optimed_pose_mat,
+                global_pose=optimed_orient_mat,
+                jaw_pose=tensor2variable(data["jaw_pose"], device),
+                left_hand_pose=tensor2variable(data["left_hand_pose"], device),
+                right_hand_pose=tensor2variable(data["right_hand_pose"], device),
+            )
+            smpl_verts = (smpl_verts + optimed_trans) * data["scale"]
+            smpl_joints = (smpl_joints + optimed_trans) * data["scale"] * torch.tensor([
+                1.0, 1.0, -1.0
+            ]).to(device)
+            # landmark errors
+            smpl_joints_3d = (
+                smpl_joints[:, dataset.smpl_data.smpl_joint_ids_45_pixie, :] + 1.0
+            ) * 0.5
+            in_tensor["smpl_joint"] = smpl_joints[:, dataset.smpl_data.smpl_joint_ids_24_pixie, :]
+            ghum_lmks = data["landmark"][:, SMPLX_object.ghum_smpl_pairs[:, 0], :2].to(device)
+            ghum_conf = data["landmark"][:, SMPLX_object.ghum_smpl_pairs[:, 0], -1].to(device)
+            smpl_lmks = smpl_joints_3d[:, SMPLX_object.ghum_smpl_pairs[:, 1], :2]
             # render optimized mesh as normal [-1,1]
             in_tensor["T_normal_F"], in_tensor["T_normal_B"] = dataset.render_normal(
+                smpl_verts * torch.tensor([1.0, -1.0, -1.0]).to(device),
+                in_tensor["smpl_faces"],
             )
+            T_mask_F, T_mask_B = dataset.render.get_image(type="mask")
             with torch.no_grad():
                 in_tensor["normal_F"], in_tensor["normal_B"] = normal_net.netG(in_tensor)
+            diff_F_smpl = torch.abs(in_tensor["T_normal_F"] - in_tensor["normal_F"])
+            diff_B_smpl = torch.abs(in_tensor["T_normal_B"] - in_tensor["normal_B"])
+            # silhouette loss
+            smpl_arr = torch.cat([T_mask_F, T_mask_B], dim=-1)
+            gt_arr = in_tensor["mask"].repeat(1, 1, 2)
+            diff_S = torch.abs(smpl_arr - gt_arr)
+            losses["silhouette"]["value"] = diff_S.mean()
+            # large cloth_overlap --> big difference between body and cloth mask
+            # for loose clothing, reply more on landmarks instead of silhouette+normal loss
+            cloth_overlap = diff_S.sum(dim=[1, 2]) / gt_arr.sum(dim=[1, 2])
+            cloth_overlap_flag = cloth_overlap > cfg.cloth_overlap_thres
+            losses["joint"]["weight"] = [50.0 if flag else 5.0 for flag in cloth_overlap_flag]
+            # small body_overlap --> large occlusion or out-of-frame
+            # for highly occluded body, reply only on high-confidence landmarks, no silhouette+normal loss
+            # BUG: PyTorch3D silhouette renderer generates dilated mask
+            bg_value = in_tensor["T_normal_F"][0, 0, 0, 0]
+            smpl_arr_fake = torch.cat([
+                in_tensor["T_normal_F"][:, 0].ne(bg_value).float(),
+                in_tensor["T_normal_B"][:, 0].ne(bg_value).float()
+            ],
+                                      dim=-1)
+            body_overlap = (gt_arr * smpl_arr_fake.gt(0.0)
+                           ).sum(dim=[1, 2]) / smpl_arr_fake.gt(0.0).sum(dim=[1, 2])
+            body_overlap_mask = (gt_arr * smpl_arr_fake).unsqueeze(1)
+            body_overlap_flag = body_overlap < cfg.body_overlap_thres
+            losses["normal"]["value"] = (
+                diff_F_smpl * body_overlap_mask[..., :512] +
+                diff_B_smpl * body_overlap_mask[..., 512:]
+            ).mean() / 2.0
+            losses["silhouette"]["weight"] = [0 if flag else 1.0 for flag in body_overlap_flag]
+            occluded_idx = torch.where(body_overlap_flag)[0]
+            ghum_conf[occluded_idx] *= ghum_conf[occluded_idx] > 0.95
+            losses["joint"]["value"] = (torch.norm(ghum_lmks - smpl_lmks, dim=2) *
+                                        ghum_conf).mean(dim=1)
+            # Weighted sum of the losses
+            smpl_loss = 0.0
+            pbar_desc = "Body Fitting -- "
+            for k in ["normal", "silhouette", "joint"]:
+                per_loop_loss = (losses[k]["value"] *
+                                 torch.tensor(losses[k]["weight"]).to(device)).mean()
+                pbar_desc += f"{k}: {per_loop_loss:.3f} | "
+                smpl_loss += per_loop_loss
+            pbar_desc += f"Total: {smpl_loss:.3f}"
+            loose_str = ''.join([str(j) for j in cloth_overlap_flag.int().tolist()])
+            occlude_str = ''.join([str(j) for j in body_overlap_flag.int().tolist()])
+            pbar_desc += colored(f"| loose:{loose_str}, occluded:{occlude_str}", "yellow")
+            loop_smpl.set_description(pbar_desc)
+            # save intermediate results
+            if (i == fitting_step - 1):
+                per_loop_lst.extend([
+                    in_tensor["image"],
+                    in_tensor["T_normal_F"],
+                    in_tensor["normal_F"],
+                    diff_S[:, :, :512].unsqueeze(1).repeat(1, 3, 1, 1),
+                ])
+                per_loop_lst.extend([
+                    in_tensor["image"],
+                    in_tensor["T_normal_B"],
+                    in_tensor["normal_B"],
+                    diff_S[:, :, 512:].unsqueeze(1).repeat(1, 3, 1, 1),
+                ])
+                per_data_lst.append(
+                    get_optim_grid_image(per_loop_lst, None, nrow=N_body * 2, type="smpl")
+                )
+            smpl_loss.backward()
+            optimizer_smpl.step()
+            scheduler_smpl.step(smpl_loss)
+        in_tensor["smpl_verts"] = smpl_verts * torch.tensor([1.0, 1.0, -1.0]).to(device)
+        in_tensor["smpl_faces"] = in_tensor["smpl_faces"][:, :, [0, 2, 1]]
+        per_data_lst[-1].save(osp.join(out_dir, cfg.name, f"png/{data['name']}_smpl.png"))
+    img_crop_path = osp.join(out_dir, cfg.name, "png", f"{data['name']}_crop.png")
+    torchvision.utils.save_image(
+        torch.cat([
+            data["img_crop"][:, :3], (in_tensor['normal_F'].detach().cpu() + 1.0) * 0.5,
+            (in_tensor['normal_B'].detach().cpu() + 1.0) * 0.5
+        ],
+                  dim=3), img_crop_path
+    )
+    rgb_norm_F = blend_rgb_norm(in_tensor["normal_F"], data)
+    rgb_norm_B = blend_rgb_norm(in_tensor["normal_B"], data)
+    img_overlap_path = osp.join(out_dir, cfg.name, f"png/{data['name']}_overlap.png")
+    torchvision.utils.save_image(
+        torch.cat([data["img_raw"], rgb_norm_F, rgb_norm_B], dim=-1) / 255., img_overlap_path
+    )
+    smpl_obj_lst = []
+    for idx in range(N_body):
+        smpl_obj = trimesh.Trimesh(
+            in_tensor["smpl_verts"].detach().cpu()[idx] * torch.tensor([1.0, -1.0, 1.0]),
+            in_tensor["smpl_faces"].detach().cpu()[0][:, [0, 2, 1]],
+            process=False,
+            maintains_order=True,
+        )
+        smpl_obj_path = f"{out_dir}/{cfg.name}/obj/{data['name']}_smpl_{idx:02d}.obj"
+        if not osp.exists(smpl_obj_path):
+            smpl_obj.export(smpl_obj_path)
+            smpl_obj.export(smpl_obj_path.replace(".obj", ".glb"))
+            smpl_info = {
+                "betas":
+                optimed_betas[idx].detach().cpu().unsqueeze(0),
+                "body_pose":
+                rotation_matrix_to_angle_axis(optimed_pose_mat[idx].detach()).cpu().unsqueeze(0),
+                "global_orient":
+                rotation_matrix_to_angle_axis(optimed_orient_mat[idx].detach()).cpu().unsqueeze(0),
+                "transl":
+                optimed_trans[idx].detach().cpu(),
+                "expression":
+                data["exp"][idx].cpu().unsqueeze(0),
+                "jaw_pose":
+                rotation_matrix_to_angle_axis(data["jaw_pose"][idx]).cpu().unsqueeze(0),
+                "left_hand_pose":
+                rotation_matrix_to_angle_axis(data["left_hand_pose"][idx]).cpu().unsqueeze(0),
+                "right_hand_pose":
+                rotation_matrix_to_angle_axis(data["right_hand_pose"][idx]).cpu().unsqueeze(0),
+                "scale":
+                data["scale"][idx].cpu(),
+            }
+            np.save(
+                smpl_obj_path.replace(".obj", ".npy"),
+                smpl_info,
+                allow_pickle=True,
             )
+        smpl_obj_lst.append(smpl_obj)
+    del optimizer_smpl
+    del optimed_betas
+    del optimed_orient
+    del optimed_pose
+    del optimed_trans
+    torch.cuda.empty_cache()
+    # ------------------------------------------------------------------------------------------------------------------
+    # clothing refinement
+    per_data_lst = []
+    batch_smpl_verts = in_tensor["smpl_verts"].detach() * torch.tensor([1.0, -1.0, 1.0],
+                                                                       device=device)
+    batch_smpl_faces = in_tensor["smpl_faces"].detach()[:, :, [0, 2, 1]]
+    in_tensor["depth_F"], in_tensor["depth_B"] = dataset.render_depth(
+        batch_smpl_verts, batch_smpl_faces
+    )
+    per_loop_lst = []
+    in_tensor["BNI_verts"] = []
+    in_tensor["BNI_faces"] = []
+    in_tensor["body_verts"] = []
+    in_tensor["body_faces"] = []
+    for idx in range(N_body):
+        final_path = f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_full.obj"
+        side_mesh = smpl_obj_lst[idx].copy()
+        face_mesh = smpl_obj_lst[idx].copy()
+        hand_mesh = smpl_obj_lst[idx].copy()
+        smplx_mesh = smpl_obj_lst[idx].copy()
+        # save normals, depths and masks
+        BNI_dict = save_normal_tensor(
+            in_tensor,
+            idx,
+            osp.join(out_dir, cfg.name, f"BNI/{data['name']}_{idx}"),
+            cfg.bni.thickness,
         )
+        # BNI process
+        BNI_object = BNI(
+            dir_path=osp.join(out_dir, cfg.name, "BNI"),
+            name=data["name"],
+            BNI_dict=BNI_dict,
+            cfg=cfg.bni,
+            device=device
+        )
+        BNI_object.extract_surface(False)
+        in_tensor["body_verts"].append(torch.tensor(smpl_obj_lst[idx].vertices).float())
+        in_tensor["body_faces"].append(torch.tensor(smpl_obj_lst[idx].faces).long())
+        # requires shape completion when low overlap
+        # replace SMPL by completed mesh as side_mesh
+        if cfg.bni.use_ifnet:
+            side_mesh_path = f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_IF.obj"
+            side_mesh = apply_face_mask(side_mesh, ~SMPLX_object.smplx_eyeball_fid_mask)
+            # mesh completion via IF-net
+            in_tensor.update(
+                dataset.depth_to_voxel({
+                    "depth_F": BNI_object.F_depth.unsqueeze(0), "depth_B":
+                    BNI_object.B_depth.unsqueeze(0)
+                })
             )
+            occupancies = VoxelGrid.from_mesh(side_mesh, cfg.vol_res, loc=[
+                0,
+            ] * 3, scale=2.0).data.transpose(2, 1, 0)
+            occupancies = np.flip(occupancies, axis=1)
+            in_tensor["body_voxels"] = torch.tensor(occupancies.copy()
+                                                   ).float().unsqueeze(0).to(device)
+            with torch.no_grad():
+                sdf = ifnet.reconEngine(netG=ifnet.netG, batch=in_tensor)
+                verts_IF, faces_IF = ifnet.reconEngine.export_mesh(sdf)
+            if ifnet.clean_mesh_flag:
+                verts_IF, faces_IF = clean_mesh(verts_IF, faces_IF)
+            side_mesh = trimesh.Trimesh(verts_IF, faces_IF)
+            side_mesh = remesh_laplacian(side_mesh, side_mesh_path)
+        else:
+            side_mesh = apply_vertex_mask(
+                side_mesh,
+                (
+                    SMPLX_object.front_flame_vertex_mask + SMPLX_object.smplx_mano_vertex_mask +
+                    SMPLX_object.eyeball_vertex_mask
+                ).eq(0).float(),
+            )
+            #register side_mesh to BNI surfaces
+            side_mesh = Meshes(
+                verts=[torch.tensor(side_mesh.vertices).float()],
+                faces=[torch.tensor(side_mesh.faces).long()],
+            ).to(device)
+            sm = SubdivideMeshes(side_mesh)
+            side_mesh = register(BNI_object.F_B_trimesh, sm(side_mesh), device)
+        side_verts = torch.tensor(side_mesh.vertices).float().to(device)
+        side_faces = torch.tensor(side_mesh.faces).long().to(device)
+        # Possion Fusion between SMPLX and BNI
+        # 1. keep the faces invisible to front+back cameras
+        # 2. keep the front-FLAME+MANO faces
+        # 3. remove eyeball faces
+        # export intermediate meshes
+        BNI_object.F_B_trimesh.export(f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_BNI.obj")
+        full_lst = []
+        if "face" in cfg.bni.use_smpl:
+            # only face
+            face_mesh = apply_vertex_mask(face_mesh, SMPLX_object.front_flame_vertex_mask)
+            face_mesh.vertices = face_mesh.vertices - np.array([0, 0, cfg.bni.thickness])
+            # remove face neighbor triangles
+            BNI_object.F_B_trimesh = part_removal(
+                BNI_object.F_B_trimesh,
+                face_mesh,
+                cfg.bni.face_thres,
+                device,
+                smplx_mesh,
+                region="face"
+            )
+            side_mesh = part_removal(
+                side_mesh, face_mesh, cfg.bni.face_thres, device, smplx_mesh, region="face"
+            )
+            face_mesh.export(f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_face.obj")
+            full_lst += [face_mesh]
+        if "hand" in cfg.bni.use_smpl and (True in data['hands_visibility'][idx]):
+            hand_mask = torch.zeros(SMPLX_object.smplx_verts.shape[0], )
+            if data['hands_visibility'][idx][0]:
+                hand_mask.index_fill_(
+                    0, torch.tensor(SMPLX_object.smplx_mano_vid_dict["left_hand"]), 1.0
+                )
+            if data['hands_visibility'][idx][1]:
+                hand_mask.index_fill_(
+                    0, torch.tensor(SMPLX_object.smplx_mano_vid_dict["right_hand"]), 1.0
                 )
+            # only hands
+            hand_mesh = apply_vertex_mask(hand_mesh, hand_mask)
+            # remove hand neighbor triangles
+            BNI_object.F_B_trimesh = part_removal(
+                BNI_object.F_B_trimesh,
+                hand_mesh,
+                cfg.bni.hand_thres,
+                device,
+                smplx_mesh,
+                region="hand"
+            )
+            side_mesh = part_removal(
+                side_mesh, hand_mesh, cfg.bni.hand_thres, device, smplx_mesh, region="hand"
+            )
+            hand_mesh.export(f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_hand.obj")
+            full_lst += [hand_mesh]
+        full_lst += [BNI_object.F_B_trimesh]
+        # initial side_mesh could be SMPLX or IF-net
+        side_mesh = part_removal(
+            side_mesh, sum(full_lst), 2e-2, device, smplx_mesh, region="", clean=False
+        )
+        full_lst += [side_mesh]
+        # # export intermediate meshes
+        BNI_object.F_B_trimesh.export(f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_BNI.obj")
+        side_mesh.export(f"{out_dir}/{cfg.name}/obj/{data['name']}_{idx}_side.obj")
+        final_mesh = poisson(
+            sum(full_lst),
+            final_path,
+            cfg.bni.poisson_depth,
+        )
+        print(
+            colored(f"\n Poisson completion to {Format.start} {final_path} {Format.end}", "yellow")
+        )
+        dataset.render.load_meshes(final_mesh.vertices, final_mesh.faces)
+        rotate_recon_lst = dataset.render.get_image(cam_type="four")
+        per_loop_lst.extend([in_tensor['image'][idx:idx + 1]] + rotate_recon_lst)
+        if cfg.bni.texture_src == 'image':
+            # coloring the final mesh (front: RGB pixels, back: normal colors)
+            final_colors = query_color(
+                torch.tensor(final_mesh.vertices).float(),
+                torch.tensor(final_mesh.faces).long(),
+                in_tensor["image"][idx:idx + 1],
+                device=device,
             )
+            final_mesh.visual.vertex_colors = final_colors
+            final_mesh.export(final_path)
+            final_mesh.export(final_path.replace(".obj", ".glb"))
+        elif cfg.bni.texture_src == 'SD':
+            # !TODO: add texture from Stable Diffusion
+            pass
+    if len(per_loop_lst) > 0:
+        per_data_lst.append(get_optim_grid_image(per_loop_lst, None, nrow=5, type="cloth"))
+        per_data_lst[-1].save(osp.join(out_dir, cfg.name, f"png/{data['name']}_cloth.png"))
+        # for video rendering
+        in_tensor["BNI_verts"].append(torch.tensor(final_mesh.vertices).float())
+        in_tensor["BNI_faces"].append(torch.tensor(final_mesh.faces).long())
+        os.makedirs(osp.join(out_dir, cfg.name, "vid"), exist_ok=True)
+        in_tensor["uncrop_param"] = data["uncrop_param"]
+        in_tensor["img_raw"] = data["img_raw"]
+        torch.save(in_tensor, osp.join(out_dir, cfg.name, f"vid/{data['name']}_in_tensor.pt"))
+    smpl_glb_path = smpl_obj_path.replace(".obj", ".glb")
+    # smpl_npy_path = smpl_obj_path.replace(".obj", ".npy")
+    refine_obj_path = final_path
+    refine_glb_path = final_path.replace(".obj", ".glb")
+    overlap_path = img_overlap_path
+    vis_tensor_path = osp.join(out_dir, cfg.name, f"vid/{data['name']}_in_tensor.pt")
+    # clean all the variables
+    for element in dir():
+        if 'path' not in element:
+            del locals()[element]
+    import gc
+    gc.collect()
+    torch.cuda.empty_cache()
+    return [
+        smpl_glb_path, refine_glb_path, refine_obj_path, overlap_path, vis_tensor_path
+    ]

apps/multi_render.py DELETED Viewed

@@ -1,25 +0,0 @@
-import argparse
-import torch
-from lib.common.render import Render
-root = "./results/econ/vid"
-# loading cfg file
-parser = argparse.ArgumentParser()
-parser.add_argument("-n", "--name", type=str, default="")
-parser.add_argument("-g", "--gpu", type=int, default=0)
-args = parser.parse_args()
-in_tensor = torch.load(f"{root}/{args.name}_in_tensor.pt")
-render = Render(size=512, device=torch.device(f"cuda:{args.gpu}"))
-# visualize the final results in self-rotation mode
-verts_lst = in_tensor["body_verts"] + in_tensor["BNI_verts"]
-faces_lst = in_tensor["body_faces"] + in_tensor["BNI_faces"]
-# self-rotated video
-render.load_meshes(verts_lst, faces_lst)
-render.get_rendered_video_multi(in_tensor, f"{root}/{args.name}_cloth.mp4")

docker-compose.yaml DELETED Viewed

@@ -1,19 +0,0 @@
-# build Image from Docker Hub
-version: "2.4"
-services:
-  econ:
-    container_name: econ-container
-    image: teddy12155555/econ:v1
-    runtime: nvidia
-    environment:
-      - NVIDIA_VISIBLE_DEVICES=all
-      - DISPLAY=${DISPLAY}
-    stdin_open: true
-    tty: true
-    volumes:
-      - .:/root/code
-      - /tmp/.X11-unix:/tmp/.X11-unix
-    ports:
-      - "8000:8000"
-    privileged: true
-    command: "bash"

docs/installation-docker.md DELETED Viewed

@@ -1,80 +0,0 @@
-## Getting started
-Start by cloning the repo:
-```bash
-git clone --depth 1 git@github.com:YuliangXiu/ECON.git
-cd ECON
-```
-## Environment
-- **GPU Memory > 12GB**
-start with [docker compose](https://docs.docker.com/compose/)
-```bash
-# you can change your container name by passing --name "parameter"
-docker compose run [--name myecon] econ
-```
-## Docker container's shell
-```bash
-# activate the pre-build env
-cd code
-conda activate econ
-# install libmesh & libvoxelize
-cd lib/common/libmesh
-python setup.py build_ext --inplace
-cd ../libvoxelize
-python setup.py build_ext --inplace
-```
-## Register at [ICON's website](https://icon.is.tue.mpg.de/)
-![Register](../assets/register.png)
-Required:
-- [SMPL](http://smpl.is.tue.mpg.de/): SMPL Model (Male, Female)
-- [SMPL-X](http://smpl-x.is.tue.mpg.de/): SMPL-X Model, used for training
-- [SMPLIFY](http://smplify.is.tue.mpg.de/): SMPL Model (Neutral)
-- [PIXIE](https://icon.is.tue.mpg.de/user.php): PIXIE SMPL-X estimator
-:warning: Click **Register now** on all dependencies, then you can download them all with **ONE** account.
-## Downloading required models and extra data
-```bash
-cd ~/code
-bash fetch_data.sh # requires username and password
-```
-## :whale2: **todo**
-- **Image Environment Infos**
-    - Ubuntu 18
-    - CUDA = 11.3
-    - Python = 3.8
-- [X] pre-built image with docker compose
-- [ ] docker run command, Dockerfile
-- [ ] verify on WSL (Windows)
-## Citation
-:+1: Please consider citing these awesome HPS approaches: PyMAF-X, PIXIE
-```
-@article{pymafx2022,
-  title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
-  author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
-  journal={arXiv preprint arXiv:2207.06400},
-  year={2022}
-}
-@inproceedings{PIXIE:2021,
-  title={Collaborative Regression of Expressive Bodies using Moderation},
-  author={Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black},
-  booktitle={International Conference on 3D Vision (3DV)},
-  year={2021}
-}
-```

docs/installation-ubuntu.md DELETED Viewed

@@ -1,80 +0,0 @@
-## Getting started
-Start by cloning the repo:
-```bash
-git clone --depth 1 git@github.com:YuliangXiu/ECON.git
-cd ECON
-```
-## Environment
-- Ubuntu 20 / 18, (Windows as well, see [issue#7](https://github.com/YuliangXiu/ECON/issues/7))
-- **CUDA=11.6, GPU Memory > 12GB**
-- Python = 3.8
-- PyTorch >= 1.13.0 (official [Get Started](https://pytorch.org/get-started/locally/))
-- Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
-- PyTorch3D = 0.7.1 (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
-```bash
-sudo apt-get install libeigen3-dev ffmpeg
-# install required packages
-cd ECON
-conda env create -f environment.yaml
-conda activate econ
-pip install -r requirements.txt
-# the installation(incl. compilation) of PyTorch3D will take ~20min
-pip install git+https://github.com/facebookresearch/pytorch3d.git@v0.7.1
-# install libmesh & libvoxelize
-cd lib/common/libmesh
-python setup.py build_ext --inplace
-cd ../libvoxelize
-python setup.py build_ext --inplace
-```
-## Register at [ICON's website](https://icon.is.tue.mpg.de/)
-![Register](../assets/register.png)
-Required:
-- [SMPL](http://smpl.is.tue.mpg.de/): SMPL Model (Male, Female)
-- [SMPL-X](http://smpl-x.is.tue.mpg.de/): SMPL-X Model, used for training
-- [SMPLIFY](http://smplify.is.tue.mpg.de/): SMPL Model (Neutral)
-- [PIXIE](https://icon.is.tue.mpg.de/user.php): PIXIE SMPL-X estimator
-:warning: Click **Register now** on all dependencies, then you can download them all with **ONE** account.
-## Downloading required models and extra data
-```bash
-cd ECON
-bash fetch_data.sh # requires username and password
-```
-## Citation
-:+1: Please consider citing these awesome HPS approaches: PyMAF-X, PIXIE
-```
-@article{pymafx2022,
-  title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
-  author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
-  journal={arXiv preprint arXiv:2207.06400},
-  year={2022}
-}
-@inproceedings{PIXIE:2021,
-  title={Collaborative Regression of Expressive Bodies using Moderation},
-  author={Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black},
-  booktitle={International Conference on 3D Vision (3DV)},
-  year={2021}
-}
-```

docs/installation-windows.md DELETED Viewed

@@ -1,100 +0,0 @@
-# Windows installation tutorial
-Another [issue#16](https://github.com/YuliangXiu/ECON/issues/16) shows the whole process to deploy ECON on *Windows*
-## Dependencies and Installation
-- Use [Anaconda](https://www.anaconda.com/products/distribution)
-- NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads)
-- [Wget for Windows](https://eternallybored.org/misc/wget/1.21.3/64/wget.exe)
-- Create a new folder on your C drive and rename it "wget" and move the downloaded "wget.exe" over there.
-- Add the path to your wget folder to your system environment variables at `Environment Variables > System Variables Path > Edit environment variable`
-![image](https://user-images.githubusercontent.com/34035011/210986038-39dbb7a1-12ef-4be9-9af4-5f658c6beb65.png)
-- Install [Git for Windows 64-bit](https://git-scm.com/download/win)
-- [Visual Studio Community 2022](https://visualstudio.microsoft.com/) (Make sure to check all the boxes as shown in the image below)
-![image](https://user-images.githubusercontent.com/34035011/210983023-4e5a0024-68f0-4adb-8089-6ff598aec220.PNG)
-## Getting started
-Start by cloning the repo:
-```bash
-git clone https://github.com/yuliangxiu/ECON.git
-cd ECON
-```
-## Environment
-- Windows 10 / 11
-- **CUDA=11.3**
-- Python = 3.8
-- PyTorch >= 1.12.1 (official [Get Started](https://pytorch.org/get-started/locally/))
-- Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
-- PyTorch3D = 0.7.1 (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
-```bash
-# install required packages
-cd ECON
-conda env create -f environment-windows.yaml
-conda activate econ
-# install pytorch and cupy
-pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
-pip install -r requirements.txt
-pip install cupy-cuda11x
-pip install git+https://github.com/facebookresearch/pytorch3d.git@v0.7.1
-# install libmesh & libvoxelize
-cd lib/common/libmesh
-python setup.py build_ext --inplace
-cd ../libvoxelize
-python setup.py build_ext --inplace
-```
-## Register at [ICON's website](https://icon.is.tue.mpg.de/)
-![Register](../assets/register.png)
-Required:
-- [SMPL](http://smpl.is.tue.mpg.de/): SMPL Model (Male, Female)
-- [SMPL-X](http://smpl-x.is.tue.mpg.de/): SMPL-X Model, used for training
-- [SMPLIFY](http://smplify.is.tue.mpg.de/): SMPL Model (Neutral)
-- [PIXIE](https://icon.is.tue.mpg.de/user.php): PIXIE SMPL-X estimator
-:warning: Click **Register now** on all dependencies, then you can download them all with **ONE** account.
-## Downloading required models and extra data (make sure to install git and wget for windows for this to work)
-```bash
-cd ECON
-bash fetch_data.sh # requires username and password
-```
-## Citation
-:+1: Please consider citing these awesome HPS approaches: PyMAF-X, PIXIE
-```
-@article{pymafx2022,
-  title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
-  author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
-  journal={arXiv preprint arXiv:2207.06400},
-  year={2022}
-}
-@inproceedings{PIXIE:2021,
-  title={Collaborative Regression of Expressive Bodies using Moderation},
-  author={Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black},
-  booktitle={International Conference on 3D Vision (3DV)},
-  year={2021}
-}
-```

docs/testing.md DELETED Viewed

@@ -1,71 +0,0 @@
-# Evaluation
-## Testing Data
-![dataset](../assets/dataset.png)
-- OOD pose (CAPE, [download](https://github.com/YuliangXiu/ICON/blob/master/docs/evaluation.md#cape-testset)): [`pose.txt`](../pose.txt)
-- OOD outfits (RenderPeople, [link](https://renderpeople.com/)): [`loose.txt`](../loose.txt)
-## Run the evaluation
-```bash
-# Benchmark of ECON_{IF}, which uses IF-Net+ for completion
-export CUDA_VISIBLE_DEVICES=0; python -m apps.benchmark -ifnet
-# Benchmark of ECON_{EX}, which uses registered SMPL for completion
-export CUDA_VISIBLE_DEVICES=1; python -m apps.benchmark
-```
-## Benchmark
-|   Method    |  $\text{ECON}_\text{IF}$  | $\text{ECON}_\text{EX}$ |
-| :---------: | :-----------------------: | :---------------------: |
-|             |     OOD poses (CAPE)      |                         |
-| Chamfer(cm) |           0.996           |        **0.926**        |
-|   P2S(cm)   |           0.967           |        **0.917**        |
-| Normal(L2)  |          0.0413           |       **0.0367**        |
-|             | OOD oufits (RenderPeople) |                         |
-| Chamfer(cm) |           1.401           |        **1.342**        |
-|   P2S(cm)   |         **1.422**         |          1.458          |
-| Normal(L2)  |          0.0516           |       **0.0478**        |
-**\*OOD: Out-of-Distribution**
-## Citation
-:+1: Please cite these CAPE-related papers
-```
-@inproceedings{xiu2022icon,
-  title     = {{ICON}: {I}mplicit {C}lothed humans {O}btained from {N}ormals},
-  author    = {Xiu, Yuliang and Yang, Jinlong and Tzionas, Dimitrios and Black, Michael J.},
-  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
-  month     = {June},
-  year      = {2022},
-  pages     = {13296-13306}
-}
-@inproceedings{CAPE:CVPR:20,
-  title = {{Learning to Dress 3D People in Generative Clothing}},
-  author = {Ma, Qianli and Yang, Jinlong and Ranjan, Anurag and Pujades, Sergi and Pons-Moll, Gerard and Tang, Siyu and Black, Michael J.},
-  booktitle = {Computer Vision and Pattern Recognition (CVPR)},
-  month = June,
-  year = {2020},
-  month_numeric = {6}
-}
-@article{Pons-Moll:Siggraph2017,
-  title = {ClothCap: Seamless 4D Clothing Capture and Retargeting},
-  author = {Pons-Moll, Gerard and Pujades, Sergi and Hu, Sonny and Black, Michael},
-  journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH)},
-  volume = {36},
-  number = {4},
-  year = {2017},
-  note = {Two first authors contributed equally},
-  crossref = {},
-  url = {http://dx.doi.org/10.1145/3072959.3073711}
-}
-```

docs/tricks.md DELETED Viewed

@@ -1,29 +0,0 @@
-## Technical tricks to improve or accelerate ECON
-### If the reconstructed geometry is not satisfying, play with the adjustable parameters in _config/econ.yaml_
-- `use_smpl: ["hand"]`
-  - [ ]: don't use either hands or face parts from SMPL-X
-  - ["hand"]: only use the **visible** hands from SMPL-X
-  - ["hand", "face"]: use both **visible** hands and face from SMPL-X
-- `thickness: 2cm`
-  - could be increased accordingly in case final reconstruction **xx_full.obj** looks flat
-- `k: 4`
-  - could be reduced accordingly in case the surface of **xx_full.obj** has discontinous artifacts
-- `hps_type: PIXIE`
-  - "pixie": more accurate for face and hands
-  - "pymafx": more robust for challenging poses
-- `texture_src: image`
-  - "image": direct mapping the aligned pixels to final mesh
-  - "SD": use Stable Diffusion to generate full texture (TODO)
-### To accelerate the inference, you could
-- `use_ifnet: False`
-  - True: use IF-Nets+ for mesh completion ( $\text{ECON}_\text{IF}$ - Better quality, **~2min / img**)
-  - False: use SMPL-X for mesh completion ( $\text{ECON}_\text{EX}$ - Faster speed, **~1.8min / img**)
-```bash
-# For single-person image-based reconstruction (w/o all visualization steps, 1.5min)
-python -m apps.infer -cfg ./configs/econ.yaml -in_dir ./examples -out_dir ./results -novis
-```

environment-windows.yaml DELETED Viewed

@@ -1,18 +0,0 @@
-name: econ
-channels:
-  - nvidia
-  - pytorch
-  - conda-forge
-  - fvcore
-  - iopath
-  - bottler
-  - defaults
-dependencies:
-  - python=3.8
-  - pytorch-cuda=11.3
-  - fvcore
-  - iopath
-  - cupy
-  - cython
-  - pip

environment.yaml DELETED Viewed

@@ -1,21 +0,0 @@
-name: econ
-channels:
-  - pytorch
-  - nvidia
-  - conda-forge
-  - fvcore
-  - iopath
-  - bottler
-  - defaults
-dependencies:
-  - python=3.8
-  - pytorch-cuda=11.6
-  - pytorch=1.13.0
-  - nvidiacub
-  - torchvision
-  - fvcore
-  - iopath
-  - pyembree
-  - cupy
-  - cython
-  - pip

fetch_data.sh DELETED Viewed

@@ -1,60 +0,0 @@
-#!/bin/bash
-urle () { [[ "${1}" ]] || return 1; local LANG=C i x; for (( i = 0; i < ${#1}; i++ )); do x="${1:i:1}"; [[ "${x}" == [a-zA-Z0-9.~-] ]] && echo -n "${x}" || printf '%%%02X' "'${x}"; done; echo; }
-mkdir -p data/smpl_related/models
-# username and password input
-echo -e "\nYou need to register at https://icon.is.tue.mpg.de/, according to Installation Instruction."
-read -p "Username (ICON):" username
-read -p "Password (ICON):" password
-username=$(urle $username)
-password=$(urle $password)
-# SMPL (Male, Female)
-echo -e "\nDownloading SMPL..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=smpl&sfile=SMPL_python_v.1.0.0.zip&resume=1' -O './data/smpl_related/models/SMPL_python_v.1.0.0.zip' --no-check-certificate --continue
-unzip data/smpl_related/models/SMPL_python_v.1.0.0.zip -d data/smpl_related/models
-mv data/smpl_related/models/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl data/smpl_related/models/smpl/SMPL_FEMALE.pkl
-mv data/smpl_related/models/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl data/smpl_related/models/smpl/SMPL_MALE.pkl
-cd data/smpl_related/models
-rm -rf *.zip __MACOSX smpl/models smpl/smpl_webuser
-cd ../../..
-# SMPL (Neutral, from SMPLIFY)
-echo -e "\nDownloading SMPLify..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=smplify&sfile=mpips_smplify_public_v2.zip&resume=1' -O './data/smpl_related/models/mpips_smplify_public_v2.zip' --no-check-certificate --continue
-unzip data/smpl_related/models/mpips_smplify_public_v2.zip -d data/smpl_related/models
-mv data/smpl_related/models/smplify_public/code/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl data/smpl_related/models/smpl/SMPL_NEUTRAL.pkl
-cd data/smpl_related/models
-rm -rf *.zip smplify_public
-cd ../../..
-# SMPL-X
-echo -e "\nDownloading SMPL-X..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=smplx&sfile=models_smplx_v1_1.zip&resume=1' -O './data/smpl_related/models/models_smplx_v1_1.zip' --no-check-certificate --continue
-unzip data/smpl_related/models/models_smplx_v1_1.zip -d data/smpl_related
-rm -f data/smpl_related/models/models_smplx_v1_1.zip
-# ECON
-echo -e "\nDownloading ECON..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=icon&sfile=econ_data.zip&resume=1' -O './data/econ_data.zip' --no-check-certificate --continue
-cd data && unzip econ_data.zip
-mv smpl_data smpl_related/
-rm -f econ_data.zip
-cd ..
-mkdir -p data/HPS
-# PIXIE
-echo -e "\nDownloading PIXIE..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=icon&sfile=HPS/pixie_data.zip&resume=1' -O './data/HPS/pixie_data.zip' --no-check-certificate --continue
-cd data/HPS && unzip pixie_data.zip
-rm -f pixie_data.zip
-cd ../..
-# PyMAF-X
-echo -e "\nDownloading PyMAF-X..."
-wget --post-data "username=$username&password=$password" 'https://download.is.tue.mpg.de/download.php?domain=icon&sfile=HPS/pymafx_data.zip&resume=1' -O './data/HPS/pymafx_data.zip' --no-check-certificate --continue
-cd data/HPS && unzip pymafx_data.zip
-rm -f pymafx_data.zip
-cd ../..

lib/dataset/TestDataset.py CHANGED Viewed

@@ -49,8 +49,7 @@ ImageFile.LOAD_TRUNCATED_IMAGES = True
 class TestDataset:
     def __init__(self, cfg, device):
-        self.image_dir = cfg["image_dir"]
-        self.seg_dir = cfg["seg_dir"]
         self.use_seg = cfg["use_seg"]
         self.hps_type = cfg["hps_type"]
         self.smpl_type = "smplx"
@@ -60,11 +59,7 @@ class TestDataset:
         self.device = device
-        keep_lst = sorted(glob.glob(f"{self.image_dir}/*"))
-        img_fmts = ["jpg", "png", "jpeg", "JPG", "bmp", "exr"]
-        self.subject_list = sorted([item for item in keep_lst if item.split(".")[-1] in img_fmts],
-                                   reverse=False)
         # smpl related
         self.smpl_data = SMPLX()

 class TestDataset:
     def __init__(self, cfg, device):
+        self.image_path = cfg["image_path"]
         self.use_seg = cfg["use_seg"]
         self.hps_type = cfg["hps_type"]
         self.smpl_type = "smplx"
         self.device = device
+        self.subject_list = [self.image_path]
         # smpl related
         self.smpl_data = SMPLX()

lib/dataset/mesh_util.py CHANGED Viewed

@@ -30,6 +30,7 @@ from pytorch3d.loss import mesh_laplacian_smoothing, mesh_normal_consistency
 from pytorch3d.renderer.mesh import rasterize_meshes
 from pytorch3d.structures import Meshes
 from scipy.spatial import cKDTree
 import lib.smplx as smplx
 from lib.common.render_utils import Pytorch3dRasterizer, face_vertices
@@ -43,28 +44,70 @@ class Format:
 class SMPLX:
     def __init__(self):
-        self.current_dir = osp.join(osp.dirname(__file__), "../../data/smpl_related")
-        self.smpl_verts_path = osp.join(self.current_dir, "smpl_data/smpl_verts.npy")
-        self.smpl_faces_path = osp.join(self.current_dir, "smpl_data/smpl_faces.npy")
-        self.smplx_verts_path = osp.join(self.current_dir, "smpl_data/smplx_verts.npy")
-        self.smplx_faces_path = osp.join(self.current_dir, "smpl_data/smplx_faces.npy")
-        self.cmap_vert_path = osp.join(self.current_dir, "smpl_data/smplx_cmap.npy")
-        self.smplx_to_smplx_path = osp.join(self.current_dir, "smpl_data/smplx_to_smpl.pkl")
-        self.smplx_eyeball_fid_path = osp.join(self.current_dir, "smpl_data/eyeball_fid.npy")
-        self.smplx_fill_mouth_fid_path = osp.join(self.current_dir, "smpl_data/fill_mouth_fid.npy")
-        self.smplx_flame_vid_path = osp.join(
-            self.current_dir, "smpl_data/FLAME_SMPLX_vertex_ids.npy"
         )
-        self.smplx_mano_vid_path = osp.join(self.current_dir, "smpl_data/MANO_SMPLX_vertex_ids.pkl")
         self.smpl_vert_seg_path = osp.join(
             osp.dirname(__file__), "../../lib/common/smpl_vert_segmentation.json"
         )
-        self.front_flame_path = osp.join(self.current_dir, "smpl_data/FLAME_face_mask_ids.npy")
-        self.smplx_vertex_lmkid_path = osp.join(
-            self.current_dir, "smpl_data/smplx_vertex_lmkid.npy"
         )
         self.smplx_faces = np.load(self.smplx_faces_path)
@@ -106,8 +149,6 @@ class SMPLX:
         self.smplx_to_smpl = cPickle.load(open(self.smplx_to_smplx_path, "rb"))
-        self.model_dir = osp.join(self.current_dir, "models")
         self.ghum_smpl_pairs = torch.tensor([(0, 24), (2, 26), (5, 25), (7, 28), (8, 27), (11, 16),
                                              (12, 17), (13, 18), (14, 19), (15, 20), (16, 21),
                                              (17, 39), (18, 44), (19, 36), (20, 41), (21, 35),
@@ -151,7 +192,7 @@ class SMPLX:
 model_init_params = dict(
     gender="male",
     model_type="smplx",
-    model_path=SMPLX().model_dir,
     create_global_orient=False,
     create_body_pose=False,
     create_betas=False,

 from pytorch3d.renderer.mesh import rasterize_meshes
 from pytorch3d.structures import Meshes
 from scipy.spatial import cKDTree
+from huggingface_hub import hf_hub_download
 import lib.smplx as smplx
 from lib.common.render_utils import Pytorch3dRasterizer, face_vertices
 class SMPLX:
     def __init__(self):
+        self.smpl_verts_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smpl_verts.npy"
+        )
+        self.smpl_faces_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smpl_faces.npy"
+        )
+        self.smplx_verts_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smplx_verts.npy"
+        )
+        self.smplx_faces_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smplx_faces.npy"
+        )
+        self.cmap_vert_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smplx_cmap.npy"
+        )
+        self.smplx_to_smplx_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smplx_to_smpl.pkl"
+        )
+        self.smplx_eyeball_fid_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/eyeball_fid.npy"
+        )
+        self.smplx_fill_mouth_fid_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/fill_mouth_fid.npy"
+        )
+        self.smplx_flame_vid_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/FLAME_SMPLX_vertex_ids.npy"
+        )
+        self.smplx_mano_vid_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/MANO_SMPLX_vertex_ids.pkl"
         )
         self.smpl_vert_seg_path = osp.join(
             osp.dirname(__file__), "../../lib/common/smpl_vert_segmentation.json"
         )
+        self.front_flame_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/FLAME_face_mask_ids.npy"
+        )
+        self.smplx_vertex_lmkid_path = hf_hub_download(
+            repo_id="Yuliang/SMPLX",
+            use_auth_token=os.environ["ICON"],
+            filename="smpl_data/smplx_vertex_lmkid.npy"
         )
         self.smplx_faces = np.load(self.smplx_faces_path)
         self.smplx_to_smpl = cPickle.load(open(self.smplx_to_smplx_path, "rb"))
         self.ghum_smpl_pairs = torch.tensor([(0, 24), (2, 26), (5, 25), (7, 28), (8, 27), (11, 16),
                                              (12, 17), (13, 18), (14, 19), (15, 20), (16, 21),
                                              (17, 39), (18, 44), (19, 36), (20, 41), (21, 35),
 model_init_params = dict(
     gender="male",
     model_type="smplx",
+    model_path="Yuliang/SMPLX",
     create_global_orient=False,
     create_body_pose=False,
     create_betas=False,

lib/pixielib/utils/config.py CHANGED Viewed

@@ -6,6 +6,7 @@ import os
 import yaml
 from yacs.config import CfgNode as CN
 cfg = CN()
@@ -13,7 +14,9 @@ abs_pixie_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".
 cfg.pixie_dir = abs_pixie_dir
 cfg.device = "cuda"
 cfg.device_id = "0"
-cfg.pretrained_modelpath = os.path.join(cfg.pixie_dir, "data/HPS/pixie_data", "pixie_model.tar")
 # smplx parameter settings
 cfg.params = CN()
 cfg.params.body_list = ["body_cam", "global_pose", "partbody_pose", "neck_pose"]
@@ -29,38 +32,47 @@ cfg.params.hand_share_list = [
 # Options for Body model
 # ---------------------------------------------------------------------------- #
 cfg.model = CN()
-cfg.model.topology_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "SMPL_X_template_FLAME_uv.obj"
 )
-cfg.model.topology_smplxtex_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "smplx_tex.obj"
 )
-cfg.model.topology_smplx_hand_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "smplx_hand.obj"
 )
-cfg.model.smplx_model_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "SMPLX_NEUTRAL_2020.npz"
 )
-cfg.model.face_mask_path = os.path.join(cfg.pixie_dir, "data/HPS/pixie_data", "uv_face_mask.png")
-cfg.model.face_eye_mask_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "uv_face_eye_mask.png"
 )
-cfg.model.tex_path = os.path.join(cfg.pixie_dir, "data/HPS/pixie_data", "FLAME_albedo_from_BFM.npz")
-cfg.model.extra_joint_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "smplx_extra_joints.yaml"
 )
-cfg.model.j14_regressor_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "SMPLX_to_J14.pkl"
 )
-cfg.model.flame2smplx_cached_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "flame2smplx_tex_1024.npy"
 )
-cfg.model.smplx_tex_path = os.path.join(cfg.pixie_dir, "data/HPS/pixie_data", "smplx_tex.png")
-cfg.model.mano_ids_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "MANO_SMPLX_vertex_ids.pkl"
 )
-cfg.model.flame_ids_path = os.path.join(
-    cfg.pixie_dir, "data/HPS/pixie_data", "SMPL-X__FLAME_vertex_ids.npy"
 )
 cfg.model.uv_size = 256
 cfg.model.n_shape = 200

 import yaml
 from yacs.config import CfgNode as CN
+from huggingface_hub import hf_hub_download
 cfg = CN()
 cfg.pixie_dir = abs_pixie_dir
 cfg.device = "cuda"
 cfg.device_id = "0"
+cfg.pretrained_modelpath = hf_hub_download(
+    repo_id="Yuliang/PIXIE", filename="pixie_model.tar", use_auth_token=os.environ["ICON"]
+)
 # smplx parameter settings
 cfg.params = CN()
 cfg.params.body_list = ["body_cam", "global_pose", "partbody_pose", "neck_pose"]
 # Options for Body model
 # ---------------------------------------------------------------------------- #
 cfg.model = CN()
+cfg.model.topology_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE",
+    use_auth_token=os.environ["ICON"],
+    filename="SMPL_X_template_FLAME_uv.obj"
+)
+cfg.model.topology_smplxtex_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="smplx_tex.obj"
+)
+cfg.model.topology_smplx_hand_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="smplx_hand.obj"
 )
+cfg.model.smplx_model_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="SMPLX_NEUTRAL_2020.npz"
 )
+cfg.model.face_mask_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="uv_face_mask.png"
 )
+cfg.model.face_eye_mask_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="uv_face_eye_mask.png"
 )
+cfg.model.extra_joint_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="smplx_extra_joints.yaml"
 )
+cfg.model.j14_regressor_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="SMPLX_to_J14.pkl"
 )
+cfg.model.flame2smplx_cached_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="flame2smplx_tex_1024.npy"
 )
+cfg.model.smplx_tex_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE", use_auth_token=os.environ["ICON"], filename="smplx_tex.png"
 )
+cfg.model.mano_ids_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE",
+    use_auth_token=os.environ["ICON"],
+    filename="MANO_SMPLX_vertex_ids.pkl"
 )
+cfg.model.flame_ids_path = hf_hub_download(
+    repo_id="Yuliang/PIXIE",
+    use_auth_token=os.environ["ICON"],
+    filename="SMPL-X__FLAME_vertex_ids.npy"
 )
 cfg.model.uv_size = 256
 cfg.model.n_shape = 200

lib/smplx/body_models.py CHANGED Viewed

@@ -1015,12 +1015,12 @@ class SMPLX(SMPLH):
         """
         # Load the model
-        if osp.isdir(model_path):
-            model_fn = "SMPLX_{}.{ext}".format(gender.upper(), ext=ext)
-            smplx_path = os.path.join(model_path, model_fn)
-        else:
-            smplx_path = model_path
-        assert osp.exists(smplx_path), "Path {} does not exist!".format(smplx_path)
         if ext == "pkl":
             with open(smplx_path, "rb") as smplx_file:

         """
         # Load the model
+        from huggingface_hub import hf_hub_download
+        model_fn = "SMPLX_{}.{ext}".format(gender.upper(), ext=ext)
+        smplx_path = hf_hub_download(
+            repo_id=model_path, use_auth_token=os.environ["ICON"], filename=f"models/{model_fn}"
+        )
         if ext == "pkl":
             with open(smplx_path, "rb") as smplx_file:

loose.txt DELETED Viewed

@@ -1,100 +0,0 @@
-renderpeople/rp_yasmin_posed_007
-renderpeople/rp_victoria_posed_006
-renderpeople/rp_tilda_posed_005
-renderpeople/rp_tiffany_posed_015
-renderpeople/rp_tanja_posed_018
-renderpeople/rp_stephanie_posed_010
-renderpeople/rp_stacy_posed_002
-renderpeople/rp_serena_posed_027
-renderpeople/rp_serena_posed_024
-renderpeople/rp_seiko_posed_031
-renderpeople/rp_seiko_posed_015
-renderpeople/rp_saki_posed_033
-renderpeople/rp_rosy_posed_014
-renderpeople/rp_rosy_posed_001
-renderpeople/rp_roberta_posed_022
-renderpeople/rp_rick_posed_016
-renderpeople/rp_ray_posed_007
-renderpeople/rp_ramon_posed_002
-renderpeople/rp_ralph_posed_013
-renderpeople/rp_philip_posed_030
-renderpeople/rp_petra_posed_008
-renderpeople/rp_olivia_posed_014
-renderpeople/rp_olivia_posed_007
-renderpeople/rp_naomi_posed_034
-renderpeople/rp_naomi_posed_030
-renderpeople/rp_martha_posed_002
-renderpeople/rp_martha_posed_001
-renderpeople/rp_marleen_posed_002
-renderpeople/rp_lina_posed_004
-renderpeople/rp_kylie_posed_017
-renderpeople/rp_kylie_posed_006
-renderpeople/rp_kylie_posed_003
-renderpeople/rp_kent_posed_005
-renderpeople/rp_kent_posed_002
-renderpeople/rp_julia_posed_022
-renderpeople/rp_julia_posed_014
-renderpeople/rp_judy_posed_002
-renderpeople/rp_jessica_posed_058
-renderpeople/rp_jessica_posed_022
-renderpeople/rp_jennifer_posed_003
-renderpeople/rp_janna_posed_046
-renderpeople/rp_janna_posed_043
-renderpeople/rp_janna_posed_034
-renderpeople/rp_janna_posed_019
-renderpeople/rp_janett_posed_016
-renderpeople/rp_jamal_posed_012
-renderpeople/rp_helen_posed_037
-renderpeople/rp_fiona_posed_002
-renderpeople/rp_felice_posed_005
-renderpeople/rp_felice_posed_004
-renderpeople/rp_eve_posed_003
-renderpeople/rp_eve_posed_002
-renderpeople/rp_eve_posed_001
-renderpeople/rp_eric_posed_048
-renderpeople/rp_emma_posed_029
-renderpeople/rp_ellie_posed_015
-renderpeople/rp_ellie_posed_014
-renderpeople/rp_debra_posed_016
-renderpeople/rp_debra_posed_014
-renderpeople/rp_debra_posed_004
-renderpeople/rp_corey_posed_020
-renderpeople/rp_corey_posed_009
-renderpeople/rp_corey_posed_004
-renderpeople/rp_cody_posed_016
-renderpeople/rp_claudia_posed_034
-renderpeople/rp_claudia_posed_033
-renderpeople/rp_claudia_posed_024
-renderpeople/rp_claudia_posed_025
-renderpeople/rp_cindy_posed_020
-renderpeople/rp_christine_posed_023
-renderpeople/rp_christine_posed_022
-renderpeople/rp_christine_posed_020
-renderpeople/rp_christine_posed_010
-renderpeople/rp_carla_posed_016
-renderpeople/rp_caren_posed_009
-renderpeople/rp_caren_posed_008
-renderpeople/rp_brandon_posed_006
-renderpeople/rp_belle_posed_001
-renderpeople/rp_beatrice_posed_025
-renderpeople/rp_beatrice_posed_024
-renderpeople/rp_beatrice_posed_023
-renderpeople/rp_beatrice_posed_021
-renderpeople/rp_beatrice_posed_019
-renderpeople/rp_beatrice_posed_017
-renderpeople/rp_anna_posed_008
-renderpeople/rp_anna_posed_007
-renderpeople/rp_anna_posed_006
-renderpeople/rp_anna_posed_003
-renderpeople/rp_anna_posed_001
-renderpeople/rp_alvin_posed_016
-renderpeople/rp_alison_posed_028
-renderpeople/rp_alison_posed_024
-renderpeople/rp_alison_posed_017
-renderpeople/rp_alexandra_posed_022
-renderpeople/rp_alexandra_posed_023
-renderpeople/rp_alexandra_posed_019
-renderpeople/rp_alexandra_posed_018
-renderpeople/rp_alexandra_posed_013
-renderpeople/rp_alexandra_posed_012
-renderpeople/rp_alexandra_posed_011

packages.txt ADDED Viewed

	@@ -0,0 +1,13 @@

+libgl1
+freeglut3-dev
+unzip
+ffmpeg
+libsm6
+libxext6
+libgl1-mesa-dri
+libegl1-mesa
+libgbm1
+build-essential
+python-wheel
+libturbojpeg
+libeigen3-dev

pose.txt DELETED Viewed

@@ -1,100 +0,0 @@
-cape/00215-jerseyshort-pose_model-000200
-cape/00134-longlong-ballet4_trial2-000250
-cape/00134-longlong-badminton_trial1-000230
-cape/00134-longlong-frisbee_trial1-000190
-cape/03375-shortlong-ballet1_trial1-000210
-cape/03375-longlong-babysit_trial2-000110
-cape/00134-shortlong-stretch_trial1-000310
-cape/03375-shortshort-lean_trial1-000060
-cape/03375-shortshort-swim_trial2-000110
-cape/03375-longlong-box_trial1-000190
-cape/03375-longlong-row_trial2-000150
-cape/00134-shortlong-hockey_trial1-000140
-cape/00134-shortlong-hockey_trial1-000090
-cape/00134-longlong-ski_trial2-000200
-cape/00134-longlong-stretch_trial1-000450
-cape/00096-shirtshort-soccer-000160
-cape/03375-shortshort-hands_up_trial2-000270
-cape/03375-shortshort-ballet1_trial1-000110
-cape/03375-longlong-babysit_trial2-000150
-cape/03375-shortshort-fashion_trial1-000140
-cape/00134-shortlong-ballet2_trial1-000110
-cape/00134-longlong-ballet2_trial1-000120
-cape/00134-shortlong-ballet2_trial1-000120
-cape/00134-shortlong-ballet2_trial1-000090
-cape/00134-longlong-ballet2_trial2-000110
-cape/00134-longlong-volleyball_trial2-000050
-cape/00134-longlong-stretch_trial1-000500
-cape/00134-longlong-housework_trial1-000380
-cape/00134-shortlong-dig_trial1-000150
-cape/03375-longlong-catchpick_trial1-000110
-cape/03375-shortlong-ballet1_trial1-000250
-cape/03375-shortlong-shoulders_trial1-000360
-cape/03375-shortlong-slack_trial2-000070
-cape/03375-shortlong-shoulders_trial1-000220
-cape/03375-shortlong-stretch_trial1-000330
-cape/00127-shortlong-ballerina_spin-000080
-cape/00127-shortlong-ballerina_spin-000200
-cape/00096-shortshort-basketball-000100
-cape/00096-shortshort-ballerina_spin-000160
-cape/00134-longlong-stretch_trial2-000440
-cape/02474-longlong-ATUsquat-000100
-cape/03375-longlong-ATUsquat_trial1-000120
-cape/02474-longlong-ATUsquat-000110
-cape/00134-longlong-ballet1_trial1-000180
-cape/00096-shirtlong-ATUsquat-000130
-cape/00032-shortshort-pose_model-000030
-cape/00134-shortlong-athletics_trial2-000070
-cape/00032-longshort-pose_model-000060
-cape/00032-shortshort-shoulders_mill-000060
-cape/00127-shortlong-pose_model-000430
-cape/00122-shortshort-ATUsquat-000120
-cape/00032-shortshort-bend_back_and_front-000220
-cape/00096-shortshort-squats-000180
-cape/00032-shortlong-squats-000090
-cape/03375-shortlong-ATUsquat_trial2-000080
-cape/03375-shortshort-lean_trial1-000130
-cape/03375-blazerlong-music_trial1-000150
-cape/03284-longlong-hips-000170
-cape/03375-shortlong-shoulders_trial1-000370
-cape/03375-shortlong-ballet1_trial1-000290
-cape/00215-jerseyshort-shoulders_mill-000320
-cape/00215-poloshort-soccer-000110
-cape/00122-shortshort-punching-000170
-cape/00096-jerseyshort-shoulders_mill-000140
-cape/00032-longshort-flying_eagle-000240
-cape/00134-shortlong-swim_trial1-000160
-cape/03375-shortshort-music_trial1-000120
-cape/03375-shortshort-handball_trial1-000120
-cape/00215-longshort-punching-000060
-cape/00134-shortlong-swim_trial2-000120
-cape/03375-shortshort-hands_up_trial1-000140
-cape/03375-shortshort-hands_up_trial1-000270
-cape/03375-shortshort-volleyball_trial1-000110
-cape/03375-shortshort-swim_trial1-000270
-cape/03375-longlong-row_trial2-000190
-cape/00215-poloshort-flying_eagle-000120
-cape/03223-shortshort-flying_eagle-000280
-cape/00096-shirtlong-shoulders_mill-000110
-cape/00096-shirtshort-pose_model-000190
-cape/03375-shortshort-swim_trial1-000190
-cape/03375-shortlong-music_trial2-000040
-cape/03375-shortlong-babysit_trial2-000070
-cape/00215-jerseyshort-flying_eagle-000110
-cape/03375-blazerlong-music_trial1-000030
-cape/03375-longlong-volleyball_trial2-000230
-cape/03375-blazerlong-lean_trial2-000110
-cape/03375-longlong-box_trial2-000110
-cape/03375-longlong-drinkeat_trial2-000050
-cape/00134-shortlong-slack_trial1-000150
-cape/03375-shortshort-climb_trial1-000170
-cape/00032-longshort-tilt_twist_left-000060
-cape/00215-longshort-chicken_wings-000060
-cape/00215-poloshort-bend_back_and_front-000130
-cape/03223-longshort-flying_eagle-000480
-cape/00215-longshort-bend_back_and_front-000100
-cape/00215-longshort-tilt_twist_left-000130
-cape/00096-longshort-tilt_twist_left-000150
-cape/03284-longshort-twist_tilt_left-000080
-cape/03223-shortshort-flying_eagle-000270
-cape/02474-longshort-improvise-000080

requirements.txt CHANGED Viewed

@@ -1,3 +1,11 @@
 matplotlib
 scikit-image
 trimesh
@@ -15,4 +23,10 @@ einops
 boto3
 open3d
 xatlas
 git+https://github.com/YuliangXiu/rembg.git

+--extra-index-url https://download.pytorch.org/whl/cu116
+torch==1.13.1+cu116
+torchvision==0.14.1+cu116
+fvcore
+iopath
+pyembree
+cupy
+cython
 matplotlib
 scikit-image
 trimesh
 boto3
 open3d
 xatlas
+transformers
+controlnet_aux
+xformers==0.0.16
+triton
 git+https://github.com/YuliangXiu/rembg.git
+git+https://github.com/huggingface/diffusers.git
+git+https://github.com/huggingface/accelerate.git