added files

LICENSE

@@ -0,0 +1,57 @@
+License
+Data & Software Copyright License for non-commercial scientific research purposes
+Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the DECO data, models and software, (the "Data & Software"), including DECO baseline models, 3D meshes, MANO/SMPL-X parameters, images, MoCap data, videos, software, and scripts. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding GitHub repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License
+
+Ownership / Licensees
+The Data & Software and the associated materials have been developed at the Max Planck Institute for Intelligent Systems (hereinafter "MPI").
+
+Any copyright or patent right is owned by and proprietary material of the Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter collectively “Max-Planck”)
+
+hereinafter the “Licensor”.
+
+License Grant
+Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:
+
+To install the Data & Software on computers owned, leased or otherwise controlled by you and/or your organization;
+To use the Data & Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects;
+Any other use, in particular any use for commercial, pornographic, military, or surveillance, purposes is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artifacts for commercial purposes. The Data & Software may not be used to create fake, libelous, misleading, or defamatory content of any kind excluding analyses in peer-reviewed scientific research. The Data & Software may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission.
+
+The Data & Software may not be used for pornographic purposes or to generate pornographic material whether commercial or not. This license also prohibits the use of the Data & Software to train methods/algorithms/neural networks/etc. for commercial, pornographic, military, surveillance, or defamatory use of any kind. By downloading the Data & Software, you agree not to reverse engineer it.
+
+No Distribution
+The Data & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive purposes only.
+
+Disclaimer of Representations and Warranties
+You expressly acknowledge and agree that the Data & Software results from basic research, is provided “AS IS”, may contain errors, and that any use of the Data & Software is at your sole risk. LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE DATA & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of the Data & Software, (ii) that the use of the Data & Software will not infringe any patents, copyrights or other intellectual property rights of a third party, and (iii) that the use of the Data & Software will not cause any damage of any kind to you or a third party.
+
+Limitation of Liability
+Because this Data & Software License Agreement qualifies as a donation, according to Section 521 of the German Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only. If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee for the resulting damage.
+Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be liable in accordance with the German Product Liability Act in the event of product liability. The foregoing applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall be excluded.
+Patent claims generated through the usage of the Data & Software cannot be directed towards the copyright holders.
+The Data & Software is provided in the state of development the licensor defines. If modified or extended by Licensee, the Licensor makes no claims about the fitness of the Data & Software and is not responsible for any problems such modifications cause.
+
+No Maintenance Services
+You understand and agree that Licensor is under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Data & Software. Licensor nevertheless reserves the right to update, modify, or discontinue the Data & Software at any time.
+
+Defects of the Data & Software must be notified in writing to the Licensor with a comprehensible description of the error symptoms. The notification of the defect should enable the reproduction of the error. The Licensee is encouraged to communicate any use, results, modification, or publication.
+
+Publications using the Data & Software
+You acknowledge that the Data & Software is a valuable scientific resource and agree to appropriately reference the following paper in any publication making use of the Data & Software.
+
+Subjects' Consent: All subjects gave informed written consent to share their data for research purposes. You further agree to delete data or change their use, in case a subject changes or withdraws their consent.
+
+Citation:
+
+@InProceedings{Tripathi_2023_ICCV,
+    author    = {Tripathi, Shashank and Chatterjee, Agniv and Passy, Jean-Claude and Yi, Hongwei and Tzionas, Dimitrios and Black, Michael J.},
+    title     = {DECO: Dense Estimation of 3D Human-Scene Contact In The Wild},
+    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+    month     = {October},
+    year      = {2023},
+    pages     = {8001-8013}
+}
+
+Commercial licensing opportunities
+For commercial uses of the Data & Software, please send emails to ps-license@tue.mpg.de
+
+This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.

README.md

@@ -1,12 +1,145 @@
----
-title: DECO
-emoji: 🏢
-colorFrom: blue
-colorTo: pink
-sdk: gradio
-sdk_version: 3.47.1
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# DECO: Dense Estimation of 3D Human-Scene Contact in the Wild [ICCV 2023 (Oral)]
+
+> Code repository for the paper:  
+> [**DECO: Dense Estimation of 3D Human-Scene Contact in the Wild**](https://openaccess.thecvf.com/content/ICCV2023/html/Tripathi_DECO_Dense_Estimation_of_3D_Human-Scene_Contact_In_The_Wild_ICCV_2023_paper.html)  
+> [Shashank Tripathi](https://sha2nkt.github.io/), [Agniv Chatterjee](https://ac5113.github.io/), [Jean-Claude Passy](https://is.mpg.de/person/jpassy), [Hongwei Yi](https://xyyhw.top/), [Dimitrios Tzionas](https://ps.is.mpg.de/person/dtzionas), [Michael J. Black](https://ps.is.mpg.de/person/black)<br />
+> *IEEE International Conference on Computer Vision (ICCV), 2023*
+
+[![arXiv](https://img.shields.io/badge/arXiv-2309.15273-00ff00.svg)](https://arxiv.org/abs/2309.15273)  [![Website shields.io](https://img.shields.io/website-up-down-green-red/http/shields.io.svg)](https://deco.is.tue.mpg.de/)     [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)]()  [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)]()
+
+![teaser](assets/teaser.png)
+
+[[Project Page](https://deco.is.tue.mpg.de)] [[Paper](https://arxiv.org/abs/2309.15273)] [[Video](https://www.youtube.com/watch?v=o7MLobqAFTQ)] [[Poster](https://www.dropbox.com/scl/fi/kvhpfnkvga2pt19ayko8u/ICCV2023_DECO_Poster_v2.pptx?rlkey=ihbf3fi6u9j0ha9x1gfk2cwd0&dl=0)] [[License](https://deco.is.tue.mpg.de/license.html)] [[Contact](mailto:deco@tue.mpg.de)]
+
+## Installation and Setup
+1. First, clone the repo. Then, we recommend creating a clean [conda](https://docs.conda.io/) environment, activating it and installing torch and torchvision, as follows:
+```shell
+git clone https://github.com/sha2nkt/deco.git
+cd deco
+conda create -n deco python=3.9 -y
+conda activate deco
+pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
+```
+Please adjust the CUDA version as required.
+
+2. Install PyTorch3D from source. Users may also refer to [PyTorch3D-install](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md) for more details.
+However, our tests show that installing using ``conda`` sometimes runs into dependency conflicts.
+Hence, users may alternatively install Pytorch3D from source following the steps below.
+```shell
+git clone https://github.com/facebookresearch/pytorch3d.git
+cd pytorch3d
+pip install .
+cd ..
+```
+
+3. Install the other dependancies and download the required data.
+```bash
+pip install -r requirements.txt
+sh fetch_data.sh
+```
+
+4. Please download [SMPL](https://smpl.is.tue.mpg.de/) (version 1.1.0) and [SMPL-X](https://smpl-x.is.tue.mpg.de/) (v1.1) files into the data folder. Please rename the SMPL files to ```SMPL_FEMALE.pkl```, ```SMPL_MALE.pkl``` and ```SMPL_NEUTRAL.pkl```. The directory structure for the ```data``` folder has been elaborated below:
+
+```
+├── preprocess
+├── smpl
+│   ├── SMPL_FEMALE.pkl
+│   ├── SMPL_MALE.pkl
+│   ├── SMPL_NEUTRAL.pkl
+│   ├── smpl_neutral_geodesic_dist.npy
+│   ├── smpl_neutral_tpose.ply
+│   ├── smplpix_vertex_colors.npy
+├── smplx
+│   ├── SMPLX_FEMALE.npz
+│   ├── SMPLX_FEMALE.pkl
+│   ├── SMPLX_MALE.npz
+│   ├── SMPLX_MALE.pkl
+│   ├── SMPLX_NEUTRAL.npz
+│   ├── SMPLX_NEUTRAL.pkl
+│   ├── smplx_neutral_tpose.ply
+├── weights
+│   ├── pose_hrnet_w32_256x192.pth
+├── J_regressor_extra.npy
+├── base_dataset.py
+├── mixed_dataset.py
+├── smpl_partSegmentation_mapping.pkl
+├── smpl_vert_segmentation.json
+└── smplx_vert_segmentation.json
+```
+
+## Run demo on images
+The following command will run DECO on all images in the specified `--img_src`, and save rendering and colored mesh in `--out_dir`. The `--model_path` flag is used to specify the specific checkpoint being used. Additionally, the base mesh color and the color of predicted contact annotation can be specified using the `--mesh_colour` and `--annot_colour` flags respectively. 
+```bash
+python inference.py \
+    --img_src example_images \
+    --out_dir demo_out
+```
+
+## Training and Evaluation
+
+We release 3 versions of the DECO model:
+<ol>
+    <li> DECO-HRNet (<em> Best performing model </em>) </li>
+    <li> DECO-HRNet w/o context branches </li>
+    <li> DECO-Swin </li>
+</ol>
+
+All the checkpoints have been downloaded to ```checkpoints```. 
+However, please note that versions 2 and 3 have been trained solely on the RICH dataset. <br>
+We recommend using the first DECO version.
+
+The dataset npz files have been downloaded to ```datasets/Release_Datasets```. Please download the actual DAMON data and place them in ```datasets``` following the instructions given.
+
+### Evaluation
+To run evaluation on the DAMON dataset, please run the following command:
+
+```bash
+python tester.py --cfg configs/cfg_test.yml
+```
+
+### Training
+The config provided (```cfg_train.yml```) is set to train and evaluate on all three datasets: DAMON, RICH and PROX. To change this, please change the value of the key ```TRAINING.DATASETS``` and ```VALIDATION.DATASETS``` in the config (please also change ```TRAINING.DATASET_MIX_PDF``` as required). <br>
+Also, the best checkpoint is stored by default at ```checkpoints/Other_Checkpoints```.
+Please run the following command to start training of the DECO model:
+
+```bash
+python train.py --cfg configs/cfg_train.yml
+```
+
+### Training on custom datasets
+
+To train on other datasets, please follow these steps:
+1. Please create an npz of the dataset, following the structure of the datasets in ```datasets/Release_Datasets``` with the corresponding keys and values.
+2. Please create scene segmentation maps, if not available. We have used [Mask2Former](https://github.com/facebookresearch/Mask2Former) in our work.
+3. For creating the part segmentation maps, this [sample script](https://github.com/sha2nkt/deco/blob/main/scripts/datascripts/get_part_seg_mask.py) can be referred to.
+4. Add the dataset name(s) to ```train.py``` ([these lines](https://github.com/sha2nkt/deco/blob/d5233ecfad1f51b71a50a78c0751420067e82c02/train.py#L83)), ```tester.py``` ([these lines](https://github.com/sha2nkt/deco/blob/d5233ecfad1f51b71a50a78c0751420067e82c02/tester.py#L51)) and ```data/mixed_dataset.py``` ([these lines](https://github.com/sha2nkt/deco/blob/d5233ecfad1f51b71a50a78c0751420067e82c02/data/mixed_dataset.py#L17)), according to the body model being used (SMPL/SMPL-X)
+5. Add the path(s) to the dataset npz(s) to ```common/constants.py``` ([these lines](https://github.com/sha2nkt/deco/blob/d5233ecfad1f51b71a50a78c0751420067e82c02/common/constants.py#L19)).
+6. Finally, change ```TRAINING.DATASETS``` and ```VALIDATION.DATASETS``` in the config file and you're good to go!
+
+## Citing
+If you find this code useful for your research, please consider citing the following paper:
+
+```bibtex
+@InProceedings{Tripathi_2023_ICCV,
+    author    = {Tripathi, Shashank and Chatterjee, Agniv and Passy, Jean-Claude and Yi, Hongwei and Tzionas, Dimitrios and Black, Michael J.},
+    title     = {DECO: Dense Estimation of 3D Human-Scene Contact In The Wild},
+    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+    month     = {October},
+    year      = {2023},
+    pages     = {8001-8013}
+}
+```
+
+### License
+
+See [LICENSE](LICENSE).
+
+### Acknowledgments
+
+We sincerely thank Alpar Cseke for his contributions to DAMON data collection and PHOSA evaluations, Sai K. Dwivedi for facilitating PROX downstream experiments, Xianghui Xie for his generous help with CHORE evaluations, Lea Muller for her help in initiating the contact annotation tool, Chun-Hao P. Huang for RICH discussions and Yixin Chen for details about the HOT paper. We are grateful to Mengqin Xue and Zhenyu Lou for their collaboration in BEHAVE evaluations, Joachim Tesch and Nikos Athanasiou for insightful visualization advice, and Tsvetelina Alexiadis for valuable data collection guidance. Their invaluable contributions enriched this research significantly. We also thank Benjamin Pellkofer for help with the website and IT support. This work was funded by the International Max Planck Research School for Intelligent Systems (IMPRS-IS).
+
+### Contact
+
+For technical questions, please create an issue. For other questions, please contact `deco@tue.mpg.de`.
+
+For commercial licensing, please contact `ps-licensing@tue.mpg.de`.

app.py

@@ -0,0 +1,7 @@
+import gradio as gr
+
+def greet(name):
+    return "Hello " + name + "!!"
+
+iface = gr.Interface(fn=greet, inputs="text", outputs="text")
+iface.launch()

common/constants.py

@@ -0,0 +1,35 @@
+from os.path import join
+
+DIST_MATRIX_PATH = 'data/smpl/smpl_neutral_geodesic_dist.npy'
+SMPL_MEAN_PARAMS = 'data/smpl_mean_params.npz'
+SMPL_MODEL_DIR = 'data/smpl/'
+SMPLX_MODEL_DIR = 'data/smplx/'
+
+N_PARTS = 24
+
+# Mean and standard deviation for normalizing input image
+IMG_NORM_MEAN = [0.485, 0.456, 0.406]
+IMG_NORM_STD = [0.229, 0.224, 0.225]
+
+# Output folder to save test/train npz files
+DATASET_NPZ_PATH = 'datasets/Release_Datasets'
+CONTACT_MAPPING_PATH = 'data/conversions'
+
+# Path to test/train npz files
+DATASET_FILES = {
+    'train': {
+        'damon': join(DATASET_NPZ_PATH, 'damon/hot_dca_trainval.npz'),
+        'rich': join(DATASET_NPZ_PATH, 'rich/rich_train_smplx_cropped_bmp.npz'),
+        'prox': join(DATASET_NPZ_PATH, 'prox/prox_train_smplx_ds4.npz'),
+    },
+    'val': {
+        'damon': join(DATASET_NPZ_PATH, 'damon/hot_dca_test.npz'),
+        'rich': join(DATASET_NPZ_PATH, 'rich/rich_test_smplx_cropped_bmp.npz'),
+        'prox': join(DATASET_NPZ_PATH, 'prox/prox_val_smplx_ds4.npz'),
+    },
+    'test': {
+        'damon': join(DATASET_NPZ_PATH, 'damon/hot_dca_test.npz'),
+        'rich': join(DATASET_NPZ_PATH, 'rich/rich_test_smplx_cropped_bmp.npz'),
+        'prox': join(DATASET_NPZ_PATH, 'prox/prox_val_smplx_ds4.npz'),
+    },
+}

common/renderer_pyrd.py

@@ -0,0 +1,112 @@
+# Copyright (C) 2022. Huawei Technologies Co., Ltd. All rights reserved.
+
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the MIT license.
+
+# This program is distributed in the hope that it will be useful, but WITHOUT ANY
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+# PARTICULAR PURPOSE. See the MIT License for more details.
+
+import os
+import trimesh
+import pyrender
+import numpy as np
+import colorsys
+import cv2
+
+
+class Renderer(object):
+
+    def __init__(self, focal_length=600, img_w=512, img_h=512, faces=None,
+                 same_mesh_color=False):
+        os.environ['PYOPENGL_PLATFORM'] = 'egl'
+        self.renderer = pyrender.OffscreenRenderer(viewport_width=img_w,
+                                                   viewport_height=img_h,
+                                                   point_size=1.0)
+        self.camera_center = [img_w // 2, img_h // 2]
+        self.focal_length = focal_length
+        self.faces = faces
+        self.same_mesh_color = same_mesh_color
+
+    def render_front_view(self, verts, bg_img_rgb=None, bg_color=(0, 0, 0, 0), vertex_colors=None, render_part_seg=False, part_label_bins=None):
+        # Create a scene for each image and render all meshes
+        scene = pyrender.Scene(bg_color=bg_color, ambient_light=np.ones(3) * (1 if render_part_seg else 0))
+        # Create camera. Camera will always be at [0,0,0]
+        camera = pyrender.camera.IntrinsicsCamera(fx=self.focal_length, fy=self.focal_length,
+                                                  cx=self.camera_center[0], cy=self.camera_center[1])
+        scene.add(camera, pose=np.eye(4))
+
+        # Create light source
+        if not render_part_seg:
+            light = pyrender.DirectionalLight(color=[1.0, 1.0, 1.0], intensity=3.0)
+            # for DirectionalLight, only rotation matters
+            light_pose = trimesh.transformations.rotation_matrix(np.radians(-45), [1, 0, 0])
+            scene.add(light, pose=light_pose)
+            light_pose = trimesh.transformations.rotation_matrix(np.radians(45), [0, 1, 0])
+            scene.add(light, pose=light_pose)
+
+        # Need to flip x-axis
+        rot = trimesh.transformations.rotation_matrix(np.radians(180), [1, 0, 0])
+        # multiple person
+        num_people = len(verts)
+        
+        # for every person in the scene
+        for n in range(num_people):
+            mesh = trimesh.Trimesh(verts[n], self.faces, process=False)
+            mesh.apply_transform(rot)
+            if self.same_mesh_color:
+                mesh_color = colorsys.hsv_to_rgb(0.6, 0.5, 1.0)
+            else:
+                mesh_color = colorsys.hsv_to_rgb(float(n) / num_people, 0.5, 1.0)
+            material = pyrender.MetallicRoughnessMaterial(
+                metallicFactor=1.0,
+                alphaMode='OPAQUE',
+                baseColorFactor=mesh_color)
+
+            if vertex_colors is not None:
+                # color individual vertices based on part labels
+                mesh.visual.vertex_colors = vertex_colors
+            mesh = pyrender.Mesh.from_trimesh(mesh, material=material, wireframe=False)
+            scene.add(mesh, 'mesh')
+
+        # Alpha channel was not working previously, need to check again
+        # Until this is fixed use hack with depth image to get the opacity
+        color_rgba, depth_map = self.renderer.render(scene, flags=pyrender.RenderFlags.RGBA)
+        color_rgb = color_rgba[:, :, :3]
+
+        if render_part_seg:
+            body_parts = color_rgb.copy()
+            # make single channel
+            body_parts = body_parts.max(-1)  # reduce to single channel
+            # convert pixel value to bucket indices
+            # body_parts = torch.bucketize(body_parts, self.part_label_bins, right=True)
+            body_parts = np.digitize(body_parts, part_label_bins, right=True)
+            # part labels start from 2 because of the binning scheme. Subtract 1 from all non-zero labels to make label
+            # go from 1 to 24. 0 is background
+            # handle background coinciding with hip label = 0
+            body_parts = body_parts + 1
+            mask = depth_map > 0
+            body_parts = body_parts * mask
+            return body_parts, color_rgb
+
+        if bg_img_rgb is None:
+            return color_rgb
+        else:
+            mask = depth_map > 0
+            bg_img_rgb[mask] = color_rgb[mask]
+            return bg_img_rgb
+
+    def render_side_view(self, verts):
+        centroid = verts.mean(axis=(0, 1))  # n*6890*3 -> 3
+        # make the centroid at the image center (the X and Y coordinates are zeros)
+        centroid[:2] = 0
+        aroundy = cv2.Rodrigues(np.array([0, np.radians(90.), 0]))[0][np.newaxis, ...]  # 1*3*3
+        pred_vert_arr_side = np.matmul((verts - centroid), aroundy) + centroid
+        side_view = self.render_front_view(pred_vert_arr_side)
+        return side_view
+
+    def delete(self):
+        """
+        Need to delete before creating the renderer next time
+        """
+        self.renderer.delete()

configs/cfg_test.yml

@@ -0,0 +1,28 @@
+EXP_NAME: 'damon_hrnet_testing'
+PROJECT_NAME: 'DECO_DAMON_Testing'
+OUTPUT_DIR: 'deco_results'
+CONDOR_DIR: ''
+DATASET:
+  BATCH_SIZE: 16
+  NUM_WORKERS: 4
+  NORMALIZE_IMAGES: [True]
+OPTIMIZER:
+  TYPE: 'adam'
+  LR: [5e-5]
+  NUM_UPDATE_LR: 3
+TRAINING:
+  ENCODER: 'hrnet'
+  CONTEXT: [True]
+  NUM_EPOCHS: 1
+  NUM_EARLY_STOP: 10
+  SUMMARY_STEPS: 5
+  CHECKPOINT_EPOCHS: 5
+  DATASETS: ['damon']
+  DATASET_MIX_PDF: ['1.']
+  DATASET_ROOT_PATH: ''
+  BEST_MODEL_PATH: './checkpoints/Release_Checkpoint/deco_best.pth'
+  PAL_LOSS_WEIGHTS: 0.0
+VALIDATION:
+  SUMMARY_STEPS: 1000
+  DATASETS: ['damon']
+  MAIN_DATASET: 'damon'

configs/cfg_train.yml

@@ -0,0 +1,29 @@
+EXP_NAME: 'demo_train'
+PROJECT_NAME: 'DECO_demo_training'
+OUTPUT_DIR: 'deco_results'
+CONDOR_DIR: ''
+DATASET:
+  BATCH_SIZE: 4
+  NUM_WORKERS: 8
+  NORMALIZE_IMAGES: [True]
+OPTIMIZER:
+  TYPE: 'adam'
+  LR: [1e-5]
+  NUM_UPDATE_LR: 3
+TRAINING:
+  ENCODER: 'hrnet'
+  CONTEXT: [True]
+  NUM_EPOCHS: 100
+  NUM_EARLY_STOP: 10
+  SUMMARY_STEPS: 5
+  CHECKPOINT_EPOCHS: 5
+  DATASETS: ['damon', 'rich', 'prox']
+  DATASET_MIX_PDF: ['0.4', '0.3', '0.3'] # should sum to 1.0 unless you want to weight by dataset size
+  DATASET_ROOT_PATH: ''
+  BEST_MODEL_PATH: './checkpoints/Other_Checkpoints/demo_train.pth'
+  LOSS_WEIGHTS: 1.
+  PAL_LOSS_WEIGHTS: 0.01
+VALIDATION:
+  SUMMARY_STEPS: 5
+  DATASETS: ['damon', 'rich', 'prox']
+  MAIN_DATASET: 'damon'

data/base_dataset.py

@@ -0,0 +1,164 @@
+import torch
+import cv2
+import numpy as np
+from torch.utils.data import Dataset
+from torchvision.transforms import Normalize
+from common import constants
+
+def mask_split(img, num_parts):
+    if not len(img.shape) == 2:
+        img = img[:, :, 0]
+    mask = np.zeros((img.shape[0], img.shape[1], num_parts))
+    for i in np.unique(img):
+        mask[:, :, i] = np.where(img == i, 1., 0.)
+    return np.transpose(mask, (2, 0, 1))
+
+class BaseDataset(Dataset):
+
+    def __init__(self, dataset, mode, model_type='smpl', normalize=False):
+        self.dataset = dataset
+        self.mode = mode
+
+        print(f'Loading dataset: {constants.DATASET_FILES[mode][dataset]} for mode: {mode}')
+
+        self.data = np.load(constants.DATASET_FILES[mode][dataset], allow_pickle=True)
+
+        self.images = self.data['imgname']
+
+        # get 3d contact labels, if available
+        try:
+            self.contact_labels_3d = self.data['contact_label']
+            # make a has_contact_3d numpy array which contains 1 if contact labels are no empty and 0 otherwise
+            self.has_contact_3d = np.array([1 if len(x) > 0 else 0 for x in self.contact_labels_3d])
+        except KeyError:
+            self.has_contact_3d = np.zeros(len(self.images))
+
+        # get 2d polygon contact labels, if available
+        try:
+            self.polygon_contacts_2d = self.data['polygon_2d_contact']
+            self.has_polygon_contact_2d = np.ones(len(self.images))
+        except KeyError:
+            self.has_polygon_contact_2d = np.zeros(len(self.images))
+
+        # Get camera parameters - only intrinsics for now
+        try:
+            self.cam_k = self.data['cam_k']
+        except KeyError:
+            self.cam_k = np.zeros((len(self.images), 3, 3))
+
+        self.sem_masks = self.data['scene_seg']
+        self.part_masks = self.data['part_seg']
+
+        # Get gt SMPL parameters, if available
+        try:
+            self.pose = self.data['pose'].astype(float)
+            self.betas = self.data['shape'].astype(float)
+            self.transl = self.data['transl'].astype(float)
+            if 'has_smpl' in self.data:
+                self.has_smpl = self.data['has_smpl']
+            else:
+                self.has_smpl = np.ones(len(self.images))
+                self.is_smplx = np.ones(len(self.images)) if model_type == 'smplx' else np.zeros(len(self.images))
+        except KeyError:
+            self.has_smpl = np.zeros(len(self.images))
+            self.is_smplx = np.zeros(len(self.images))
+
+        if model_type == 'smpl':
+            self.n_vertices = 6890
+        elif model_type == 'smplx':
+            self.n_vertices = 10475
+        else:
+            raise NotImplementedError
+
+        self.normalize = normalize
+        self.normalize_img = Normalize(mean=constants.IMG_NORM_MEAN, std=constants.IMG_NORM_STD)
+
+    def __getitem__(self, index):
+        item = {}
+
+        # Load image
+        img_path = self.images[index]
+        try:
+            img = cv2.imread(img_path)
+            img_h, img_w, _ = img.shape
+            img = cv2.resize(img, (256, 256), cv2.INTER_CUBIC)
+            img = img.transpose(2, 0, 1) / 255.0
+        except:
+            print('Img: ', img_path)
+
+        img_scale_factor = np.array([256 / img_w, 256 / img_h])
+
+        # Get SMPL parameters, if available
+        if self.has_smpl[index]:
+            pose = self.pose[index].copy()
+            betas = self.betas[index].copy()
+            transl = self.transl[index].copy()
+        else:
+            pose = np.zeros(72)
+            betas = np.zeros(10)
+            transl = np.zeros(3)
+
+        # Load vertex_contact
+        if self.has_contact_3d[index]:
+            contact_label_3d = self.contact_labels_3d[index]
+        else:
+            contact_label_3d = np.zeros(self.n_vertices)
+
+        sem_mask_path = self.sem_masks[index]
+        try:
+            sem_mask = cv2.imread(sem_mask_path)
+            sem_mask = cv2.resize(sem_mask, (256, 256), cv2.INTER_CUBIC)
+            sem_mask = mask_split(sem_mask, 133)
+        except:
+            print('Scene seg: ', sem_mask_path)
+
+        try:
+            part_mask_path = self.part_masks[index]
+            part_mask = cv2.imread(part_mask_path)
+            part_mask = cv2.resize(part_mask, (256, 256), cv2.INTER_CUBIC)
+            part_mask = mask_split(part_mask, 26)
+        except:
+            print('Part seg: ', part_mask_path)
+
+        try:
+            if self.has_polygon_contact_2d[index]:
+                polygon_contact_2d_path = self.polygon_contacts_2d[index]
+                polygon_contact_2d = cv2.imread(polygon_contact_2d_path)
+                polygon_contact_2d = cv2.resize(polygon_contact_2d, (256, 256), cv2.INTER_NEAREST)
+                # binarize the part mask
+                polygon_contact_2d = np.where(polygon_contact_2d > 0, 1, 0)
+            else:
+                polygon_contact_2d = np.zeros((256, 256, 3))
+        except:
+            print('2D polygon contact: ', polygon_contact_2d_path)
+
+        if self.normalize:
+            img = torch.tensor(img, dtype=torch.float32)
+            item['img'] = self.normalize_img(img)
+        else:
+            item['img'] = torch.tensor(img, dtype=torch.float32)
+
+        if self.is_smplx[index]:
+            # Add 6 zeros to the end of the pose vector to match with smpl
+            pose = np.concatenate((pose, np.zeros(6)))
+
+        item['img_path'] = img_path
+        item['pose'] = torch.tensor(pose, dtype=torch.float32)
+        item['betas'] = torch.tensor(betas, dtype=torch.float32)
+        item['transl'] = torch.tensor(transl, dtype=torch.float32)
+        item['cam_k'] = self.cam_k[index]
+        item['img_scale_factor'] = torch.tensor(img_scale_factor, dtype=torch.float32)
+        item['contact_label_3d'] = torch.tensor(contact_label_3d, dtype=torch.float32)
+        item['sem_mask'] = torch.tensor(sem_mask, dtype=torch.float32)
+        item['part_mask'] = torch.tensor(part_mask, dtype=torch.float32)
+        item['polygon_contact_2d'] = torch.tensor(polygon_contact_2d, dtype=torch.float32)
+
+        item['has_smpl'] = self.has_smpl[index]
+        item['is_smplx'] = self.is_smplx[index]
+        item['has_contact_3d'] = self.has_contact_3d[index]
+        item['has_polygon_contact_2d'] = self.has_polygon_contact_2d[index]
+
+        return item
+
+    def __len__(self):
+        return len(self.images)

data/mixed_dataset.py

@@ -0,0 +1,42 @@
+"""
+This file contains the definition of different heterogeneous datasets used for training
+"""
+import torch
+import numpy as np
+
+from .base_dataset import BaseDataset
+
+class MixedDataset(torch.utils.data.Dataset):
+
+    def __init__(self, ds_list, mode, dataset_mix_pdf, **kwargs):
+        self.dataset_list = ds_list
+        print('Training Dataset list: ', self.dataset_list)
+        self.num_datasets = len(self.dataset_list)
+
+        self.datasets = []
+        for ds in self.dataset_list:
+            if ds in ['rich', 'prox']:
+                self.datasets.append(BaseDataset(ds, mode, model_type='smplx', **kwargs))
+            elif ds in ['damon']:
+                self.datasets.append(BaseDataset(ds, mode, model_type='smpl', **kwargs))
+            else:
+                raise ValueError('Dataset not supported')
+
+        total_length = sum([len(ds) for ds in self.datasets])
+        length_itw = sum([len(ds) for ds in self.datasets])
+        self.length = max([len(ds) for ds in self.datasets])
+
+        # convert list of strings to list of floats
+        self.partition = [float(i) for i in dataset_mix_pdf] # should sum to 1.0 unless you want to weight by dataset size
+        assert sum(self.partition) == 1.0, "Dataset Mix PDF must sum to 1.0 unless you want to weight by dataset size"
+        assert len(self.partition) == self.num_datasets, "Number of partitions must be equal to number of datasets"
+        self.partition = np.array(self.partition).cumsum()
+
+    def __getitem__(self, index):
+        p = np.random.rand()
+        for i in range(self.num_datasets):
+            if p <= self.partition[i]:
+                return self.datasets[i][index % len(self.datasets[i])]
+
+    def __len__(self):
+        return self.length

data/preprocess/behave-30fps-error_frames.json

@@ -0,0 +1,511 @@
+{
+  "error_seqs": [
+    "Date01_Sub01_boxsmall_hand",
+    "Date01_Sub01_boxtiny_hand",
+    "Date02_Sub02_backpack_back",
+    "Date02_Sub02_backpack_hand",
+    "Date02_Sub02_backpack_twohand",
+    "Date02_Sub02_boxtiny_hand",
+    "Date02_Sub02_monitor_move2",
+    "Date02_Sub02_plasticcontainer",
+    "Date02_Sub02_toolbox",
+    "Date02_Sub02_toolbox_part2",
+    "Date02_Sub02_yogamat",
+    "Date03_Sub03_boxlong",
+    "Date03_Sub03_plasticcontainer",
+    "Date03_Sub04_boxsmall",
+    "Date03_Sub04_boxtiny",
+    "Date03_Sub04_boxtiny_part2",
+    "Date03_Sub04_plasticcontainer_lift",
+    "Date03_Sub04_toolbox",
+    "Date03_Sub05_boxsmall",
+    "Date03_Sub05_boxtiny",
+    "Date04_Sub05_boxlarge",
+    "Date04_Sub05_boxlong",
+    "Date04_Sub05_boxtiny",
+    "Date04_Sub05_stool",
+    "Date05_Sub06_boxlong",
+    "Date05_Sub06_boxtiny",
+    "Date05_Sub06_toolbox",
+    "Date06_Sub07_boxlong",
+    "Date06_Sub07_boxtiny",
+    "Date06_Sub07_stool_sit",
+    "Date06_Sub07_suitcase_lift",
+    "Date06_Sub07_toolbox",
+    "Date06_Sub07_yogamat",
+    "Date07_Sub04_boxlong",
+    "Date07_Sub04_boxsmall",
+    "Date07_Sub04_boxtiny",
+    "Date07_Sub04_plasticcontainer",
+    "Date07_Sub04_suitcase_lift",
+    "Date07_Sub05_suitcase_open",
+    "Date07_Sub05_tablesmall",
+    "Date07_Sub08_boxtiny",
+    "Date07_Sub08_yogamat"
+  ],
+  "good_quality": [
+    "Date01_Sub01_backpack_back",
+    "Date01_Sub01_backpack_hand",
+    "Date01_Sub01_backpack_hug",
+    "Date01_Sub01_boxlarge_hand",
+    "Date01_Sub01_boxlong_hand",
+    "Date01_Sub01_boxmedium_hand",
+    "Date01_Sub01_chairblack_hand",
+    "Date01_Sub01_chairblack_lift",
+    "Date01_Sub01_chairblack_sit",
+    "Date01_Sub01_chairwood_hand",
+    "Date01_Sub01_chairwood_lift",
+    "Date01_Sub01_chairwood_sit",
+    "Date01_Sub01_monitor_hand",
+    "Date01_Sub01_monitor_move",
+    "Date01_Sub01_plasticcontainer",
+    "Date01_Sub01_stool_move",
+    "Date01_Sub01_stool_sit",
+    "Date01_Sub01_suitcase",
+    "Date01_Sub01_suitcase_lift",
+    "Date01_Sub01_tablesmall_lean",
+    "Date01_Sub01_tablesmall_lift",
+    "Date01_Sub01_tablesmall_move",
+    "Date01_Sub01_tablesquare_hand",
+    "Date01_Sub01_tablesquare_lift",
+    "Date01_Sub01_tablesquare_sit",
+    "Date01_Sub01_toolbox",
+    "Date01_Sub01_trashbin",
+    "Date01_Sub01_yogaball",
+    "Date01_Sub01_yogaball_play",
+    "Date01_Sub01_yogamat_hand",
+    "Date02_Sub02_boxlarge_hand",
+    "Date02_Sub02_boxlong_hand",
+    "Date02_Sub02_boxmedium_hand",
+    "Date02_Sub02_boxsmall_hand",
+    "Date02_Sub02_chairblack_hand",
+    "Date02_Sub02_chairblack_lift",
+    "Date02_Sub02_chairblack_sit",
+    "Date02_Sub02_chairwood_hand",
+    "Date02_Sub02_chairwood_sit",
+    "Date02_Sub02_monitor_hand",
+    "Date02_Sub02_monitor_move",
+    "Date02_Sub02_stool_move",
+    "Date02_Sub02_stool_sit",
+    "Date02_Sub02_suitcase_ground",
+    "Date02_Sub02_suitcase_lift",
+    "Date02_Sub02_tablesmall_lean",
+    "Date02_Sub02_tablesmall_lift",
+    "Date02_Sub02_tablesmall_move",
+    "Date02_Sub02_tablesquare_lift",
+    "Date02_Sub02_tablesquare_move",
+    "Date02_Sub02_tablesquare_sit",
+    "Date02_Sub02_trashbin",
+    "Date02_Sub02_yogaball_play",
+    "Date02_Sub02_yogaball_sit",
+    "Date03_Sub03_backpack_back",
+    "Date03_Sub03_backpack_hand",
+    "Date03_Sub03_backpack_hug",
+    "Date03_Sub03_boxlarge",
+    "Date03_Sub03_boxmedium",
+    "Date03_Sub03_boxsmall",
+    "Date03_Sub03_boxtiny",
+    "Date03_Sub03_chairblack_hand",
+    "Date03_Sub03_chairblack_lift",
+    "Date03_Sub03_chairblack_sit",
+    "Date03_Sub03_chairblack_sitstand",
+    "Date03_Sub03_chairwood_hand",
+    "Date03_Sub03_chairwood_lift",
+    "Date03_Sub03_chairwood_sit",
+    "Date03_Sub03_monitor_move",
+    "Date03_Sub03_stool_lift",
+    "Date03_Sub03_stool_sit",
+    "Date03_Sub03_suitcase_lift",
+    "Date03_Sub03_suitcase_move",
+    "Date03_Sub03_tablesmall_lean",
+    "Date03_Sub03_tablesmall_lift",
+    "Date03_Sub03_tablesmall_move",
+    "Date03_Sub03_tablesquare_lift",
+    "Date03_Sub03_tablesquare_move",
+    "Date03_Sub03_tablesquare_sit",
+    "Date03_Sub03_toolbox",
+    "Date03_Sub03_trashbin",
+    "Date03_Sub03_yogaball_play",
+    "Date03_Sub03_yogaball_sit",
+    "Date03_Sub03_yogamat",
+    "Date03_Sub04_backpack_back",
+    "Date03_Sub04_backpack_hand",
+    "Date03_Sub04_backpack_hug",
+    "Date03_Sub04_boxlarge",
+    "Date03_Sub04_boxlong",
+    "Date03_Sub04_boxmedium",
+    "Date03_Sub04_chairblack_hand",
+    "Date03_Sub04_chairblack_liftreal",
+    "Date03_Sub04_chairblack_sit",
+    "Date03_Sub04_chairwood_hand",
+    "Date03_Sub04_chairwood_lift",
+    "Date03_Sub04_chairwood_sit",
+    "Date03_Sub04_monitor_hand",
+    "Date03_Sub04_monitor_move",
+    "Date03_Sub04_stool_move",
+    "Date03_Sub04_stool_sit",
+    "Date03_Sub04_suitcase_ground",
+    "Date03_Sub04_suitcase_lift",
+    "Date03_Sub04_tablesmall_hand",
+    "Date03_Sub04_tablesmall_lean",
+    "Date03_Sub04_tablesmall_lift",
+    "Date03_Sub04_tablesquare_hand",
+    "Date03_Sub04_tablesquare_lift",
+    "Date03_Sub04_tablesquare_sit",
+    "Date03_Sub04_trashbin",
+    "Date03_Sub04_yogaball_play",
+    "Date03_Sub04_yogaball_play2",
+    "Date03_Sub04_yogaball_sit",
+    "Date03_Sub04_yogamat",
+    "Date03_Sub05_backpack",
+    "Date03_Sub05_boxlarge",
+    "Date03_Sub05_boxlong",
+    "Date03_Sub05_boxmedium",
+    "Date03_Sub05_chairblack",
+    "Date03_Sub05_chairwood",
+    "Date03_Sub05_chairwood_part2",
+    "Date03_Sub05_monitor",
+    "Date03_Sub05_plasticcontainer",
+    "Date03_Sub05_stool",
+    "Date03_Sub05_suitcase",
+    "Date03_Sub05_tablesmall",
+    "Date03_Sub05_tablesquare",
+    "Date03_Sub05_toolbox",
+    "Date03_Sub05_trashbin",
+    "Date03_Sub05_yogaball",
+    "Date03_Sub05_yogamat",
+    "Date04_Sub05_backpack",
+    "Date04_Sub05_boxmedium",
+    "Date04_Sub05_boxsmall",
+    "Date04_Sub05_chairblack",
+    "Date04_Sub05_chairwood",
+    "Date04_Sub05_monitor",
+    "Date04_Sub05_monitor_part2",
+    "Date04_Sub05_monitor_sit",
+    "Date04_Sub05_plasticcontainer",
+    "Date04_Sub05_suitcase",
+    "Date04_Sub05_suitcase_open",
+    "Date04_Sub05_tablesmall",
+    "Date04_Sub05_tablesquare",
+    "Date04_Sub05_toolbox",
+    "Date04_Sub05_trashbin",
+    "Date04_Sub05_yogaball",
+    "Date04_Sub05_yogamat",
+    "Date05_Sub05_backpack",
+    "Date05_Sub05_chairblack",
+    "Date05_Sub05_chairwood",
+    "Date05_Sub05_yogaball",
+    "Date05_Sub06_backpack_back",
+    "Date05_Sub06_backpack_hand",
+    "Date05_Sub06_backpack_twohand",
+    "Date05_Sub06_boxlarge",
+    "Date05_Sub06_boxmedium",
+    "Date05_Sub06_boxsmall",
+    "Date05_Sub06_chairblack_hand",
+    "Date05_Sub06_chairblack_lift",
+    "Date05_Sub06_chairblack_sit",
+    "Date05_Sub06_chairwood_hand",
+    "Date05_Sub06_chairwood_lift",
+    "Date05_Sub06_chairwood_sit",
+    "Date05_Sub06_monitor_hand",
+    "Date05_Sub06_monitor_move",
+    "Date05_Sub06_plasticcontainer",
+    "Date05_Sub06_stool_lift",
+    "Date05_Sub06_stool_sit",
+    "Date05_Sub06_suitcase_hand",
+    "Date05_Sub06_suitcase_lift",
+    "Date05_Sub06_tablesmall_hand",
+    "Date05_Sub06_tablesmall_lean",
+    "Date05_Sub06_tablesmall_lift",
+    "Date05_Sub06_tablesquare_lift",
+    "Date05_Sub06_tablesquare_move",
+    "Date05_Sub06_tablesquare_sit",
+    "Date05_Sub06_trashbin",
+    "Date05_Sub06_yogaball_play",
+    "Date05_Sub06_yogaball_sit",
+    "Date05_Sub06_yogamat",
+    "Date06_Sub07_backpack_back",
+    "Date06_Sub07_backpack_hand",
+    "Date06_Sub07_backpack_twohand",
+    "Date06_Sub07_boxlarge",
+    "Date06_Sub07_boxmedium",
+    "Date06_Sub07_boxsmall",
+    "Date06_Sub07_chairblack_hand",
+    "Date06_Sub07_chairblack_lift",
+    "Date06_Sub07_chairblack_sit",
+    "Date06_Sub07_chairwood_hand",
+    "Date06_Sub07_chairwood_lift",
+    "Date06_Sub07_chairwood_sit",
+    "Date06_Sub07_monitor_move",
+    "Date06_Sub07_plasticcontainer",
+    "Date06_Sub07_stool_lift",
+    "Date06_Sub07_suitcase_move",
+    "Date06_Sub07_tablesmall_lean",
+    "Date06_Sub07_tablesmall_lift",
+    "Date06_Sub07_tablesmall_move",
+    "Date06_Sub07_tablesquare_lift",
+    "Date06_Sub07_tablesquare_move",
+    "Date06_Sub07_tablesquare_sit",
+    "Date06_Sub07_trashbin",
+    "Date06_Sub07_yogaball_play",
+    "Date06_Sub07_yogaball_sit",
+    "Date07_Sub04_backpack_back",
+    "Date07_Sub04_backpack_hand",
+    "Date07_Sub04_backpack_twohand",
+    "Date07_Sub04_boxlarge",
+    "Date07_Sub04_boxmedium",
+    "Date07_Sub04_chairblack_hand",
+    "Date07_Sub04_chairblack_lift",
+    "Date07_Sub04_chairblack_sit",
+    "Date07_Sub04_chairwood_hand",
+    "Date07_Sub04_chairwood_lift",
+    "Date07_Sub04_chairwood_sit",
+    "Date07_Sub04_monitor_hand",
+    "Date07_Sub04_monitor_move",
+    "Date07_Sub04_stool_lift",
+    "Date07_Sub04_stool_sit",
+    "Date07_Sub04_suitcase_open",
+    "Date07_Sub04_tablesmall_lean",
+    "Date07_Sub04_tablesmall_lift",
+    "Date07_Sub04_tablesmall_move",
+    "Date07_Sub04_tablesquare_lift",
+    "Date07_Sub04_tablesquare_move",
+    "Date07_Sub04_tablesquare_sit",
+    "Date07_Sub04_toolbox_lift",
+    "Date07_Sub04_trashbin",
+    "Date07_Sub04_yogaball_play",
+    "Date07_Sub04_yogaball_sit",
+    "Date07_Sub04_yogamat",
+    "Date07_Sub05_suitcase_lift",
+    "Date07_Sub05_tablesquare",
+    "Date07_Sub08_backpack_back",
+    "Date07_Sub08_backpack_hand",
+    "Date07_Sub08_backpack_hug",
+    "Date07_Sub08_boxlarge",
+    "Date07_Sub08_boxlong",
+    "Date07_Sub08_boxmedium",
+    "Date07_Sub08_boxsmall",
+    "Date07_Sub08_chairblack_hand",
+    "Date07_Sub08_chairblack_lift",
+    "Date07_Sub08_chairblack_sit",
+    "Date07_Sub08_chairwood_hand",
+    "Date07_Sub08_chairwood_lift",
+    "Date07_Sub08_chairwood_sit",
+    "Date07_Sub08_monitor_hand",
+    "Date07_Sub08_monitor_move",
+    "Date07_Sub08_plasticcontainer",
+    "Date07_Sub08_stool",
+    "Date07_Sub08_suitcase",
+    "Date07_Sub08_tablesmall",
+    "Date07_Sub08_tablesquare",
+    "Date07_Sub08_toolbox",
+    "Date07_Sub08_trashbin",
+    "Date07_Sub08_yogaball"
+  ],
+  "error_frames": {
+  "Date01_Sub01_boxsmall_hand": [
+    "t0008.467"
+  ],
+  "Date01_Sub01_boxtiny_hand": [
+    "t0045.000"
+  ],
+  "Date02_Sub02_backpack_back": [
+    "t0035.200"
+  ],
+  "Date02_Sub02_backpack_hand": [
+    "t0031.800"
+  ],
+  "Date02_Sub02_backpack_twohand": [
+    "t0037.033"
+  ],
+  "Date02_Sub02_boxtiny_hand": [
+    "t0009.800",
+    "t0010.333",
+    "t0010.467",
+    "t0013.100",
+    "t0013.300"
+  ],
+  "Date02_Sub02_monitor_move2": [
+    "t0015.167"
+  ],
+  "Date02_Sub02_plasticcontainer": [
+    "t0022.300"
+  ],
+  "Date02_Sub02_toolbox": [
+    "t0010.433",
+    "t0033.000",
+    "t0040.467"
+  ],
+  "Date02_Sub02_toolbox_part2": [
+    "t0048.533"
+  ],
+  "Date02_Sub02_yogamat": [
+    "t0014.300"
+  ],
+  "Date03_Sub03_boxlong": [
+    "t0024.367"
+  ],
+  "Date03_Sub03_plasticcontainer": [
+    "t0045.633"
+  ],
+  "Date03_Sub04_boxsmall": [
+    "t0020.000",
+    "t0021.000",
+    "t0024.100",
+    "t0024.133",
+    "t0025.000",
+    "t0025.633",
+    "t0027.000"
+  ],
+  "Date03_Sub04_boxtiny": [
+    "t0005.967",
+    "t0006.967",
+    "t0007.833",
+    "t0019.000",
+    "t0020.400",
+    "t0020.467"
+  ],
+  "Date03_Sub04_boxtiny_part2": [
+    "t0021.133"
+  ],
+  "Date03_Sub04_plasticcontainer_lift": [
+    "t0023.233"
+  ],
+  "Date03_Sub04_toolbox": [
+    "t0032.000",
+    "t0034.600",
+    "t0035.467",
+    "t0035.633",
+    "t0036.167",
+    "t0036.200",
+    "t0036.967",
+    "t0041.533"
+  ],
+  "Date03_Sub05_boxsmall": [
+    "t0006.700",
+    "t0036.767"
+  ],
+  "Date03_Sub05_boxtiny": [
+    "t0006.433",
+    "t0008.100",
+    "t0009.167",
+    "t0010.200",
+    "t0010.433",
+    "t0011.800",
+    "t0011.933",
+    "t0013.400",
+    "t0038.000",
+    "t0040.000",
+    "t0042.433",
+    "t0045.067"
+  ],
+  "Date04_Sub05_boxlarge": [
+    "t0038.000",
+    "t0038.067",
+    "t0039.000"
+  ],
+  "Date04_Sub05_boxlong": [
+    "t0031.333"
+  ],
+  "Date04_Sub05_boxtiny": [
+    "t0030.800",
+    "t0031.000",
+    "t0036.233"
+  ],
+  "Date04_Sub05_stool": [
+    "t0020.000"
+  ],
+  "Date05_Sub06_boxlong": [
+    "t0038.167"
+  ],
+  "Date05_Sub06_boxtiny": [
+    "t0016.933",
+    "t0035.033",
+    "t0037.000",
+    "t0043.067",
+    "t0043.200",
+    "t0044.967",
+    "t0045.000",
+    "t0047.000",
+    "t0047.200"
+  ],
+  "Date05_Sub06_toolbox": [
+    "t0011.000",
+    "t0012.200",
+    "t0022.000"
+  ],
+  "Date06_Sub07_boxlong": [
+    "t0042.000"
+  ],
+  "Date06_Sub07_boxtiny": [
+    "t0004.633",
+    "t0004.800",
+    "t0004.867"
+  ],
+  "Date06_Sub07_stool_sit": [
+    "t0046.000",
+    "t0046.900"
+  ],
+  "Date06_Sub07_suitcase_lift": [
+    "t0006.933",
+    "t0007.000"
+  ],
+  "Date06_Sub07_toolbox": [
+    "t0007.733",
+    "t0008.000",
+    "t0012.000",
+    "t0013.000",
+    "t0040.000",
+    "t0041.000",
+    "t0044.000",
+    "t0047.000"
+  ],
+  "Date06_Sub07_yogamat": [
+    "t0028.000"
+  ],
+  "Date07_Sub04_boxlong": [
+    "t0012.767"
+  ],
+  "Date07_Sub04_boxsmall": [
+    "t0046.300"
+  ],
+  "Date07_Sub04_boxtiny": [
+    "t0008.667",
+    "t0009.333",
+    "t0017.367",
+    "t0038.600",
+    "t0040.000",
+    "t0040.067",
+    "t0040.900",
+    "t0042.000",
+    "t0042.100",
+    "t0042.133",
+    "t0042.400",
+    "t0042.800",
+    "t0042.900"
+  ],
+  "Date07_Sub04_plasticcontainer": [
+    "t0024.167",
+    "t0025.633",
+    "t0025.833"
+  ],
+  "Date07_Sub04_suitcase_lift": [
+    "t0024.000",
+    "t0025.000"
+  ],
+  "Date07_Sub05_suitcase_open": [
+    "t0031.000",
+    "t0032.000"
+  ],
+  "Date07_Sub05_tablesmall": [
+    "t0076.000",
+    "t0117.567"
+  ],
+  "Date07_Sub08_boxtiny": [
+    "t0017.933"
+  ],
+  "Date07_Sub08_yogamat": [
+    "t0005.867"
+  ]
+}
+}

data/preprocess/behave_test/behave_simple_test.npz.py

@@ -0,0 +1,37 @@
+# load split.json and make an npz with all the folders in the test split
+import argparse
+import json
+import os
+import glob
+import numpy as np
+
+BEHAVE_PATH = '/ps/project/datasets/BEHAVE/sequences/'
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default=BEHAVE_PATH)
+    parser.add_argument('--split_file', type=str, default='data/preprocess/behave_test/split.json')
+    parser.add_argument('--out_file', type=str, default='data/dataset_extras/behave/behave_simple_test.npz')
+    args = parser.parse_args()
+
+    with open(args.split_file, 'r') as f:
+        split = json.load(f)
+
+    test_split = split['test']
+
+    # structs we use
+    imgnames_ = []
+
+    data = {}
+    for seq_name in test_split:
+        print(seq_name)
+        seq_dir = os.path.join(args.data_dir, seq_name)
+        # get recusive images in the seq_dir folder
+        images = glob.glob(os.path.join(seq_dir, '**/*color.jpg'), recursive=True)
+        print(len(images))
+        images.sort()
+        imgnames_.extend(images)
+
+    np.savez(args.out_file, imgname=imgnames_,)
+    print('Saved to ', args.out_file)
+

data/preprocess/behave_test/split.json

@@ -0,0 +1,327 @@
+{
+  "train": [
+    "Date01_Sub01_backpack_back",
+    "Date01_Sub01_backpack_hand",
+    "Date01_Sub01_backpack_hug",
+    "Date01_Sub01_basketball",
+    "Date01_Sub01_boxlarge_hand",
+    "Date01_Sub01_boxlong_hand",
+    "Date01_Sub01_boxmedium_hand",
+    "Date01_Sub01_boxsmall_hand",
+    "Date01_Sub01_boxtiny_hand",
+    "Date01_Sub01_chairblack_hand",
+    "Date01_Sub01_chairblack_lift",
+    "Date01_Sub01_chairblack_sit",
+    "Date01_Sub01_chairwood_hand",
+    "Date01_Sub01_chairwood_lift",
+    "Date01_Sub01_chairwood_sit",
+    "Date01_Sub01_keyboard_move",
+    "Date01_Sub01_keyboard_typing",
+    "Date01_Sub01_monitor_hand",
+    "Date01_Sub01_monitor_move",
+    "Date01_Sub01_plasticcontainer",
+    "Date01_Sub01_stool_move",
+    "Date01_Sub01_stool_sit",
+    "Date01_Sub01_suitcase",
+    "Date01_Sub01_suitcase_lift",
+    "Date01_Sub01_tablesmall_lean",
+    "Date01_Sub01_tablesmall_lift",
+    "Date01_Sub01_tablesmall_move",
+    "Date01_Sub01_tablesquare_hand",
+    "Date01_Sub01_tablesquare_lift",
+    "Date01_Sub01_tablesquare_sit",
+    "Date01_Sub01_toolbox",
+    "Date01_Sub01_trashbin",
+    "Date01_Sub01_yogaball",
+    "Date01_Sub01_yogaball_play",
+    "Date01_Sub01_yogamat_hand",
+    "Date02_Sub02_backpack_back",
+    "Date02_Sub02_backpack_hand",
+    "Date02_Sub02_backpack_twohand",
+    "Date02_Sub02_basketball",
+    "Date02_Sub02_boxlarge_hand",
+    "Date02_Sub02_boxlong_hand",
+    "Date02_Sub02_boxmedium_hand",
+    "Date02_Sub02_boxsmall_hand",
+    "Date02_Sub02_boxtiny_hand",
+    "Date02_Sub02_chairblack_hand",
+    "Date02_Sub02_chairblack_lift",
+    "Date02_Sub02_chairblack_sit",
+    "Date02_Sub02_chairwood_hand",
+    "Date02_Sub02_chairwood_sit",
+    "Date02_Sub02_keyboard_move",
+    "Date02_Sub02_keyboard_typing",
+    "Date02_Sub02_monitor_hand",
+    "Date02_Sub02_monitor_move",
+    "Date02_Sub02_plasticcontainer",
+    "Date02_Sub02_stool_move",
+    "Date02_Sub02_stool_sit",
+    "Date02_Sub02_suitcase_ground",
+    "Date02_Sub02_suitcase_lift",
+    "Date02_Sub02_tablesmall_lean",
+    "Date02_Sub02_tablesmall_lift",
+    "Date02_Sub02_tablesmall_move",
+    "Date02_Sub02_tablesquare_lift",
+    "Date02_Sub02_tablesquare_move",
+    "Date02_Sub02_tablesquare_sit",
+    "Date02_Sub02_toolbox",
+    "Date02_Sub02_trashbin",
+    "Date02_Sub02_yogaball_play",
+    "Date02_Sub02_yogaball_sit",
+    "Date02_Sub02_yogamat",
+    "Date04_Sub05_backpack",
+    "Date04_Sub05_basketball",
+    "Date04_Sub05_boxlarge",
+    "Date04_Sub05_boxlong",
+    "Date04_Sub05_boxmedium",
+    "Date04_Sub05_boxsmall",
+    "Date04_Sub05_boxtiny",
+    "Date04_Sub05_chairblack",
+    "Date04_Sub05_chairwood",
+    "Date04_Sub05_keyboard",
+    "Date04_Sub05_monitor",
+    "Date04_Sub05_monitor_sit",
+    "Date04_Sub05_plasticcontainer",
+    "Date04_Sub05_stool",
+    "Date04_Sub05_suitcase",
+    "Date04_Sub05_suitcase_open",
+    "Date04_Sub05_tablesmall",
+    "Date04_Sub05_tablesquare",
+    "Date04_Sub05_toolbox",
+    "Date04_Sub05_trashbin",
+    "Date04_Sub05_yogaball",
+    "Date04_Sub05_yogamat",
+    "Date05_Sub05_backpack",
+    "Date05_Sub05_chairblack",
+    "Date05_Sub05_chairwood",
+    "Date05_Sub05_yogaball",
+    "Date05_Sub06_backpack_back",
+    "Date05_Sub06_backpack_hand",
+    "Date05_Sub06_backpack_twohand",
+    "Date05_Sub06_basketball",
+    "Date05_Sub06_boxlarge",
+    "Date05_Sub06_boxlong",
+    "Date05_Sub06_boxmedium",
+    "Date05_Sub06_boxsmall",
+    "Date05_Sub06_boxtiny",
+    "Date05_Sub06_chairblack_hand",
+    "Date05_Sub06_chairblack_lift",
+    "Date05_Sub06_chairblack_sit",
+    "Date05_Sub06_chairwood_hand",
+    "Date05_Sub06_chairwood_lift",
+    "Date05_Sub06_chairwood_sit",
+    "Date05_Sub06_keyboard_hand",
+    "Date05_Sub06_keyboard_move",
+    "Date05_Sub06_monitor_hand",
+    "Date05_Sub06_monitor_move",
+    "Date05_Sub06_plasticcontainer",
+    "Date05_Sub06_stool_lift",
+    "Date05_Sub06_stool_sit",
+    "Date05_Sub06_suitcase_hand",
+    "Date05_Sub06_suitcase_lift",
+    "Date05_Sub06_tablesmall_hand",
+    "Date05_Sub06_tablesmall_lean",
+    "Date05_Sub06_tablesmall_lift",
+    "Date05_Sub06_tablesquare_lift",
+    "Date05_Sub06_tablesquare_move",
+    "Date05_Sub06_tablesquare_sit",
+    "Date05_Sub06_toolbox",
+    "Date05_Sub06_trashbin",
+    "Date05_Sub06_yogaball_play",
+    "Date05_Sub06_yogaball_sit",
+    "Date05_Sub06_yogamat",
+    "Date06_Sub07_backpack_back",
+    "Date06_Sub07_backpack_hand",
+    "Date06_Sub07_backpack_twohand",
+    "Date06_Sub07_basketball",
+    "Date06_Sub07_boxlarge",
+    "Date06_Sub07_boxlong",
+    "Date06_Sub07_boxmedium",
+    "Date06_Sub07_boxsmall",
+    "Date06_Sub07_boxtiny",
+    "Date06_Sub07_chairblack_hand",
+    "Date06_Sub07_chairblack_lift",
+    "Date06_Sub07_chairblack_sit",
+    "Date06_Sub07_chairwood_hand",
+    "Date06_Sub07_chairwood_lift",
+    "Date06_Sub07_chairwood_sit",
+    "Date06_Sub07_keyboard_move",
+    "Date06_Sub07_keyboard_typing",
+    "Date06_Sub07_monitor_move",
+    "Date06_Sub07_plasticcontainer",
+    "Date06_Sub07_stool_lift",
+    "Date06_Sub07_stool_sit",
+    "Date06_Sub07_suitcase_lift",
+    "Date06_Sub07_suitcase_move",
+    "Date06_Sub07_tablesmall_lean",
+    "Date06_Sub07_tablesmall_lift",
+    "Date06_Sub07_tablesmall_move",
+    "Date06_Sub07_tablesquare_lift",
+    "Date06_Sub07_tablesquare_move",
+    "Date06_Sub07_tablesquare_sit",
+    "Date06_Sub07_toolbox",
+    "Date06_Sub07_trashbin",
+    "Date06_Sub07_yogaball_play",
+    "Date06_Sub07_yogaball_sit",
+    "Date06_Sub07_yogamat",
+    "Date07_Sub04_backpack_back",
+    "Date07_Sub04_backpack_hand",
+    "Date07_Sub04_backpack_twohand",
+    "Date07_Sub04_basketball",
+    "Date07_Sub04_boxlarge",
+    "Date07_Sub04_boxlong",
+    "Date07_Sub04_boxmedium",
+    "Date07_Sub04_boxsmall",
+    "Date07_Sub04_boxtiny",
+    "Date07_Sub04_chairblack_hand",
+    "Date07_Sub04_chairblack_lift",
+    "Date07_Sub04_chairblack_sit",
+    "Date07_Sub04_chairwood_hand",
+    "Date07_Sub04_chairwood_lift",
+    "Date07_Sub04_chairwood_sit",
+    "Date07_Sub04_keyboard_move",
+    "Date07_Sub04_keyboard_typing",
+    "Date07_Sub04_monitor_hand",
+    "Date07_Sub04_monitor_move",
+    "Date07_Sub04_plasticcontainer",
+    "Date07_Sub04_stool_lift",
+    "Date07_Sub04_stool_sit",
+    "Date07_Sub04_suitcase_lift",
+    "Date07_Sub04_suitcase_open",
+    "Date07_Sub04_tablesmall_lean",
+    "Date07_Sub04_tablesmall_lift",
+    "Date07_Sub04_tablesmall_move",
+    "Date07_Sub04_tablesquare_lift",
+    "Date07_Sub04_tablesquare_move",
+    "Date07_Sub04_tablesquare_sit",
+    "Date07_Sub04_toolbox_lift",
+    "Date07_Sub04_trashbin",
+    "Date07_Sub04_yogaball_play",
+    "Date07_Sub04_yogaball_sit",
+    "Date07_Sub04_yogamat",
+    "Date07_Sub05_suitcase_lift",
+    "Date07_Sub05_suitcase_open",
+    "Date07_Sub05_tablesmall",
+    "Date07_Sub05_tablesquare",
+    "Date07_Sub08_backpack_back",
+    "Date07_Sub08_backpack_hand",
+    "Date07_Sub08_backpack_hug",
+    "Date07_Sub08_basketball",
+    "Date07_Sub08_boxlarge",
+    "Date07_Sub08_boxlong",
+    "Date07_Sub08_boxmedium",
+    "Date07_Sub08_boxsmall",
+    "Date07_Sub08_boxtiny",
+    "Date07_Sub08_chairblack_hand",
+    "Date07_Sub08_chairblack_lift",
+    "Date07_Sub08_chairblack_sit",
+    "Date07_Sub08_chairwood_hand",
+    "Date07_Sub08_chairwood_lift",
+    "Date07_Sub08_chairwood_sit",
+    "Date07_Sub08_keyboard_move",
+    "Date07_Sub08_keyboard_typing",
+    "Date07_Sub08_monitor_hand",
+    "Date07_Sub08_monitor_move",
+    "Date07_Sub08_plasticcontainer",
+    "Date07_Sub08_stool",
+    "Date07_Sub08_suitcase",
+    "Date07_Sub08_tablesmall",
+    "Date07_Sub08_tablesquare",
+    "Date07_Sub08_toolbox",
+    "Date07_Sub08_trashbin",
+    "Date07_Sub08_yogaball",
+    "Date07_Sub08_yogamat"
+  ],
+  "test": [
+    "Date03_Sub03_backpack_back",
+    "Date03_Sub03_backpack_hand",
+    "Date03_Sub03_backpack_hug",
+    "Date03_Sub03_basketball",
+    "Date03_Sub03_boxlarge",
+    "Date03_Sub03_boxlong",
+    "Date03_Sub03_boxmedium",
+    "Date03_Sub03_boxsmall",
+    "Date03_Sub03_boxtiny",
+    "Date03_Sub03_chairblack_hand",
+    "Date03_Sub03_chairblack_lift",
+    "Date03_Sub03_chairblack_sit",
+    "Date03_Sub03_chairblack_sitstand",
+    "Date03_Sub03_chairwood_hand",
+    "Date03_Sub03_chairwood_lift",
+    "Date03_Sub03_chairwood_sit",
+    "Date03_Sub03_keyboard_move",
+    "Date03_Sub03_keyboard_typing",
+    "Date03_Sub03_monitor_move",
+    "Date03_Sub03_plasticcontainer",
+    "Date03_Sub03_stool_lift",
+    "Date03_Sub03_stool_sit",
+    "Date03_Sub03_suitcase_lift",
+    "Date03_Sub03_suitcase_move",
+    "Date03_Sub03_tablesmall_lean",
+    "Date03_Sub03_tablesmall_lift",
+    "Date03_Sub03_tablesmall_move",
+    "Date03_Sub03_tablesquare_lift",
+    "Date03_Sub03_tablesquare_move",
+    "Date03_Sub03_tablesquare_sit",
+    "Date03_Sub03_toolbox",
+    "Date03_Sub03_trashbin",
+    "Date03_Sub03_yogaball_play",
+    "Date03_Sub03_yogaball_sit",
+    "Date03_Sub03_yogamat",
+    "Date03_Sub04_backpack_back",
+    "Date03_Sub04_backpack_hand",
+    "Date03_Sub04_backpack_hug",
+    "Date03_Sub04_basketball",
+    "Date03_Sub04_boxlarge",
+    "Date03_Sub04_boxlong",
+    "Date03_Sub04_boxmedium",
+    "Date03_Sub04_boxsmall",
+    "Date03_Sub04_boxtiny",
+    "Date03_Sub04_chairblack_hand",
+    "Date03_Sub04_chairblack_liftreal",
+    "Date03_Sub04_chairblack_sit",
+    "Date03_Sub04_chairwood_hand",
+    "Date03_Sub04_chairwood_lift",
+    "Date03_Sub04_chairwood_sit",
+    "Date03_Sub04_keyboard_move",
+    "Date03_Sub04_keyboard_typing",
+    "Date03_Sub04_monitor_hand",
+    "Date03_Sub04_monitor_move",
+    "Date03_Sub04_plasticcontainer_lift",
+    "Date03_Sub04_stool_move",
+    "Date03_Sub04_stool_sit",
+    "Date03_Sub04_suitcase_ground",
+    "Date03_Sub04_suitcase_lift",
+    "Date03_Sub04_tablesmall_hand",
+    "Date03_Sub04_tablesmall_lean",
+    "Date03_Sub04_tablesmall_lift",
+    "Date03_Sub04_tablesquare_hand",
+    "Date03_Sub04_tablesquare_lift",
+    "Date03_Sub04_tablesquare_sit",
+    "Date03_Sub04_toolbox",
+    "Date03_Sub04_trashbin",
+    "Date03_Sub04_yogaball_play",
+    "Date03_Sub04_yogaball_sit",
+    "Date03_Sub04_yogamat",
+    "Date03_Sub05_backpack",
+    "Date03_Sub05_basketball",
+    "Date03_Sub05_boxlarge",
+    "Date03_Sub05_boxlong",
+    "Date03_Sub05_boxmedium",
+    "Date03_Sub05_boxsmall",
+    "Date03_Sub05_boxtiny",
+    "Date03_Sub05_chairblack",
+    "Date03_Sub05_chairwood",
+    "Date03_Sub05_keyboard",
+    "Date03_Sub05_monitor",
+    "Date03_Sub05_plasticcontainer",
+    "Date03_Sub05_stool",
+    "Date03_Sub05_suitcase",
+    "Date03_Sub05_tablesmall",
+    "Date03_Sub05_tablesquare",
+    "Date03_Sub05_toolbox",
+    "Date03_Sub05_trashbin",
+    "Date03_Sub05_yogaball",
+    "Date03_Sub05_yogamat"
+  ]
+}

data/preprocess/hot_dca.py

@@ -0,0 +1,220 @@
+import os
+import cv2
+import numpy as np
+from tqdm import tqdm
+import sys
+import imagesize
+import argparse
+import torch
+import pandas as pd
+import json
+
+import monai.metrics as metrics
+
+HOT_TRAIN_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_train.odgt"
+HOT_VAL_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_validation.odgt"
+HOT_TEST_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_test.odgt"
+
+def metric(mask, pred, back=True):
+  iou = metrics.compute_meaniou(pred, mask, back, False)
+  iou = iou.mean()
+  return iou
+
+
+def combine_hot_prox_split(split):
+    if split == 'train':
+        with open(HOT_TRAIN_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'val':
+        with open(HOT_VAL_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'test':
+        with open(HOT_TEST_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    return records
+
+def hot_extract(img_dataset_path, smpl_params_path, dca_csv_path, out_dir, split=None, vis_path=None, visualize=False, include_supporting=True):
+
+    n_vertices = 6890
+
+    # structs we use
+    imgnames_ = []
+    poses_, shapes_, transls_ = [], [], []
+    cams_k_ = []
+    polygon_2d_contact_ = []
+    contact_3d_labels_ = []
+    scene_seg_, part_seg_ = [], []
+
+    img_dir = os.path.join(img_dataset_path, 'images', 'training')
+    smpl_params = np.load(smpl_params_path)
+    # smpl_params = np.load(smpl_params_path, allow_pickle=True)
+    # smpl_params = smpl_params['arr_0'].item()
+    annotations_dir = img_dir.replace('images', 'annotations')
+    records = combine_hot_prox_split(split)
+
+    # load dca csv
+    dca_csv = pd.read_csv(dca_csv_path)
+
+    iou_thresh = 0
+
+    num_with_3d_contact = 0
+
+    focal_length_accumulator = []
+    for i, record in enumerate(tqdm(records, dynamic_ncols=True)):
+        imgpath = record['fpath_img']
+        imgname = os.path.basename(imgpath)
+        # save image in temp_images
+        if visualize:
+            img = cv2.imread(os.path.join(img_dir, imgname))
+            cv2.imwrite(os.path.join(vis_path, os.path.basename(imgname)), img)
+
+        # load image to get the size
+        img_w, img_h = record["width"], record["height"]
+
+        # get mask anns
+        polygon_2d_contact_path = os.path.join(annotations_dir, os.path.splitext(imgname)[0] + '.png')
+
+
+        # Get 3D contact annotations from DCA mturk csv
+        dca_row = dca_csv.loc[dca_csv['imgnames'] == imgname] # if no imgnames column, run scripts/datascripts/add_imgname_column_to_deco_csv.py
+        if len(dca_row) == 0:
+            contact_3d_labels = []
+            continue
+        else:
+            num_with_3d_contact += 1
+            supporting_object = dca_row['supporting_object'].values[0]
+            vertices = eval(dca_row['vertices'].values[0])
+            contact_3d_list = vertices[os.path.join('hot/training/', imgname)]
+            # Aggregate values in all keys
+            contact_3d_idx = []
+            for item in contact_3d_list:
+                # one iteration loop as it is a list of one dict key value
+                for k, v in item.items():
+                    if include_supporting:
+                        contact_3d_idx.extend(v)
+                    else:
+                        if k != 'SUPPORTING':
+                            contact_3d_idx.extend(v)
+            # removed repeated values
+            contact_3d_idx = list(set(contact_3d_idx))
+            contact_3d_labels = np.zeros(n_vertices) # smpl has 6980 vertices
+            contact_3d_labels[contact_3d_idx] = 1.
+
+        # find indices that match the imname
+        inds = np.where(smpl_params['imgname'] == os.path.join(img_dir, imgname))[0]
+        select_inds = []
+        ious = []
+        for ind in inds:
+            # part mask
+            part_path = smpl_params['part_seg'][ind]
+            # load the part_mask
+            part_mask = cv2.imread(part_path)
+            # binarize the part mask
+            part_mask = np.where(part_mask > 0, 1, 0)
+            # save part mask
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(part_path)), part_mask*255)
+
+            # load gt polygon mask
+            polygon_2d_contact = cv2.imread(polygon_2d_contact_path)
+            # binarize the gt polygon mask
+            polygon_2d_contact = np.where(polygon_2d_contact > 0, 1, 0)
+
+            # save gt polygon mask in temp_images
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(polygon_2d_contact_path)), polygon_2d_contact*255)
+
+            polygon_2d_contact = torch.from_numpy(polygon_2d_contact)[None,:].permute(0,3,1,2)
+            part_mask = torch.from_numpy(part_mask)[None,:].permute(0,3,1,2)
+            # compute iou with part mask and gt polygon mask
+            iou = metric(polygon_2d_contact, part_mask)
+            if iou > iou_thresh:
+                ious.append(iou)
+                select_inds.append(ind)
+
+        # get select_ind with maximum iou
+        if len(select_inds) > 0:
+            max_iou_ind = select_inds[np.argmax(ious)]
+        else:
+            continue
+
+        # part mask
+        part_path = smpl_params['part_seg'][max_iou_ind]
+
+        # scene mask
+        scene_path = smpl_params['scene_seg'][max_iou_ind]
+
+        # get smpl params
+        pose = smpl_params['pose'][max_iou_ind]
+        shape = smpl_params['shape'][max_iou_ind]
+        transl = smpl_params['global_t'][max_iou_ind]
+        focal_length = smpl_params['focal_l'][max_iou_ind]
+        camC = np.array([[img_w//2, img_h//2]])
+
+        # read GT 2D keypoints
+        K = np.eye(3, dtype=np.float64)
+        K[0, 0] = focal_length
+        K[1, 1] = focal_length
+        K[:2, 2:] = camC.T
+
+        # store data
+        imgnames_.append(os.path.join(img_dir, imgname))
+        polygon_2d_contact_.append(polygon_2d_contact_path)
+        # we use the heuristic that the 3D contact labeled is for the person with maximum iou with HOT contacts
+        contact_3d_labels_.append(contact_3d_labels)
+        scene_seg_.append(scene_path)
+        part_seg_.append(part_path)
+        poses_.append(pose.squeeze())
+        transls_.append(transl.squeeze())
+        shapes_.append(shape.squeeze())
+        cams_k_.append(K.tolist())
+        focal_length_accumulator.append(focal_length)
+
+    print('Average focal length: ', np.mean(focal_length_accumulator))
+    print('Median focal length: ', np.median(focal_length_accumulator))
+    print('Std Dev focal length: ', np.std(focal_length_accumulator))
+
+    # store the data struct
+    os.makedirs(out_dir, exist_ok=True)
+    out_file = os.path.join(out_dir, f'hot_dca_supporting_{str(include_supporting)}_{split}.npz')
+    np.savez(out_file, imgname=imgnames_,
+                       pose=poses_,
+                       transl=transls_,
+                       shape=shapes_,
+                       cam_k=cams_k_,
+                       polygon_2d_contact=polygon_2d_contact_,
+                       contact_label=contact_3d_labels_,
+                        scene_seg=scene_seg_,
+                        part_seg=part_seg_
+             )
+    print(f'Total number of rows: {len(imgnames_)}')
+    print('Saved to ', out_file)
+    print(f'Number of images with 3D contact labels: {num_with_3d_contact}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--img_dataset_path', type=str, default='/ps/project/datasets/HOT/Contact_Data/')
+    parser.add_argument('--smpl_params_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/hot.npz')
+    parser.add_argument('--dca_csv_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/dca.csv')
+    parser.add_argument('--out_dir', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/data/dataset_extras')
+    parser.add_argument('--vis_path', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/temp_images')
+    parser.add_argument('--visualize', action='store_true', default=False)
+    parser.add_argument('--include_supporting', action='store_true', default=False)
+    parser.add_argument('--split', type=str, default='train')
+    args = parser.parse_args()
+
+    hot_extract(img_dataset_path=args.img_dataset_path,
+                smpl_params_path=args.smpl_params_path,
+                dca_csv_path=args.dca_csv_path,
+                out_dir=args.out_dir,
+                vis_path=args.vis_path,
+                visualize=args.visualize,
+                split=args.split,
+                include_supporting=args.include_supporting)
+

data/preprocess/hot_noprox.py

@@ -0,0 +1,241 @@
+import os
+import cv2
+import numpy as np
+from tqdm import tqdm
+import sys
+import imagesize
+import argparse
+import torch
+import pandas as pd
+import json
+
+import monai.metrics as metrics
+
+HOT_TRAIN_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_train.odgt"
+HOT_VAL_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_validation.odgt"
+HOT_TEST_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/hot_test.odgt"
+
+def metric(mask, pred, back=True):
+  iou = metrics.compute_meaniou(pred, mask, back, False)
+  iou = iou.mean()
+  return iou
+
+
+def combine_hot_prox_split(split):
+    if split == 'train':
+        with open(HOT_TRAIN_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'val':
+        with open(HOT_VAL_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'test':
+        with open(HOT_TEST_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'trainval':
+        with open(HOT_TRAIN_SPLIT, "r") as f:
+            train_records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+        with open(HOT_VAL_SPLIT, "r") as f:
+            val_records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+        records = train_records + val_records
+    return records
+
+def hot_extract(img_dataset_path, smpl_params_path, dca_csv_path, out_dir, split=None, vis_path=None, visualize=False, record_idx=None, include_supporting=True):
+
+    n_vertices = 6890
+
+    # structs we use
+    imgnames_ = []
+    poses_, shapes_, transls_ = [], [], []
+    cams_k_ = []
+    polygon_2d_contact_ = []
+    contact_3d_labels_ = []
+    scene_seg_, part_seg_ = [], []
+
+    img_dir = os.path.join(img_dataset_path, 'images', 'training')
+    smpl_params = np.load(smpl_params_path)
+    # smpl_params = np.load(smpl_params_path, allow_pickle=True)
+    # smpl_params = smpl_params['arr_0'].item()
+    annotations_dir = img_dir.replace('images', 'annotations')
+    records = combine_hot_prox_split(split)
+    # split records list into 4 sublists
+    if record_idx is not None:
+        records = np.array_split(records, 4)[record_idx]
+
+    # load dca csv
+    dca_csv = pd.read_csv(dca_csv_path)
+
+    iou_thresh = 0
+
+    num_with_3d_contact = 0
+
+    focal_length_accumulator = []
+    for i, record in enumerate(tqdm(records, dynamic_ncols=True)):
+        imgpath = record['fpath_img']
+        imgname = os.path.basename(imgpath)
+        # save image in temp_images
+        if visualize:
+            img = cv2.imread(os.path.join(img_dir, imgname))
+            cv2.imwrite(os.path.join(vis_path, os.path.basename(imgname)), img)
+
+        # load image to get the size
+        img_w, img_h = record["width"], record["height"]
+
+        # get mask anns
+        polygon_2d_contact_path = os.path.join(annotations_dir, os.path.splitext(imgname)[0] + '.png')
+
+
+        # Get 3D contact annotations from DCA mturk csv
+        dca_row = dca_csv.loc[dca_csv['imgnames'] == imgname] # if no imgnames column, run scripts/datascripts/add_imgname_column_to_deco_csv.py
+        if len(dca_row) == 0:
+            contact_3d_labels = []
+        else:
+            num_with_3d_contact += 1
+            supporting_object = dca_row['supporting_object'].values[0]
+            vertices = eval(dca_row['vertices'].values[0])
+            contact_3d_list = vertices[os.path.join('hot/training/', imgname)]
+            # Aggregate values in all keys
+            contact_3d_idx = []
+            for item in contact_3d_list:
+                # one iteration loop as it is a list of one dict key value
+                for k, v in item.items():
+                    if include_supporting:
+                        contact_3d_idx.extend(v)
+                    else:
+                        if k != 'SUPPORTING':
+                            contact_3d_idx.extend(v)
+            # removed repeated values
+            contact_3d_idx = list(set(contact_3d_idx))
+            contact_3d_labels = np.zeros(n_vertices) # smpl has 6980 vertices
+            contact_3d_labels[contact_3d_idx] = 1.
+
+        # find indices that match the imname
+        inds = np.where(smpl_params['imgname'] == os.path.join(img_dir, imgname))[0]
+        select_inds = []
+        ious = []
+        for ind in inds:
+            # part mask
+            part_path = smpl_params['part_seg'][ind]
+            # load the part_mask
+            part_mask = cv2.imread(part_path)
+            # binarize the part mask
+            part_mask = np.where(part_mask > 0, 1, 0)
+            # save part mask
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(part_path)), part_mask*255)
+
+            # load gt polygon mask
+            polygon_2d_contact = cv2.imread(polygon_2d_contact_path)
+            # binarize the gt polygon mask
+            polygon_2d_contact = np.where(polygon_2d_contact > 0, 1, 0)
+
+            # save gt polygon mask in temp_images
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(polygon_2d_contact_path)), polygon_2d_contact*255)
+
+            polygon_2d_contact = torch.from_numpy(polygon_2d_contact)[None,:].permute(0,3,1,2)
+            part_mask = torch.from_numpy(part_mask)[None,:].permute(0,3,1,2)
+            # compute iou with part mask and gt polygon mask
+            iou = metric(polygon_2d_contact, part_mask)
+            if iou > iou_thresh:
+                ious.append(iou)
+                select_inds.append(ind)
+
+        # get select_ind with maximum iou
+        if len(select_inds) > 0:
+            max_iou_ind = select_inds[np.argmax(ious)]
+        else:
+            continue
+
+        for ind in select_inds:
+            # part mask
+            part_path = smpl_params['part_seg'][ind]
+
+            # scene mask
+            scene_path = smpl_params['scene_seg'][ind]
+
+            # get smpl params
+            pose = smpl_params['pose'][ind]
+            shape = smpl_params['shape'][ind]
+            transl = smpl_params['global_t'][ind]
+            focal_length = smpl_params['focal_l'][ind]
+            camC = np.array([[img_w//2, img_h//2]])
+
+            # read GT 2D keypoints
+            K = np.eye(3, dtype=np.float64)
+            K[0, 0] = focal_length
+            K[1, 1] = focal_length
+            K[:2, 2:] = camC.T
+
+            # store data
+            imgnames_.append(os.path.join(img_dir, imgname))
+            polygon_2d_contact_.append(polygon_2d_contact_path)
+            # we use the heuristic that the 3D contact labeled is for the person with maximum iou with HOT contacts
+            if ind == max_iou_ind:
+                contact_3d_labels_.append(contact_3d_labels)
+            else:
+                contact_3d_labels_.append([])
+            scene_seg_.append(scene_path)
+            part_seg_.append(part_path)
+            poses_.append(pose.squeeze())
+            transls_.append(transl.squeeze())
+            shapes_.append(shape.squeeze())
+            cams_k_.append(K.tolist())
+        focal_length_accumulator.append(focal_length)
+
+    print('Average focal length: ', np.mean(focal_length_accumulator))
+    print('Median focal length: ', np.median(focal_length_accumulator))
+    print('Std Dev focal length: ', np.std(focal_length_accumulator))
+
+    # store the data struct
+    os.makedirs(out_dir, exist_ok=True)
+    if record_idx is not None:
+        out_file = os.path.join(out_dir, f'hot_noprox_supporting_{str(include_supporting)}_{split}_{record_idx}.npz')
+    else:
+        out_file = os.path.join(out_dir, f'hot_noprox_supporting_{str(include_supporting)}_{split}_combined.npz')
+    np.savez(out_file, imgname=imgnames_,
+                       pose=poses_,
+                       transl=transls_,
+                       shape=shapes_,
+                       cam_k=cams_k_,
+                       polygon_2d_contact=polygon_2d_contact_,
+                       contact_label=contact_3d_labels_,
+                        scene_seg=scene_seg_,
+                        part_seg=part_seg_
+             )
+    print(f'Total number of rows: {len(imgnames_)}')
+    print('Saved to ', out_file)
+    print(f'Number of images with 3D contact labels: {num_with_3d_contact}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--img_dataset_path', type=str, default='/ps/project/datasets/HOT/Contact_Data/')
+    parser.add_argument('--smpl_params_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/hot.npz')
+    parser.add_argument('--dca_csv_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/dca.csv')
+    parser.add_argument('--out_dir', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/data/dataset_extras')
+    parser.add_argument('--vis_path', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/temp_images')
+    parser.add_argument('--visualize', action='store_true', default=False)
+    parser.add_argument('--include_supporting', action='store_true', default=False)
+    parser.add_argument('--record_idx', type=int, default=None)
+    parser.add_argument('--split', type=str, default='train')
+    args = parser.parse_args()
+
+    hot_extract(img_dataset_path=args.img_dataset_path,
+                smpl_params_path=args.smpl_params_path,
+                dca_csv_path=args.dca_csv_path,
+                out_dir=args.out_dir,
+                vis_path=args.vis_path,
+                visualize=args.visualize,
+                split=args.split,
+                record_idx=args.record_idx,
+                include_supporting=args.include_supporting)
+

data/preprocess/hot_prox.py

@@ -0,0 +1,217 @@
+import os
+import cv2
+import numpy as np
+from tqdm import tqdm
+import sys
+import imagesize
+import argparse
+import torch
+import pandas as pd
+import json
+
+import monai.metrics as metrics
+
+PROX_TRAIN_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/prox_train.odgt"
+PROX_VAL_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/prox_validation.odgt"
+PROX_TEST_SPLIT = "/ps/scratch/ps_shared/ychen2/4shashank/split/prox_test.odgt"
+
+def metric(mask, pred, back=True):
+  iou = metrics.compute_meaniou(pred, mask, back, False)
+  iou = iou.mean()
+  return iou
+
+
+def combine_hot_prox_split(split):
+    if split == 'train':
+        with open(PROX_TRAIN_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'val':
+        with open(PROX_VAL_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    elif split == 'test':
+        with open(PROX_TEST_SPLIT, "r") as f:
+            records = [
+                json.loads(line.strip("\n")) for line in f.readlines()
+            ]
+    return records
+
+def hot_extract(img_dataset_path, smpl_params_path, dca_csv_path, out_dir, split=None, vis_path=None, visualize=False, downsample_factor=4):
+
+    n_vertices = 6890
+
+    # structs we use
+    imgnames_ = []
+    poses_, shapes_, transls_ = [], [], []
+    cams_k_ = []
+    polygon_2d_contact_ = []
+    contact_3d_labels_ = []
+    scene_seg_, part_seg_ = [], []
+
+    img_dir = os.path.join(img_dataset_path, 'images', 'training')
+    smpl_params = np.load(smpl_params_path)
+    # smpl_params = np.load(smpl_params_path, allow_pickle=True)
+    # smpl_params = smpl_params['arr_0'].item()
+    annotations_dir = img_dir.replace('images', 'annotations')
+    records = combine_hot_prox_split(split)
+
+    # load dca csv
+    dca_csv = pd.read_csv(dca_csv_path)
+
+    iou_thresh = 0
+
+    num_with_3d_contact = 0
+
+    focal_length_accumulator = []
+    for i, record in enumerate(tqdm(records, dynamic_ncols=True)):
+        imgpath = record['fpath_img']
+        imgname = os.path.basename(imgpath)
+        # save image in temp_images
+        if visualize:
+            img = cv2.imread(os.path.join(img_dir, imgname))
+            cv2.imwrite(os.path.join(vis_path, os.path.basename(imgname)), img)
+
+        # load image to get the size
+        img_w, img_h = record["width"], record["height"]
+
+        # get mask anns
+        polygon_2d_contact_path = os.path.join(annotations_dir, os.path.splitext(imgname)[0] + '.png')
+
+
+        # Get 3D contact annotations from DCA mturk csv
+        dca_row = dca_csv.loc[dca_csv['imgnames'] == imgname] # if no imgnames column, run scripts/datascripts/add_imgname_column_to_deco_csv.py
+        if len(dca_row) == 0:
+            contact_3d_labels = []
+        else:
+            num_with_3d_contact += 1
+            supporting_object = dca_row['supporting_object'].values[0]
+            vertices = eval(dca_row['vertices'].values[0])
+            contact_3d_list = vertices[os.path.join('hot/training/', imgname)]
+            # Aggregate values in all keys
+            contact_3d_idx = []
+            for item in contact_3d_list:
+                # one iteration loop as it is a list of one dict key value
+                for k, v in item.items():
+                    contact_3d_idx.extend(v)
+            # removed repeated values
+            contact_3d_idx = list(set(contact_3d_idx))
+            contact_3d_labels = np.zeros(n_vertices) # smpl has 6980 vertices
+            contact_3d_labels[contact_3d_idx] = 1.
+
+        # find indices that match the imname
+        inds = np.where(smpl_params['imgname'] == os.path.join(img_dir, imgname))[0]
+        select_inds = []
+        ious = []
+        for ind in inds:
+            # part mask
+            part_path = smpl_params['part_seg'][ind]
+            # load the part_mask
+            part_mask = cv2.imread(part_path)
+            # binarize the part mask
+            part_mask = np.where(part_mask > 0, 1, 0)
+            # save part mask
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(part_path)), part_mask*255)
+
+            # load gt polygon mask
+            polygon_2d_contact = cv2.imread(polygon_2d_contact_path)
+            # binarize the gt polygon mask
+            polygon_2d_contact = np.where(polygon_2d_contact > 0, 1, 0)
+
+            # save gt polygon mask in temp_images
+            if visualize:
+                cv2.imwrite(os.path.join(vis_path, os.path.basename(polygon_2d_contact_path)), polygon_2d_contact*255)
+
+            polygon_2d_contact = torch.from_numpy(polygon_2d_contact)[None,:].permute(0,3,1,2)
+            part_mask = torch.from_numpy(part_mask)[None,:].permute(0,3,1,2)
+            # compute iou with part mask and gt polygon mask
+            iou = metric(polygon_2d_contact, part_mask)
+            if iou > iou_thresh:
+                ious.append(iou)
+                select_inds.append(ind)
+
+        # get select_ind with maximum iou
+        if len(select_inds) > 0:
+            max_iou_ind = select_inds[np.argmax(ious)]
+        else:
+            continue
+
+        for ind in select_inds:
+            # part mask
+            part_path = smpl_params['part_seg'][ind]
+
+            # scene mask
+            scene_path = smpl_params['scene_seg'][ind]
+
+            # get smpl params
+            pose = smpl_params['pose'][ind]
+            shape = smpl_params['shape'][ind]
+            transl = smpl_params['global_t'][ind]
+            focal_length = smpl_params['focal_l'][ind]
+            camC = np.array([[img_w//2, img_h//2]])
+
+            # read GT 2D keypoints
+            K = np.eye(3, dtype=np.float64)
+            K[0, 0] = focal_length
+            K[1, 1] = focal_length
+            K[:2, 2:] = camC.T
+
+            # store data
+            imgnames_.append(os.path.join(img_dir, imgname))
+            polygon_2d_contact_.append(polygon_2d_contact_path)
+            # we use the heuristic that the 3D contact labeled is for the person with maximum iou with HOT contacts
+            if ind == max_iou_ind:
+                contact_3d_labels_.append(contact_3d_labels)
+            else:
+                contact_3d_labels_.append([])
+            scene_seg_.append(scene_path)
+            part_seg_.append(part_path)
+            poses_.append(pose.squeeze())
+            transls_.append(transl.squeeze())
+            shapes_.append(shape.squeeze())
+            cams_k_.append(K.tolist())
+        focal_length_accumulator.append(focal_length)
+
+    print('Average focal length: ', np.mean(focal_length_accumulator))
+    print('Median focal length: ', np.median(focal_length_accumulator))
+    print('Std Dev focal length: ', np.std(focal_length_accumulator))
+
+    # store the data struct
+    os.makedirs(out_dir, exist_ok=True)
+    out_file = os.path.join(out_dir, f'hot_prox_{split}.npz')
+    np.savez(out_file, imgname=imgnames_,
+                       pose=poses_,
+                       transl=transls_,
+                       shape=shapes_,
+                       cam_k=cams_k_,
+                       polygon_2d_contact=polygon_2d_contact_,
+                       contact_label=contact_3d_labels_,
+                        scene_seg=scene_seg_,
+                        part_seg=part_seg_
+             )
+    print(f'Total number of rows: {len(imgnames_)}')
+    print('Saved to ', out_file)
+    print(f'Number of images with 3D contact labels: {num_with_3d_contact}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--img_dataset_path', type=str, default='/ps/project/datasets/HOT/Contact_Data/')
+    parser.add_argument('--smpl_params_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/hot.npz')
+    parser.add_argument('--dca_csv_path', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/dca.csv')
+    parser.add_argument('--out_dir', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/data/dataset_extras')
+    parser.add_argument('--vis_path', type=str, default='/is/cluster/work/stripathi/pycharm_remote/dca_contact/temp_images')
+    parser.add_argument('--visualize', action='store_true', default=False)
+    parser.add_argument('--split', type=str, default='train')
+    args = parser.parse_args()
+
+    hot_extract(img_dataset_path=args.img_dataset_path,
+                smpl_params_path=args.smpl_params_path,
+                dca_csv_path=args.dca_csv_path,
+                out_dir=args.out_dir,
+                vis_path=args.vis_path,
+                visualize=args.visualize,
+                split=args.split)
+

data/preprocess/prepare_damon_behave_split.py

@@ -0,0 +1,69 @@
+import os.path as osp
+import os
+import shutil
+import json
+import argparse
+import numpy as np
+from PIL import Image
+from tqdm import tqdm
+
+objects = {
+    "backpack": 24,
+    "chair": 56,
+    "keyboard": 66,
+    "suitcase": 28
+}
+
+def copy_images_to_behave_format(in_img_dir, in_image_list, in_part_dir, in_seg_dir, out_dir):
+    """
+    Copy images from in_img_dir to out_dir
+    :param in_img_dir: input directory containing images
+    :param out_dir: output directory to copy images to
+    :return:
+    """
+    # read image list
+    with open(in_image_list, 'r') as fp:
+        img_list_dict = json.load(fp)
+
+    for k, v in img_list_dict.items():
+        out_dir_object = osp.join(out_dir, k)
+        os.makedirs(out_dir_object, exist_ok=True)
+        # copy images to out_dir
+        for img_name in tqdm(v, dynamic_ncols=True):
+            input_image_path = osp.join(in_img_dir, img_name)
+            input_part_path = osp.join(in_part_dir, img_name.replace('.jpg', '_0.png'))
+            input_seg_path = osp.join(in_seg_dir, img_name.replace('.jpg', '.png'))
+            if not osp.exists(input_part_path) or not osp.exists(input_image_path) or not osp.exists(input_seg_path):
+                print(f'{input_image_path} or {input_part_path} or {input_seg_path} does not exist')
+                continue
+            out_dir_image = osp.join(out_dir_object, img_name)
+            os.makedirs(out_dir_image, exist_ok=True)
+            shutil.copy(input_image_path, osp.join(out_dir_image, 'k1.color.jpg'))
+
+            # load body mask
+            body_mask = Image.open(input_part_path)
+            # convert all non-zero pixels to 255
+            body_mask = np.array(body_mask)
+            body_mask[body_mask > 0] = 255
+            body_mask = Image.fromarray(body_mask)
+            body_mask.save(osp.join(out_dir_image, 'k1.person_mask.png'))
+
+            # load seg mask
+            body_mask = Image.open(input_seg_path)
+            # convert all non-object pixels to 255
+            body_mask = np.array(body_mask)
+            object_num = objects[k]
+            body_mask[body_mask == object_num] = 255
+            body_mask[body_mask != 255] = 0
+            body_mask = Image.fromarray(body_mask)
+            body_mask.save(osp.join(out_dir_image, 'k1.object_rend.png'))
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--in_img_dir', type=str, default='/ps/project/datasets/HOT/Contact_Data/images/training')
+    parser.add_argument('--in_part_dir', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot/parts/training')
+    parser.add_argument('--in_seg_dir', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot_behave_split/agniv/masks')
+    parser.add_argument('--in_image_list', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot_behave_split/imgnames_per_object_dict.json')
+    parser.add_argument('--out_dir', type=str, default='/ps/scratch/ps_shared/stripathi/deco/4agniv/hot_behave_split/training')
+    args = parser.parse_args()
+    copy_images_to_behave_format(args.in_img_dir, args.in_image_list, args.in_part_dir, args.in_seg_dir, args.out_dir)

data/preprocess/rich_smplx.py

@@ -0,0 +1,222 @@
+import os
+os.environ["CDF_LIB"] = "/is/cluster/scratch/stripathi/data/cdf37_1-dist/src/lib"
+
+import cv2
+import pandas as pd
+import json
+import glob
+import h5py
+import torch
+import trimesh
+import numpy as np
+import pickle as pkl
+from xml.dom import minidom
+import xml.etree.ElementTree as ET
+from tqdm import tqdm
+from spacepy import pycdf
+# from .read_openpose import read_openpose
+import sys
+sys.path.append('../../')
+from models import hmr, SMPL
+import config
+import constants
+
+import shutil
+
+import smplx
+import pytorch3d.transforms as p3dt
+
+from utils.geometry import batch_rodrigues, batch_rot2aa, ea2rm
+
+
+model_type = 'smplx'
+model_folder = '/ps/project/common/smplifyx/models/'
+body_model_params = dict(model_path=model_folder,
+                         model_type=model_type,
+                         create_global_orient=True,
+                         create_body_pose=True,
+                         create_betas=True,
+                         num_betas=10,
+                         create_left_hand_pose=True,
+                         create_right_hand_pose=True,
+                         create_expression=True,
+                         create_jaw_pose=True,
+                         create_leye_pose=True,
+                         create_reye_pose=True,
+                         create_transl=True,
+                         use_pca=False)
+body_model = smplx.create(gender='neutral', **body_model_params).to('cuda')
+
+def rich_extract(img_dataset_path, out_path, split=None, vis_path=None, visualize=False, downsample_factor=4):
+
+    # structs we use
+    imgnames_ = []
+    poses_, shapes_, transls_ = [], [], []
+    cams_k_ = []
+    contact_label_ = []
+    scene_seg_, part_seg_ = [], []
+    
+    for i, fl in tqdm(enumerate(sorted(os.listdir(os.path.join(img_dataset_path, 'images', split)))), dynamic_ncols=True):
+        ind = fl.index('cam')
+        location = fl[:ind-1]
+
+        cam_num = fl[ind:ind+6]
+
+        img = fl[ind+7:-3] + 'jpeg'
+
+        imgname = os.path.join(location, cam_num, img)
+
+        mask_name = fl
+        sp = mask_name.split('_')
+        indx = mask_name.index('cam')
+        st = mask_name[indx-1:indx+7]
+        mask_name = mask_name.replace(st, '/')
+        mask_name = mask_name[:-7]
+        new_p = mask_name.split('/')
+        mask_name = new_p[0] + '/' + new_p[1] + '/' + sp[1] + '.pkl'
+        mask_path = os.path.join(img_dataset_path, 'labels', split, mask_name)
+        df = pd.read_pickle(mask_path)
+        mask = df['contact'] 
+
+        scene_path = os.path.join(img_dataset_path, 'segmentation_masks', split, fl[:-3] + 'png')
+
+        part_path = os.path.join(img_dataset_path, 'parts', split, fl[:-3] + 'png')
+
+        dataset_path = '/ps/project/datasets/RICH'
+
+        ind = fl.index('cam')
+        frame_id = fl[:ind-1]
+        location = frame_id.split('_')[0]
+
+        if location == 'LectureHall':
+            if 'chair' in frame_id:
+                cam2world_location = location + '_' + 'chair'
+            else:
+                cam2world_location = location + '_' + 'yoga'  
+        else:
+            cam2world_location = location        
+
+        img_num = fl.split('_')[-2]
+
+        cam_num = int(fl.split('_')[-1][:2])
+
+        # get ioi2scan transformation per sequence
+        ioi2scan_fn = os.path.join(dataset_path, 'website_release/multicam2world', cam2world_location + '_multicam2world.json')
+
+        try:
+            camera_fn = os.path.join(dataset_path, 'rich_toolkit/data/scan_calibration', location, f'calibration/{cam_num:03d}.xml')
+            focal_length_x, focal_length_y, camC, camR, camT, _, _, _ = extract_cam_param_xml(camera_fn)
+        except:
+            print(f'camera calibration file not found: {camera_fn}')
+            continue
+
+        # path to smpl params
+        smplx_param = os.path.join(dataset_path, 'rich_toolkit/data/bodies', split, frame_id, str(img_num), frame_id.split('_')[1] + '.pkl')
+
+        # get smpl parameters
+        ## body resides in multi-ioi coordidate, where camera 0 is world zero.
+        with open(smplx_param, 'rb') as f:
+            body_params = pkl.load(f)
+            # in ioi coordinates: cam 0
+            beta = body_params['betas']
+            pose_aa = body_params['body_pose']
+            pose_rotmat = p3dt.axis_angle_to_matrix(torch.FloatTensor(pose_aa.reshape(-1,3))).numpy()
+
+            transl = body_params['transl']
+            global_orient = body_params['global_orient']
+            global_orient = p3dt.axis_angle_to_matrix(torch.FloatTensor(global_orient.reshape(-1,3))).numpy()
+
+        smpl_body_cam0 = body_model(betas=torch.FloatTensor(beta).to('cuda')) # canonical body with shape
+        vertices_cam0 = smpl_body_cam0.vertices.detach().cpu().numpy().squeeze()
+        joints_cam0 = smpl_body_cam0.joints.detach().cpu().numpy()
+        pelvis_cam0 = joints_cam0[:, 0, :]
+
+        # ## rigid transformation between multi-ioi and Leica scan (world)
+        with open(ioi2scan_fn, 'r') as f:
+            ioi2scan_dict = json.load(f)
+            R_ioi2world = np.array(ioi2scan_dict['R']) # Note: R is transposed
+            t_ioi2world= np.array(ioi2scan_dict['t']).reshape(1, 3)     
+
+        # # get SMPL params in camera coordinates
+        global_orient_cam = np.matmul(np.array(camR), global_orient)
+        full_pose_rotmat_cam = np.concatenate((global_orient_cam, pose_rotmat), axis=0).squeeze()
+        theta_cam = batch_rot2aa(torch.FloatTensor(full_pose_rotmat_cam)).reshape(-1, 66).cpu().numpy()
+
+        # read GT 2D keypoints
+        K = np.eye(3, dtype=np.float64)
+        K[0, 0] = focal_length_x / downsample_factor
+        K[1, 1] = focal_length_y / downsample_factor
+        K[:2, 2:] = camC.T / downsample_factor
+
+        # get camera parameters wrt to scan
+        R_worldtocam = np.matmul(camR, R_ioi2world) # Note: R_ioi2world is transposed
+        T_worldtocam = -t_ioi2world + camT
+
+        # store data
+        imgnames_.append(os.path.join('/ps/project/datasets/RICH_JPG', split, imgname))
+        contact_label_.append(mask)
+        scene_seg_.append(scene_path)
+        part_seg_.append(part_path)
+        poses_.append(theta_cam.squeeze())
+        transls_.append(transl.squeeze())
+        shapes_.append(beta.squeeze())
+        cams_k_.append(K.tolist())
+
+    # store the data struct
+    if not os.path.isdir(out_path):
+        os.makedirs(out_path)
+    out_file = os.path.join(out_path, f'rich_{split}_smplx.npz')
+    np.savez(out_file, imgname=imgnames_,
+                       pose=poses_,
+                       transl=transls_,
+                       shape=shapes_,
+                       cam_k=cams_k_,
+                       contact_label=contact_label_,
+                        scene_seg=scene_seg_,
+                        part_seg=part_seg_
+             )
+    print('Saved to ', out_file)
+
+def rectify_pose(camera_r, body_aa):
+    body_r = batch_rodrigues(body_aa).reshape(-1,3,3)
+    final_r = camera_r @ body_r
+    body_aa = batch_rot2aa(final_r)
+    return body_aa
+
+
+def extract_cam_param_xml(xml_path: str = '', dtype=float):
+    import xml.etree.ElementTree as ET
+    tree = ET.parse(xml_path)
+
+    extrinsics_mat = [float(s) for s in tree.find('./CameraMatrix/data').text.split()]
+    intrinsics_mat = [float(s) for s in tree.find('./Intrinsics/data').text.split()]
+    distortion_vec = [float(s) for s in tree.find('./Distortion/data').text.split()]
+
+    focal_length_x = intrinsics_mat[0]
+    focal_length_y = intrinsics_mat[4]
+    center = np.array([[intrinsics_mat[2], intrinsics_mat[5]]], dtype=dtype)
+
+    rotation = np.array([[extrinsics_mat[0], extrinsics_mat[1], extrinsics_mat[2]],
+                         [extrinsics_mat[4], extrinsics_mat[5], extrinsics_mat[6]],
+                         [extrinsics_mat[8], extrinsics_mat[9], extrinsics_mat[10]]], dtype=dtype)
+
+    translation = np.array([[extrinsics_mat[3], extrinsics_mat[7], extrinsics_mat[11]]], dtype=dtype)
+
+    # t = -Rc --> c = -R^Tt
+    cam_center = [-extrinsics_mat[0] * extrinsics_mat[3] - extrinsics_mat[4] * extrinsics_mat[7] - extrinsics_mat[8] *
+                  extrinsics_mat[11],
+                  -extrinsics_mat[1] * extrinsics_mat[3] - extrinsics_mat[5] * extrinsics_mat[7] - extrinsics_mat[9] *
+                  extrinsics_mat[11],
+                  -extrinsics_mat[2] * extrinsics_mat[3] - extrinsics_mat[6] * extrinsics_mat[7] - extrinsics_mat[10] *
+                  extrinsics_mat[11]]
+
+    cam_center = np.array([cam_center], dtype=dtype)
+
+    k1 = np.array([distortion_vec[0]], dtype=dtype)
+    k2 = np.array([distortion_vec[1]], dtype=dtype)
+
+    return focal_length_x, focal_length_y, center, rotation, translation, cam_center, k1, k2
+
+rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/work/achatterjee/rich/npzs', split='train')
+rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/work/achatterjee/rich/npzs', split='val')
+rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/work/achatterjee/rich/npzs', split='test')

data/preprocess/rich_smplx_agniv.py

@@ -0,0 +1,578 @@
+import os
+os.environ["CDF_LIB"] = "/is/cluster/scratch/stripathi/data/cdf37_1-dist/src/lib"
+
+import cv2
+import pandas as pd
+# import json
+# import glob
+# import h5py
+import torch
+# import trimesh
+import numpy as np
+import pickle as pkl
+# from xml.dom import minidom
+# import xml.etree.ElementTree as ET
+from tqdm import tqdm
+from spacepy import pycdf
+# from .read_openpose import read_openpose
+import sys
+sys.path.append('../../')
+# from models import hmr, SMPL
+# import config
+# import constants
+import argparse
+
+# import shutil
+
+import smplx
+import pytorch3d.transforms as p3dt
+
+from utils.geometry import batch_rodrigues, batch_rot2aa, ea2rm
+# from vis_utils.world_vis import overlay_mesh, vis_smpl_with_ground
+
+
+model_type = 'smplx'
+model_folder = '/ps/project/common/smplifyx/models/'
+body_model_params = dict(model_path=model_folder,
+                         model_type=model_type,
+                         create_global_orient=True,
+                         create_body_pose=True,
+                         create_betas=True,
+                         num_betas=10,
+                         create_left_hand_pose=True,
+                         create_right_hand_pose=True,
+                         create_expression=True,
+                         create_jaw_pose=True,
+                         create_leye_pose=True,
+                         create_reye_pose=True,
+                         create_transl=True,
+                         use_pca=False)
+body_model = smplx.create(gender='neutral', **body_model_params).to('cuda')
+
+def rich_extract(img_dataset_path, out_path, scene_indx, split=None, vis_path=None, visualize=False, downsample_factor=1):
+    
+    # structs we use
+    imgnames_ = []
+    # scales_, centers_, parts_, Ss_, Ss_world_, openposes_ = [], [], [], [], [], []
+    poses_, shapes_, transls_ = [], [], []
+    # poses_world_, transls_world_, cams_r_, cams_t_ = [], [], [], []
+    cams_k_ = []
+    # ground_offset_ = []
+    # in_bos_label_ = []
+    contact_label_ = []
+    scene_seg_, part_seg_ = [], []
+
+    # # seqs in validation set
+    # if split == 'val':
+    #     seq_list = ['2021-06-15_Multi_IOI_ID_00176_Yoga2',
+    #                 '2021-06-15_Multi_IOI_ID_00228_Yoga1',
+    #                 '2021-06-15_Multi_IOI_ID_03588_Yoga1',
+    #                 '2021-06-15_Multi_IOI_ID_00176_Yoga1']
+
+    # # seqs in testing set
+    # if split == 'test':
+    #     seq_list = ['2021-06-15_Multi_IOI_ID_00186_Yoga1',
+    #                 '2021-06-15_Multi_IOI_ID_03588_Yoga2',
+    #                 'MultiIOI_201019_ID03581_parkingLot_Calibration06_Settings06_PushUp__2',
+    #                 'Multi-IOI_ID00227_Scene_ParkingLot_Calibration_03_CameraSettings_4_pushup_1']
+    
+    scenes = sorted(os.listdir('/ps/project/datasets/RICH/rich_toolkit/data/images/test'))
+    # scene = scenes[scene_indx]
+    scene = 'ParkingLot2_009_impro5'
+    
+    for scene_name in [scene]:
+    # for scene_name in ['LectureHall_003_wipingchairs1']:
+        out_file = os.path.join(out_path, f'rich_{scene_name}_smplx.npz')
+        if os.path.exists(out_file): return
+        print(scene_name)
+        for i, fl in tqdm(enumerate(sorted(os.listdir(os.path.join(img_dataset_path, 'images', split)))), dynamic_ncols=True):
+            if not scene_name in fl: continue
+
+            ind = fl.index('cam')
+            location = fl[:ind-1]
+
+            cam_num = fl[ind:ind+6]
+
+            img = fl[ind+7:-3] + 'jpeg'
+
+            imgname = os.path.join(location, cam_num, img)
+
+            mask_name = fl
+            sp = mask_name.split('_')
+            indx = mask_name.index('cam')
+            st = mask_name[indx-1:indx+7]
+            mask_name = mask_name.replace(st, '/')
+            mask_name = mask_name[:-7]
+            new_p = mask_name.split('/')
+            mask_name = new_p[0] + '/' + new_p[1] + '/' + sp[1] + '.pkl'
+            mask_path = os.path.join(img_dataset_path, 'labels', split, mask_name)
+            df = pd.read_pickle(mask_path)
+            mask = df['contact'] 
+
+            scene_path = os.path.join('/ps/scratch/ps_shared/stripathi/deco/4agniv/rich/seg_masks_new', split, fl[:-3] + 'png')
+
+            part_path = os.path.join('/ps/scratch/ps_shared/stripathi/deco/4agniv/rich/part_masks_new', split, fl[:-3] + 'png')
+
+            dataset_path = '/ps/project/datasets/RICH'
+
+            ind = fl.index('cam')
+            frame_id = fl[:ind-1]
+            location = frame_id.split('_')[0]
+
+            if location == 'LectureHall':
+                if 'chair' in frame_id:
+                    cam2world_location = location + '_' + 'chair'
+                else:
+                    cam2world_location = location + '_' + 'yoga'  
+            else:
+                cam2world_location = location        
+
+            img_num = fl.split('_')[-2]
+
+            cam_num = int(fl.split('_')[-1][:2])
+
+            # get ioi2scan transformation per sequence
+            ioi2scan_fn = os.path.join(dataset_path, 'website_release/multicam2world', cam2world_location + '_multicam2world.json')
+
+            try:
+                camera_fn = os.path.join(dataset_path, 'rich_toolkit/data/scan_calibration', location, f'calibration/{cam_num:03d}.xml')
+                focal_length_x, focal_length_y, camC, camR, camT, _, _, _ = extract_cam_param_xml(camera_fn)
+            except:
+                print(f'camera calibration file not found: {camera_fn}')
+                continue
+
+            # print('X: ', focal_length_x)
+            # print('Y: ', focal_length_y)
+            
+            # path to smpl params
+            smplx_param = os.path.join(dataset_path, 'rich_toolkit/data/bodies', split, frame_id, str(img_num), frame_id.split('_')[1] + '.pkl')
+
+            # # path to GT bounding boxes
+            # bbox_path = os.path.join(dataset_path, 'preprocessed', split, frame_id, img_num, frame_id.split('_')[1], 'bbox_refine', f'{img_num}_{cam_num:02d}.json')
+            # # path with 2D openpose keypoints
+            # openpose_path = os.path.join(dataset_path, 'preprocessed', split, frame_id, img_num, frame_id.split('_')[1], 'keypoints_refine', f'{img_num}_{str(cam_num).zfill(2)}_keypoints.json')
+            # # path to image crops
+            # img_path = os.path.join(dataset_path, 'preprocessed', split, frame_id, img_num, frame_id.split('_')[1], 'images_refine', f'{img_num}_{cam_num:02d}.png')
+
+            # # bbox file
+            # try:
+            #     with open(bbox_path, 'r') as f:
+            #         bbox_dict = json.load(f)
+            # except:
+            #     print(f'bbox file not found: {bbox_path}')
+            #     continue
+
+            # # read GT bounding box
+            # x1_ul = bbox_dict['x1'] // downsample_factor
+            # y1_ul = bbox_dict['y1'] // downsample_factor
+            # x2_br = bbox_dict['x2'] // downsample_factor
+            # y2_br = bbox_dict['y2'] // downsample_factor
+            # bbox = np.array([x1_ul, y1_ul, x2_br, y2_br])
+            # center = [(bbox[2]+bbox[0])/2, (bbox[3]+bbox[1])/2]
+            # scale = 0.9 * max(bbox[2] - bbox[0], bbox[3] - bbox[1]) / 200.
+
+            # get smpl parameters
+            ## body resides in multi-ioi coordidate, where camera 0 is world zero.
+            with open(smplx_param, 'rb') as f:
+                body_params = pkl.load(f)
+                # in ioi coordinates: cam 0
+                beta = body_params['betas']
+                pose_aa = body_params['body_pose']
+                pose_rotmat = p3dt.axis_angle_to_matrix(torch.FloatTensor(pose_aa.reshape(-1,3))).numpy()
+
+                transl = body_params['transl']
+                global_orient = body_params['global_orient']
+                global_orient = p3dt.axis_angle_to_matrix(torch.FloatTensor(global_orient.reshape(-1,3))).numpy()
+
+            smpl_body_cam0 = body_model(betas=torch.FloatTensor(beta).to('cuda')) # canonical body with shape
+            vertices_cam0 = smpl_body_cam0.vertices.detach().cpu().numpy().squeeze()
+            joints_cam0 = smpl_body_cam0.joints.detach().cpu().numpy()
+            pelvis_cam0 = joints_cam0[:, 0, :]
+
+            # ## rigid transformation between multi-ioi and Leica scan (world)
+            # with open(ioi2scan_fn, 'r') as f:
+            #     ioi2scan_dict = json.load(f)
+            #     R_ioi2world = np.array(ioi2scan_dict['R']) # Note: R is transposed
+            #     t_ioi2world= np.array(ioi2scan_dict['t']).reshape(1, 3)     
+
+            # # # get SMPL params in world coordinates
+            # # # import ipdb; ipdb.set_trace()
+            # global_orient_world = np.matmul(R_ioi2world.T, global_orient)
+            # transl_world = np.matmul((pelvis_cam0+transl), R_ioi2world) + t_ioi2world - pelvis_cam0 # right multiplication to avoid transpose
+            # full_pose_rotmat_world = np.concatenate((global_orient_world, pose_rotmat), axis=0).squeeze()
+            # theta_world = batch_rot2aa(torch.FloatTensor(full_pose_rotmat_world)).reshape(-1, 66).cpu().numpy()
+
+            # smpl_body_world = body_model(betas=torch.FloatTensor(beta).to('cuda'),
+            #                             body_pose=torch.FloatTensor(theta_world[:, 3:]).to('cuda'),
+            #                             transl=torch.FloatTensor(transl_world).to('cuda'),
+            #                             global_orient=torch.FloatTensor(theta_world[:, :3]).to('cuda'))
+            # vertices_world = smpl_body_world.vertices.detach().cpu().numpy().squeeze()
+            # joints3d_world = smpl_body_world.joints[:, 25:, :].detach().cpu().numpy().squeeze()
+
+            # # get SMPL params in camera coordinates
+            global_orient_cam = np.matmul(np.array(camR), global_orient)
+            transl_cam = np.matmul(camR, (pelvis_cam0 + transl).T).T + camT - pelvis_cam0
+            full_pose_rotmat_cam = np.concatenate((global_orient_cam, pose_rotmat), axis=0).squeeze()
+            theta_cam = batch_rot2aa(torch.FloatTensor(full_pose_rotmat_cam)).reshape(-1, 66).cpu().numpy()
+
+            # read GT 2D keypoints
+            K = np.eye(3, dtype=np.float64)
+            K[0, 0] = focal_length_x / downsample_factor
+            K[1, 1] = focal_length_y / downsample_factor
+            K[:2, 2:] = camC.T / downsample_factor
+
+            # # get openpose 2D keypoints
+            # try:
+            #     with open(openpose_path, 'r') as f:
+            #         openpose = json.load(f)
+            #     openpose = np.array(openpose['people'][0]['pose_keypoints_2d']).reshape([-1, 3])
+            # except:
+            #     print(f'No openpose !! Missing {openpose_path}')
+            #     continue
+
+            # get camera parameters wrt to scan
+            # R_worldtocam = np.matmul(camR, R_ioi2world) # Note: R_ioi2world is transposed
+            # T_worldtocam = -t_ioi2world + camT
+
+            # ground offset
+            # ground_offset = ground_eq[2]
+
+            # store data
+            jpg_img_path = os.path.join('/ps/project/datasets/RICH_JPG', split, imgname)
+            bmp_img_path = jpg_img_path.replace('/ps/project/datasets/RICH_JPG', '/ps/project/datasets/RICH/rich_toolkit/data/images')
+            bmp_img_path = bmp_img_path.replace('.jpeg', '.bmp')
+            if not os.path.exists(bmp_img_path):
+                bmp_img_path = bmp_img_path.replace('.bmp', '.png')
+            imgnames_.append(bmp_img_path)
+            contact_label_.append(mask)
+            scene_seg_.append(scene_path)
+            part_seg_.append(part_path)
+            # centers_.append(center)
+            # scales_.append(scale)
+            # openposes_.append(openpose)
+            poses_.append(theta_cam.squeeze())
+            transls_.append(transl_cam.squeeze())
+            # poses_world_.append(theta_world.squeeze())
+            # transls_world_.append(transl_world.squeeze())
+            shapes_.append(beta.squeeze())
+            # cams_r_.append(R_worldtocam.tolist())
+            # # Todo: note that T_worldtocam here is (1,3) whereas in h36m T_worldtocam is (1,3)
+            # cams_t_.append(T_worldtocam.tolist())
+            cams_k_.append(K.tolist())
+            # ground_offset_.append(ground_offset)
+
+        # for seq_i in tqdm(seq_list):
+        #     print(f'Processing sequence: {seq_i}')
+
+        #     # path with GT bounding boxes
+        #     params_path = os.path.join(dataset_path, seq_i, 'params')
+
+        #     # path to metadata for files
+        #     md_path = os.path.join(dataset_path, seq_i, 'data')
+
+        #     # glob all folders in params path
+        #     frame_param_paths = sorted(glob.glob(os.path.join(params_path, '*')))
+        #     frame_param_paths = [p for p in frame_param_paths if '.yaml' not in p]
+
+        #     # get ioi2scan transformation per sequence
+        #     ioi2scan_fn = os.path.join(dataset_path, seq_i, 'cam2scan.json')
+
+        #     ## ground resides in Leica scan coordinate, which is (roughly) axis aligned.
+        #     # ground_mesh = trimesh.load(os.path.join(dataset_path, seq_i, 'ground_mesh.ply'), process=False)
+        #     # ground_eq = np.mean(ground_mesh.vertices, axis=0)
+
+        #     # list all files in the folder
+        #     cam_files = os.listdir(os.path.join(dataset_path, seq_i, f'calibration'))
+        #     cam_list = sorted([int(os.path.splitext(f)[0]) for f in cam_files if '.xml' in f])
+
+        #     # if split == 'val':
+        #     #     cam_list = cam_list[1:] # remove first camera in val
+        #     for cam_num in cam_list[:1]:
+        #         camera_fn = os.path.join(dataset_path, seq_i, f'calibration/{cam_num:03d}.xml')
+        #         focal_length_x, focal_length_y, camC, camR, camT, _, _, _ = extract_cam_param_xml(camera_fn)
+
+        #         for frame_param_path in tqdm(frame_param_paths):
+        #             frame_id = os.path.basename(frame_param_path)
+        #             frame_num = int(frame_id)
+
+        #             # path to smpl params
+        #             try:
+        #                 smplx_param = os.path.join(frame_param_path, '00', 'results/000.pkl')
+        #             except:
+        #                 import ipdb; ipdb.set_trace()
+
+        #             # path to GT bounding boxes
+        #             bbox_path = os.path.join(md_path, frame_id, '00', 'bbox_refine', f'{frame_id}_{cam_num:02d}.json')
+        #             # path with 2D openpose keypoints
+        #             openpose_path = os.path.join(md_path, frame_id, '00', 'keypoints_refine', f'{frame_id}_{str(cam_num).zfill(2)}_keypoints.json')
+        #             # path to image crops
+        #             if downsample_factor == 1:
+        #                 img_path = os.path.join(md_path, frame_id, '00', 'images_orig', f'{frame_id}_{cam_num:02d}.png')
+        #             else:
+        #                 img_path = os.path.join(md_path, frame_id, '00', 'images_orig_720p', f'{frame_id}_{cam_num:02d}.png')
+
+        #             if not os.path.isfile(img_path):
+        #                 print(f'image not found: {img_path}')
+        #                 continue
+        #                 # raise FileNotFoundError
+
+        #             # bbox file
+        #             try:
+        #                 with open(bbox_path, 'r') as f:
+        #                     bbox_dict = json.load(f)
+        #             except:
+        #                 print(f'bbox file not found: {bbox_path}')
+        #                 continue
+        #             # read GT bounding box
+        #             x1_ul = bbox_dict['x1'] // downsample_factor
+        #             y1_ul = bbox_dict['y1'] // downsample_factor
+        #             x2_br = bbox_dict['x2'] // downsample_factor
+        #             y2_br = bbox_dict['y2'] // downsample_factor
+        #             bbox = np.array([x1_ul, y1_ul, x2_br, y2_br])
+        #             center = [(bbox[2]+bbox[0])/2, (bbox[3]+bbox[1])/2]
+        #             scale = 0.9 * max(bbox[2] - bbox[0], bbox[3] - bbox[1]) / 200.
+
+        #             # get smpl parameters
+        #             ## body resides in multi-ioi coordidate, where camera 0 is world zero.
+        #             with open(smplx_param, 'rb') as f:
+        #                 body_params = pkl.load(f)
+        #                 # in ioi coordinates: cam 0
+        #                 beta = body_params['betas']
+        #                 pose_aa = body_params['body_pose']
+        #                 pose_rotmat = p3dt.axis_angle_to_matrix(torch.FloatTensor(pose_aa.reshape(-1,3))).numpy()
+
+        #                 transl = body_params['transl']
+        #                 global_orient = body_params['global_orient']
+        #                 global_orient = p3dt.axis_angle_to_matrix(torch.FloatTensor(global_orient.reshape(-1,3))).numpy()
+
+        #             smpl_body_cam0 = body_model(betas=torch.FloatTensor(beta).to('cuda')) # canonical body with shape
+        #             vertices_cam0 = smpl_body_cam0.vertices.detach().cpu().numpy().squeeze()
+        #             joints_cam0 = smpl_body_cam0.joints.detach().cpu().numpy()
+        #             pelvis_cam0 = joints_cam0[:, 0, :]
+
+
+        #             ## rigid transformation between multi-ioi and Leica scan (world)
+        #             with open(ioi2scan_fn, 'r') as f:
+        #                 ioi2scan_dict = json.load(f)
+        #                 R_ioi2world = np.array(ioi2scan_dict['R']) # Note: R is transposed
+        #                 t_ioi2world= np.array(ioi2scan_dict['t']).reshape(1, 3)
+
+        #             # get SMPL params in world coordinates
+        #             # import ipdb; ipdb.set_trace()
+        #             global_orient_world = np.matmul(R_ioi2world.T, global_orient)
+        #             transl_world = np.matmul((pelvis_cam0+transl), R_ioi2world) + t_ioi2world - pelvis_cam0 # right multiplication to avoid transpose
+        #             full_pose_rotmat_world = np.concatenate((global_orient_world, pose_rotmat), axis=0).squeeze()
+        #             theta_world = batch_rot2aa(torch.FloatTensor(full_pose_rotmat_world)).reshape(-1, 66).cpu().numpy()
+
+        #             smpl_body_world = body_model(betas=torch.FloatTensor(beta).to('cuda'),
+        #                                 body_pose=torch.FloatTensor(theta_world[:, 3:]).to('cuda'),
+        #                                 transl=torch.FloatTensor(transl_world).to('cuda'),
+        #                                 global_orient=torch.FloatTensor(theta_world[:, :3]).to('cuda'))
+        #             vertices_world = smpl_body_world.vertices.detach().cpu().numpy().squeeze()
+        #             joints3d_world = smpl_body_world.joints[:, 25:, :].detach().cpu().numpy().squeeze()
+
+        #             mesh = trimesh.Trimesh(vertices_world, body_model.faces,
+        #                                    process=False,
+        #                                    maintain_order=True)
+        #             mesh.export('gt_mesh_world_smplx.obj')
+
+
+
+        #             # smpl_body_world = body_model(betas=torch.FloatTensor(beta).to('cuda'),
+        #             #                     body_pose=torch.FloatTensor(pose_rotmat[None, ...]).to('cuda'),
+        #             #                     transl=torch.FloatTensor(transl_world[None, ...]).to('cuda'),
+        #             #                     global_orient=torch.FloatTensor(global_orient_world[None, ...]).to('cuda'),
+        #             #                     left_hand_pose=torch.eye(3).reshape(1, 1, 3, 3).expand(1, 15, -1, -1).to('cuda'),
+        #             #                     right_hand_pose=torch.eye(3).reshape(1, 1, 3, 3).expand(1, 15, -1, -1).to('cuda'),
+        #             #                     leye_pose= torch.eye(3).reshape(1, 1, 3, 3).expand(batch_size, -1, -1, -1),
+        #             #
+        #             #                       pose2rot=False)
+        #             # vertices_world = smpl_body_world.vertices.detach().cpu().numpy().squeeze()
+        #             # joints3d_world = smpl_body_world.joints[:, 25:, :].detach().cpu().numpy().squeeze()
+
+        #             # mesh = trimesh.Trimesh(vertices_world, body_model.faces,
+        #             #                        process=False,
+        #             #                        maintain_order=True)
+        #             # mesh.export('gt_mesh_world_smplx.obj')
+
+        #             # get SMPL params in camera coordinates
+        #             global_orient_cam = np.matmul(camR, global_orient)
+        #             transl_cam = np.matmul(camR, (pelvis_cam0 + transl).T).T + camT - pelvis_cam0
+        #             full_pose_rotmat_cam = np.concatenate((global_orient_cam, pose_rotmat), axis=0).squeeze()
+        #             theta_cam = batch_rot2aa(torch.FloatTensor(full_pose_rotmat_cam)).reshape(-1, 66).cpu().numpy()
+        #             # smpl_body_cam = body_model(betas=torch.FloatTensor(beta).to('cuda'),
+        #             #                          body_pose=torch.FloatTensor(pose_rotmat).to('cuda'),
+        #             #                          transl=torch.FloatTensor(transl_cam).to('cuda'),
+        #             #                            global_orient=torch.FloatTensor(global_orient_cam).to('cuda'),
+        #             #                            pose2rot=False)
+        #             # vertices_cam = smpl_body_cam.vertices.detach().cpu().numpy().squeeze()
+        #             # joints3d_cam = smpl_body_cam.joints[:, 25:, :].detach().cpu().numpy().squeeze()
+        #             #
+        #             # mesh = trimesh.Trimesh(vertices_cam, body_model.faces,
+        #             #                        process=False,
+        #             #                        maintain_order=True)
+        #             # mesh.export('mesh_in_cam0.obj')
+
+        #             # read GT 2D keypoints
+        #             K = np.eye(3, dtype=np.float)
+        #             K[0, 0] = focal_length_x / downsample_factor
+        #             K[1, 1] = focal_length_y / downsample_factor
+        #             K[:2, 2:] = camC.T / downsample_factor
+        #             # projected_points = (K @ joints3d_cam.T).T
+        #             # joints2d = projected_points[:, :2] / np.hstack((projected_points[:, 2:], projected_points[:, 2:]))
+        #             # part = np.hstack((joints2d, np.ones((joints2d.shape[0], 1))))
+
+        #             # get openpose 2D keypoints
+        #             try:
+        #                 with open(openpose_path, 'r') as f:
+        #                     openpose = json.load(f)
+        #                 openpose = np.array(openpose['people'][0]['pose_keypoints_2d']).reshape([-1, 3])
+        #             except:
+        #                 print(f'No openpose !! Missing {openpose_path}')
+        #                 continue
+
+        #             # get camera parameters wrt to scan
+        #             R_worldtocam = np.matmul(camR, R_ioi2world) # Note: R_ioi2world is transposed
+        #             T_worldtocam = -t_ioi2world + camT
+
+        #             # ground offset
+        #             # ground_offset = ground_eq[2]/
+
+        #             # # get stability labels: 1: stable, 0: unstable but in contact, -1: unstable and not `in contact
+        #             # in_bos_label, contact_label, contact_mask = vis_smpl_with_ground(theta_world, transl_world, beta, seq_i,
+        #             #                                                      vis_path,
+        #             #                                                      start_idx=frame_num,
+        #             #                                                      sub_sample=1,
+        #             #                                                      ground_offset=ground_offset,
+        #             #                                                      smpl_batch_size=1,
+        #             #                                                      visualize=False)
+        #             # in_bos_label = in_bos_label.detach().cpu().numpy()
+        #             # contact_label = contact_label.detach().cpu().numpy()
+        #             # contact_mask = contact_mask.detach().cpu().numpy()
+
+        #             # visualize world smpl on ground plane
+        #             # if visualize:
+        #             #     if cam_num == 0:
+        #             #         vis_smpl_with_ground(theta_world, transl_world, beta, split+'_'+seq_i, vis_path,
+        #             #                              start_idx=frame_num,
+        #             #                              sub_sample=1,
+        #             #                              ground_offset=ground_offset,
+        #             #                              smpl_batch_size=1,
+        #             #                              visualize=True)
+
+
+        #             # ## visualize projected points
+        #             # img = cv2.imread(img_path)
+        #             # joints2d = joints2d.astype(np.int)
+        #             # img[joints2d[:, 1], joints2d[:, 0], :] = [0, 255, 0]
+
+        #             # read GT 3D pose in cam coordinates
+        #             # S24 = joints3d_cam
+        #             # pelvis_cam = (S24[[2], :] + S24[[3], :]) / 2
+        #             # S24 -= pelvis_cam
+        #             # S24 = np.hstack([S24, np.ones((S24.shape[0], 1))])
+
+        #             # read GT 3D pose in world coordinates
+        #             # S24_world = joints3d_world
+        #             # S24_world = np.hstack([S24_world, np.ones((S24_world.shape[0], 1))])
+
+        #             # store data
+        #             imgnames_.append(img_path)
+        #             centers_.append(center)
+        #             scales_.append(scale)
+        #             # parts_.append(part)
+        #             # Ss_.append(S24)
+        #             # Ss_world_.append(S24_world)
+        #             openposes_.append(openpose)
+        #             poses_.append(theta_cam.squeeze())
+        #             transls_.append(transl.squeeze())
+        #             poses_world_.append(theta_world.squeeze())
+        #             transls_world_.append(transl_world.squeeze())
+        #             shapes_.append(beta.squeeze())
+        #             cams_r_.append(R_worldtocam.tolist())
+        #             # Todo: note that T_worldtocam here is (1,3) whereas in h36m T_worldtocam is (1,3)
+        #             cams_t_.append(T_worldtocam.tolist())
+        #             cams_k_.append(K.tolist())
+        #             # in_bos_label_.append(in_bos_label)
+        #             # contact_label_.append(contact_label)
+        #             ground_offset_.append(ground_offset)
+
+
+        # store the data struct
+        if not os.path.isdir(out_path):
+            os.makedirs(out_path)
+        out_file = os.path.join(out_path, f'rich_{scene_name}_smplx.npz')
+        np.savez(out_file, imgname=imgnames_,
+                        #    center=centers_,
+                        #    scale=scales_,
+                        #    part=parts_,
+                        #    S=Ss_,
+                        #    S_world=Ss_world_,
+                        pose=poses_,
+                        transl=transls_,
+                        shape=shapes_,
+                        #    openpose=openposes_,
+                        #    pose_world=poses_world_,
+                        #    transl_world=transls_world_,
+                        #    cam_r=cams_r_,
+                        #    cam_t=cams_t_,
+                        cam_k=cams_k_,
+                        #    in_bos_label=in_bos_label_,
+                        contact_label=contact_label_,
+                        #    ground_offset=ground_offset_
+                            scene_seg=scene_seg_,
+                            part_seg=part_seg_
+                )
+        print('Saved to ', out_file)
+
+def rectify_pose(camera_r, body_aa):
+    body_r = batch_rodrigues(body_aa).reshape(-1,3,3)
+    final_r = camera_r @ body_r
+    body_aa = batch_rot2aa(final_r)
+    return body_aa
+
+
+def extract_cam_param_xml(xml_path: str = '', dtype=float):
+    import xml.etree.ElementTree as ET
+    tree = ET.parse(xml_path)
+
+    extrinsics_mat = [float(s) for s in tree.find('./CameraMatrix/data').text.split()]
+    intrinsics_mat = [float(s) for s in tree.find('./Intrinsics/data').text.split()]
+    distortion_vec = [float(s) for s in tree.find('./Distortion/data').text.split()]
+
+    focal_length_x = intrinsics_mat[0]
+    focal_length_y = intrinsics_mat[4]
+    center = np.array([[intrinsics_mat[2], intrinsics_mat[5]]], dtype=dtype)
+
+    rotation = np.array([[extrinsics_mat[0], extrinsics_mat[1], extrinsics_mat[2]],
+                         [extrinsics_mat[4], extrinsics_mat[5], extrinsics_mat[6]],
+                         [extrinsics_mat[8], extrinsics_mat[9], extrinsics_mat[10]]], dtype=dtype)
+
+    translation = np.array([[extrinsics_mat[3], extrinsics_mat[7], extrinsics_mat[11]]], dtype=dtype)
+
+    # t = -Rc --> c = -R^Tt
+    cam_center = [-extrinsics_mat[0] * extrinsics_mat[3] - extrinsics_mat[4] * extrinsics_mat[7] - extrinsics_mat[8] *
+                  extrinsics_mat[11],
+                  -extrinsics_mat[1] * extrinsics_mat[3] - extrinsics_mat[5] * extrinsics_mat[7] - extrinsics_mat[9] *
+                  extrinsics_mat[11],
+                  -extrinsics_mat[2] * extrinsics_mat[3] - extrinsics_mat[6] * extrinsics_mat[7] - extrinsics_mat[10] *
+                  extrinsics_mat[11]]
+
+    cam_center = np.array([cam_center], dtype=dtype)
+
+    k1 = np.array([distortion_vec[0]], dtype=dtype)
+    k2 = np.array([distortion_vec[1]], dtype=dtype)
+
+    return focal_length_x, focal_length_y, center, rotation, translation, cam_center, k1, k2
+
+# rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/work/achatterjee/rich/npzs', split='train')
+# rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/fast/achatterjee/rich/scene_npzs', split='val')
+# rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/fast/achatterjee/rich/scene_npzs/test', split='test')
+# rich_extract(dataset_path='/ps/scratch/ps_shared/stripathi/4yogi/RICH/val/', out_path='/home/achatterjee/rich_ext', split='val')
+
+if __name__=='__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--index', type=int)
+    args = parser.parse_args()
+
+    rich_extract(img_dataset_path='/is/cluster/work/achatterjee/rich', out_path='/is/cluster/fast/achatterjee/rich/scene_npzs/test', scene_indx=args.index, split='test')

data/preprocess/yoga-82_test/yoga82_simple_test.npz.py

@@ -0,0 +1,34 @@
+# load split.json and make an npz with all the folders in the test split
+import argparse
+import json
+import os
+import glob
+import numpy as np
+
+Yoga_82_PATH = '/is/cluster/work/stripathi/pycharm_remote/yogi/data/Yoga-82/yoga_dataset_images'
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default=Yoga_82_PATH)
+    parser.add_argument('--out_file', type=str, default='data/dataset_extras/yoga-82/yoga-82_simple_test_20each.npz')
+    args = parser.parse_args()
+
+    # structs we use
+    imgnames_ = []
+
+    # get all ffolder names in the data_dir
+    folders = glob.glob(os.path.join(args.data_dir, '*'))
+    folders.sort()
+    for folder in folders:
+        print(folder)
+        # get all images in the folder
+        images = glob.glob(os.path.join(folder, '*.jpg'), recursive=True)
+        print(len(images))
+        # only take random 50 images from each folder
+        images = np.random.choice(images, 20, replace=False)
+        imgnames_.extend(images)
+
+
+    np.savez(args.out_file, imgname=imgnames_,)
+    print('Saved to ', args.out_file)
+

fetch_data.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Script that fetches checkpoints and other necessary data for inference
+
+# Download utility files and other constants
+wget https://keeper.mpdl.mpg.de/f/50cf65320b824391854b/?dl=1 --max-redirect=2 --trust-server-names  && tar -xvf data.tar.gz && rm -r data.tar.gz
+
+# Downloading existing checkpoint
+mkdir checkpoints
+wget https://keeper.mpdl.mpg.de/f/6f2e2258558f46ceb269/?dl=1 --max-redirect=2 --trust-server-names  && tar -xvf Release_Checkpoint.tar.gz --directory checkpoints && rm -r Release_Checkpoint.tar.gz
+
+# Downloading other checkpoint
+wget https://keeper.mpdl.mpg.de/f/9cb970221b1e45d185b8/?dl=1 --max-redirect=2 --trust-server-names  && tar -xvf Other_Checkpoints.tar.gz --directory checkpoints && rm -r Other_Checkpoints.tar.gz
+
+# Downloading training datasets
+mkdir datasets
+wget https://keeper.mpdl.mpg.de/f/81c3ec9997dd440b8db3/?dl=1 --max-redirect=2 --trust-server-names  && tar -xvf Release_Datasets.tar.gz --directory datasets && rm -r Release_Datasets.tar.gz
+

hot_analysis/.ipynb_checkpoints/hico_analysis-checkpoint.ipynb

@@ -0,0 +1,307 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import pandas as pd \n",
+    "import ipdb\n",
+    "import os\n",
+    "import pickle as pkl\n",
+    "import os.path as osp\n",
+    "import numpy as np\n",
+    "from PIL import Image\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total images found: 9642\n",
+      "Images after KP filtering: 3895\n",
+      "['hake_train2015_HICO_train2015_00005476.jpg', 'hake_train2015_HICO_train2015_00008329.jpg', 'hake_train2015_HICO_train2015_00008027.jpg', 'hake_train2015_HICO_train2015_00013408.jpg', 'hake_train2015_HICO_train2015_00010656.jpg']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load Agniv VITpose-base hico filtering\n",
+    "filter_path = './agniv_pose_filter/hico.npy'\n",
+    "pose_md = np.load(filter_path, allow_pickle=True)\n",
+    "pose_md = pose_md.item()\n",
+    "filter_img_names = {}\n",
+    "\n",
+    "print(f'Total images found: {len(pose_md)}')\n",
+    "\n",
+    "# Filter out images with < 10 visible keypoints \n",
+    "kp_thresh = 10\n",
+    "\n",
+    "for imgname, pose_num in pose_md.items():\n",
+    "    if pose_num > kp_thresh:\n",
+    "        filter_img_names[imgname] = pose_num\n",
+    "        \n",
+    "print(f'Images after KP filtering: {len(filter_img_names)}')\n",
+    "\n",
+    "print(list(filter_img_names.keys())[:5])\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# # Load Agniv VITpose-base hot dict\n",
+    "# filter_path = \"./agniv_pose_filter/hot_dict.pkl\"\n",
+    "# with open(filter_path, 'rb') as f:\n",
+    "#     pose_md_dict = pkl.load(f)\n",
+    "    \n",
+    "# hico_dict = {}\n",
+    "\n",
+    "# for k, v in pose_md_dict.items():\n",
+    "#     if 'hake' in k:\n",
+    "#         hico_dict[k] = v\n",
+    "    \n",
+    "# print(f'Total images found: {len(hico_dict)}')\n",
+    "\n",
+    "# # Filter out images with < 10 visible keypoints \n",
+    "# kp_thresh = 10\n",
+    "\n",
+    "# filter_img_names = {}\n",
+    "\n",
+    "# for imgname, kp_md in hico_dict.items():\n",
+    "#     if kp_md == 0:\n",
+    "#         continue\n",
+    "#     if kp_md[\"num_persons\"] == 1 and kp_md[\"num_kpt\"][0.5][0] > kp_thresh:\n",
+    "#         filter_img_names[imgname] = kp_md[\"num_kpt\"][0.5][0]\n",
+    "        \n",
+    "# print(f'Images after KP filtering: {len(filter_img_names)}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "       0         1               2\n",
+      "0      1  airplane           board\n",
+      "1      2  airplane          direct\n",
+      "2      3  airplane            exit\n",
+      "3      4  airplane             fly\n",
+      "4      5  airplane         inspect\n",
+      "..   ...       ...             ...\n",
+      "595  596     zebra            feed\n",
+      "596  597     zebra            hold\n",
+      "597  598     zebra             pet\n",
+      "598  599     zebra           watch\n",
+      "599  600     zebra  no_interaction\n",
+      "\n",
+      "[600 rows x 3 columns]\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "hico_annot = json.load(open('/ps/project/datasets/HICO/hico-image-level/hico-training-set-image-level.json', 'rb'))\n",
+    "hoi_mapping = pd.read_csv('/ps/project/datasets/HICO/hico-image-level/hico_hoi_list.txt', header=None, delim_whitespace=True)\n",
+    "print(hoi_mapping)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Final number of images 3154\n"
+     ]
+    }
+   ],
+   "source": [
+    "version = '1'\n",
+    "out_dir = f'./filtered_data/v_{version}'\n",
+    "os.makedirs(out_dir, exist_ok=True)\n",
+    "\n",
+    "objectwise_img_names = {}\n",
+    "imgwise_object_names = {}\n",
+    "img_dir = '/ps/project/datasets/HICO/hico_20150920/images/train2015'\n",
+    "\n",
+    "bad_object_names = ['bear', 'bird', 'cat', 'cow', \n",
+    "                    'dog', 'elephant', 'giraffe', 'horse', \n",
+    "                   'mouse', 'person', 'sheep', 'zebra' ]\n",
+    "bad_action_names = ['buy', 'chase', 'direct', 'greet', 'herd', 'hose',\n",
+    "                   'hug', 'hunt', 'milk', 'no_interaction', 'pet', 'point', 'teach',\n",
+    "                   'watch', 'wave']\n",
+    "\n",
+    "for i, (img_name, img_md) in enumerate(hico_annot.items()):\n",
+    "    \n",
+    "    # Apply keypoint number filtering on the images \n",
+    "    full_img_name = 'hake_train2015_' + img_name\n",
+    "    if full_img_name not in filter_img_names.keys():\n",
+    "        continue\n",
+    "    \n",
+    "    # show the image\n",
+    "    if i < 0:\n",
+    "        img = Image.open(osp.join(img_dir,img_name))\n",
+    "        display(img)\n",
+    "    \n",
+    "    obj_names = []\n",
+    "    action_names = []\n",
+    "    kp_num = filter_img_names[full_img_name]\n",
+    "    \n",
+    "    # travel through all hoi in the metadata, save obj_names and action_names for the hois\n",
+    "    for hoi_id in img_md['hoi_id']:\n",
+    "        img_md_row = hoi_mapping.loc[hoi_mapping[0] == hoi_id].iloc[0]\n",
+    "\n",
+    "        obj_name = img_md_row[1]\n",
+    "        if obj_name in bad_object_names:\n",
+    "            continue\n",
+    "        action_name = img_md_row[2]\n",
+    "        if action_name in bad_action_names:\n",
+    "            continue\n",
+    "            \n",
+    "        obj_names.append(obj_name)\n",
+    "        action_names.append(action_name)\n",
+    "        \n",
+    "    if len(set(obj_names)) == 0 or len(action_names) == 0:\n",
+    "        continue\n",
+    "    \n",
+    "    imgwise_object_names.setdefault(full_img_name,[]).extend(list(set(obj_names)))\n",
+    "    \n",
+    "# #     # Display images with multiple objects\n",
+    "#     if len(set(obj_names)) > 1:\n",
+    "#         print(img_name)\n",
+    "#         print(obj_names)\n",
+    "#         print(action_names)\n",
+    "#         print(f'Number of Kps: {kp_num}')\n",
+    "#         img = Image.open(osp.join(img_dir,img_name))\n",
+    "#         display(img)\n",
+    "            \n",
+    "    for obj_name in set(obj_names):\n",
+    "        objectwise_img_names.setdefault(obj_name,[]).append(full_img_name)\n",
+    "        \n",
+    "print(f'Final number of images {len(imgwise_object_names)}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "# Save the imagewise_object dict\n",
+    "out_path = osp.join(out_dir, 'object_per_image_dict.json')\n",
+    "with open(out_path, 'w') as fp:\n",
+    "    json.dump(imgwise_object_names, fp)\n",
+    "    print(f'saved at {out_path}')\n",
+    "    \n",
+    "# # save image_list \n",
+    "# out_path = osp.join(out_dir, 'hico_imglist_all_140223.txt')\n",
+    "# with open(out_path, 'w') as f:\n",
+    "#     f.write('\\n'.join(imgwise_object_names.keys()))\n",
+    "#     print(f'saved at {out_path}')\n",
+    "\n",
+    "\n",
+    "# Save the object_wise dict\n",
+    "out_path = osp.join(out_dir, 'imgnames_per_object_dict.json')\n",
+    "with open(out_path, 'w') as fp:\n",
+    "    json.dump(objectwise_img_names, fp)\n",
+    "    print(f'saved at {out_path}')\n",
+    "\n",
+    "    \n",
+    "\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names = dict(sorted(objectwise_img_names.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# Extract object names and image counts\n",
+    "obj_names = list(objectwise_img_names.keys())\n",
+    "img_counts = [len(objectwise_img_names[obj]) for obj in objectwise_img_names]\n",
+    "print(sum(img_counts))\n",
+    "\n",
+    "# Create bar plot\n",
+    "sns.barplot(x=obj_names, y=img_counts)\n",
+    "\n",
+    "# Add x-axis and y-axis labels\n",
+    "plt.xlabel('Object')\n",
+    "plt.ylabel('Number of Images')\n",
+    "\n",
+    "plt.xticks(rotation=45, ha='right', fontsize=3)\n",
+    "\n",
+    "# Save the plot as a high-resolution image file\n",
+    "out_path = osp.join(out_dir, 'image_per_object_category.png')\n",
+    "plt.savefig(out_path, dpi=300)\n",
+    "\n",
+    "# Show plot\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

hot_analysis/.ipynb_checkpoints/vcoco_analysis-checkpoint.ipynb

@@ -0,0 +1,276 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import pandas as pd \n",
+    "import ipdb\n",
+    "import os\n",
+    "import pickle as pkl\n",
+    "import os.path as osp\n",
+    "import numpy as np\n",
+    "from PIL import Image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total images found: 5459\n",
+      "Images after KP filtering: 2386\n",
+      "['vcoco_000000051899.jpg', 'vcoco_000000093127.jpg', 'vcoco_000000455665.jpg', 'vcoco_000000248389.jpg', 'vcoco_000000403947.jpg']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load Agniv VITpose-base hot dict\n",
+    "filter_path = \"./agniv_pose_filter/hot_dict.pkl\"\n",
+    "with open(filter_path, 'rb') as f:\n",
+    "    pose_md_dict = pkl.load(f)\n",
+    "    \n",
+    "vcoco_dict = {}\n",
+    "\n",
+    "for k, v in pose_md_dict.items():\n",
+    "    if 'vcoco' in k:\n",
+    "        vcoco_dict[k] = v\n",
+    "    \n",
+    "print(f'Total images found: {len(vcoco_dict)}')\n",
+    "\n",
+    "# Filter out images with < 10 visible keypoints \n",
+    "kp_thresh = 10\n",
+    "\n",
+    "filter_img_names = {}\n",
+    "\n",
+    "for imgname, kp_md in vcoco_dict.items():\n",
+    "    if kp_md == 0:\n",
+    "        continue\n",
+    "    if kp_md[\"num_persons\"] == 1 and kp_md[\"num_kpt\"][0.5][0] > kp_thresh:\n",
+    "        filter_img_names[imgname] = kp_md[\"num_kpt\"][0.5][0]\n",
+    "        \n",
+    "print(f'Images after KP filtering: {len(filter_img_names)}')\n",
+    "\n",
+    "print(list(filter_img_names.keys())[:5])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "loading annotations into memory...\n",
+      "Done (t=1.30s)\n",
+      "creating index...\n",
+      "index created!\n",
+      "Final number of images 2368\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pycocotools.coco import COCO\n",
+    "version = '1'\n",
+    "out_dir = f'./filtered_data/v_{version}/vcoco'\n",
+    "os.makedirs(out_dir, exist_ok=True)\n",
+    "\n",
+    "coco_dir = '/is/cluster/work/stripathi/pycharm_remote/v-coco/data'\n",
+    "coco = COCO(os.path.join(coco_dir, 'instances_vcoco_all_2014.json'))\n",
+    "\n",
+    "imgwise_object_names = {}\n",
+    "img_dir = '/ps/project/datasets/HOT/Contact_Data/images/training/'\n",
+    "\n",
+    "bad_object_names = ['bird', 'cat', 'dog', 'horse', 'sheep', 'cow', \n",
+    "                    'elephant', 'bear', 'zebra', 'giraffe']\n",
+    "\n",
+    "# # Get all coco objects\n",
+    "# count = 0\n",
+    "# for id in range(91):\n",
+    "#     try: \n",
+    "#         print(coco.loadCats(id))\n",
+    "#         count += 1\n",
+    "#     except:\n",
+    "#         print(f'ID:{id} does not exist')\n",
+    "#         continue\n",
+    "# print(count)\n",
+    "\n",
+    "objectwise_img_names = {}\n",
+    "for img_name in filter_img_names.keys():\n",
+    "    img_id = int(os.path.splitext(img_name.split('_')[-1])[0])\n",
+    "    ann_ids = coco.getAnnIds([img_id])\n",
+    "    anns = coco.loadAnns(ann_ids)\n",
+    "    object_list = []\n",
+    "    for ann in anns:\n",
+    "        object_cat = coco.loadCats(ann[\"category_id\"])\n",
+    "        if len(object_cat) > 1:\n",
+    "            ipdb.set_trace()\n",
+    "        object_name = object_cat[0]['name']\n",
+    "        if object_name in bad_object_names:\n",
+    "            continue\n",
+    "        if object_name != 'person':\n",
+    "            object_list.append(object_name)\n",
+    "    if len(object_list) != 0:\n",
+    "        imgwise_object_names[img_name] = list(set(object_list))\n",
+    "        \n",
+    "    for obj_name in set(object_list):\n",
+    "        objectwise_img_names.setdefault(obj_name,[]).append(img_name)\n",
+    "        \n",
+    "#         # Visualize images \n",
+    "#         print(img_name)\n",
+    "#         print(list(set(object_list)))\n",
+    "#         img = Image.open(osp.join(img_dir,img_name))\n",
+    "#         display(img)\n",
+    "    \n",
+    "    \n",
+    "print(f'Final number of images {len(imgwise_object_names)}')\n",
+    "    \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "saved at ./filtered_data/v_1/vcoco/object_per_image_dict.json\n",
+      "saved at ./filtered_data/v_1/vcoco/vcoco_imglist_all_170223.txt\n",
+      "saved at ./filtered_data/v_1/vcoco/imgnames_per_object_dict.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Save the imagewise_object dict\n",
+    "out_path = osp.join(out_dir, 'object_per_image_dict.json')\n",
+    "with open(out_path, 'w') as fp:\n",
+    "    json.dump(imgwise_object_names, fp)\n",
+    "    print(f'saved at {out_path}')\n",
+    "    \n",
+    "# save image_list \n",
+    "out_path = osp.join(out_dir, 'vcoco_imglist_all_170223.txt')\n",
+    "with open(out_path, 'w') as f:\n",
+    "    f.write('\\n'.join(imgwise_object_names.keys()))\n",
+    "    print(f'saved at {out_path}')\n",
+    "\n",
+    "\n",
+    "# Save the object_wise dict\n",
+    "out_path = osp.join(out_dir, 'imgnames_per_object_dict.json')\n",
+    "with open(out_path, 'w') as fp:\n",
+    "    json.dump(objectwise_img_names, fp)\n",
+    "    print(f'saved at {out_path}')\n",
+    "\n",
+    "    \n",
+    "\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "5072\n"
+     ]
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEPCAYAAACtCNj2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsQklEQVR4nO3debhcVZnv8e+bBDJDRpKQBBI0TGGUEIEgIAgEAhkIoyC0oqgNtt22fRW19enbzdW+rfa1xQFUFAdATBhCCGMUFEFCGEOYEgRJEEhAwhAgQHjvH+9blV1FnXMqyalT5yS/z/Oc59TetYe19157vWutPZS5OyIiIgDdmp0AERHpPBQURESkTEFBRETKFBRERKRMQUFERMoUFEREpKxHsxOwMYYMGeJjxoxpdjJERLqUu++++3l3H1rruy4dFMaMGcPChQubnQwRkS7FzP7S0nfqPhIRkTIFBRERKVNQEBGRMgUFEREpU1AQEZEyBQURESlTUBARkTIFBRERKevSD68BrPzBLyuGh376tCalRESk61NLQUREyhQURESkTEFBRETKFBRERKRMQUFERMoUFEREpExBQUREyhQURESkTEFBRETKFBRERKRMQUFERMoUFEREpExBQUREyhQURESkTEFBRETKFBRERKRMQUFERMoUFEREpExBQUREyhoeFMysu5nda2Zzc3iQmd1kZkvy/8DCtOea2VIze9TMjmx02kREpFJHtBQ+CzxcGP4iMN/dxwHzcxgz2xU4GRgPTAa+b2bdOyB9IiKSGhoUzGwUMAX4cWH0NODi/HwxML0w/jJ3X+PuTwBLgYmNTJ+IiFRqdEvh/wH/C3inMG6Yuz8DkP+3yfEjgWWF6ZbnOBER6SANCwpmdgywwt3vrneWGuO8xnLPMrOFZrZw5cqVG5VGERGp1MiWwiRgqpk9CVwGHGpmvwSeM7MRAPl/RU6/HBhdmH8U8Nfqhbr7he4+wd0nDB06tIHJFxHZ/DQsKLj7ue4+yt3HEBeQf+vupwFzgDNysjOAq/PzHOBkM+tpZmOBccCCRqVPRETerUcT1vkN4HIzOxN4CjgBwN0Xm9nlwEPA28DZ7r62CekTEdlsdUhQcPdbgFvy8wvAYS1Mdx5wXkekSURE3k1PNIuISJmCgoiIlCkoiIhImYKCiIiUKSiIiEiZgoKIiJQpKIiISJmCgoiIlCkoiIhImYKCiIiUKSiIiEiZgoKIiJQpKIiISJmCgoiIlCkoiIhImYKCiIiUKSiIiEiZgoKIiJQpKIiISJmCgoiIlCkoiIhImYKCiIiUKSiIiEiZgoKIiJQpKIiISFmbQcHMTjCz/vn5K2Z2hZm9r/FJExGRjlZPS+Ff3f0VMzsQOBK4GPhBY5MlIiLNUE9QWJv/pwA/cPergS0blyQREWmWeoLC02Z2AXAiMM/MetY5n4iIdDH1FO4nAjcAk919FTAI+JdGJkpERJqjzaDg7q8BK4ADc9TbwJJGJkpERJqjnruPvgZ8ATg3R20B/LKRiRIRkeaop/toBjAVWA3g7n8F+jcyUSIi0hz1BIU33d0BBzCzvo1NkoiINEs9QeHyvPtogJl9ArgZ+FFjkyUiIs3Qo60J3P2bZnY48DKwE/BVd7+p4SkTEZEO12ZQAMggoEAgIrKJq+fuo1fM7OWqv2VmdqWZ7dDKfL3MbIGZ3W9mi83s33L8IDO7ycyW5P+BhXnONbOlZvaomR3ZPpsoIiL1qqel8G3gr8AlgAEnA8OBR4GLgENamG8NcKi7v2pmWwC3mdl1wHHAfHf/hpl9Efgi8AUz2zWXPR7YFrjZzHZ097UtLF9ERNpZPReaJ7v7Be7+iru/7O4XAke7+6+BgS3N5OHVHNwi/xyYRrxUj/w/PT9PAy5z9zXu/gSwFJi43lskIiIbrJ6g8I6ZnWhm3fLvxMJ33tqMZtbdzO4jnoi+yd3vBIa5+zMA+X+bnHwksKww+/IcJyIiHaSeoHAq8BGiYH8uP59mZr2Bc1qb0d3XuvtewChgopnt1srkVmsR75rI7CwzW2hmC1euXFlH8kVEpF713JL6Z+DYFr6+rZ6VuPsqM7sFmAw8Z2Yj3P0ZMxtBBBuIlsHowmyjiGsZ1cu6ELgQYMKECa22VEREZP3Uc/dRLzM728y+b2YXlf7qmG+omQ3Iz72BDwGPAHOAM3KyM4Cr8/Mc4GQz62lmY4FxwIL13iIREdlg9XQf/YK42+hI4FaiBv9KHfONAH5nZg8AdxHXFOYC3wAON7MlwOE5jLsvBi4HHgKuB87WnUciIh2rnltS3+vuJ5jZNHe/2MwuIX5foVXu/gCwd43xLwCHtTDPecB5daRJREQaoJ6Wwlv5f1VeKN4aGNOwFImISNPU01K4MJ86/lei378f8NWGpmojrfzhTyqGh37qzCalRESka6nn7qMf58dbgRZfayEiIl1fm0Eh7yA6negyKk/v7v/QsFSJiEhT1NN9NA/4E7AIeKexyWmclT88v2J46Kdafe5ORGSzVE9Q6OXun2t4SkREpOnqek7BzD5hZiPytdeDzGxQw1MmIiIdrp6WwpvAfwFfZt27iBxddBYR2eTUExQ+RzzA9nyjEyMiIs1VT/fRYuC1RidERESar56WwlrgPjP7HfFraoBuSRUR2RTVExSuyj8REdnE1fNE88VtTSMiIpuGFoOCmS2ilZ/bdPc9GpIiERFpmtZaCsd0WCpERKRTaDEouPtfOjIhIiLSfPXckioiIpsJBQURESlrMSiY2fz8/58dlxwREWmm1i40jzCzg4GpZnYZYMUv3f2ehqZMREQ6XGtB4avAF4FRwLervnPg0EYlSkREmqO1u49mAbPM7F/d/d87ME0iItIk9TzR/O9mNhU4KEfd4u5zG5ssERFphjbvPjKzrwOfBR7Kv8/mOBER2cTU80K8KcBe7v4OgJldDNwLnNvIhHWE535QeWPVsE9/oUkpERHpHOp9TmFA4fPWDUiHiIh0AvW0FL4O3Ju/p2DEtYUu30oQEZF3q+dC86VmdguwLxEUvuDuzzY6YSIi0vHqaSng7s8AcxqcFhERaTK9+0hERMrqailsTp75/pfLn0f8/XlNTImISMdrtaVgZt3M7MGOSoyIiDRXq0Ehn02438y266D0iIhIE9XTfTQCWGxmC4DVpZHuPrVhqRIRkaaoJyj8W8NTISIinUI9zyncambbA+Pc/WYz6wN0b3zSRESko9XzQrxPALOAC3LUSOCqBqZJRESapJ7uo7OBicCdAO6+xMy2aWsmMxsN/BwYDrwDXOju3zGzQcCvgTHAk8CJ7v5iznMucCawFvgHd79hfTeovS0//5MVw6POuaCFKUVEur56Hl5b4+5vlgbMrAfxy2tteRv4Z3ffBdgPONvMdiV+zW2+u48D5ucw+d3JwHhgMvB9M1M3lYhIB6onKNxqZl8CepvZ4cBvgGvamsndnyn9jrO7vwI8THQ9TQMuzskuBqbn52nAZe6+xt2fAJYSLRQREekg9QSFLwIrgUXAJ4F5wFfWZyVmNgbYm+iCGpbvUiq9U6nUFTUSWFaYbXmOExGRDlLP3Ufv5A/r3El0Gz3q7vV0HwFgZv2A2cA/uvvLZtbipLVWX2N5ZwFnAWy3nZ6pExFpT/XcfTQFeBz4H+B8YKmZHVXPws1sCyIg/Mrdr8jRz5nZiPx+BLAixy8HRhdmHwX8tXqZ7n6hu09w9wlDhw6tJxkiIlKnerqPvgV80N0PcfeDgQ8C/93WTBZNgp8AD7v7twtfzQHOyM9nAFcXxp9sZj3NbCwwDlhQ32aIiEh7qOeW1BXuvrQw/GfW1e5bMwn4CLDIzO7LcV8CvgFcbmZnAk8BJwC4+2Izuxx4iLhz6Wx3X1vXVoiISLtoMSiY2XH5cbGZzQMuJ/r4TwDuamvB7n4bta8TABzWwjznAXpftYhIk7TWUji28Pk54OD8vBIY2LAUiYhI07QYFNz9ox2ZEBERab42rynkRd/PEK+lKE+/Ob86+4nvTq8YHvuZq5qSDhGR9lbPheariLuIriHeYSQiIpuoeoLCG+7+Pw1PiYiINF09QeE7ZvY14EZgTWlk6b1GIiKy6agnKOxOPG9wKOu6jzyHRURkE1JPUJgB7FB8fbaIiGya6gkK9wMDqO8p5s3WQ9+rvBlr17PncO8Pj60Yt/enruFPFx5TMW6/s+Y2PG0iIvWqJygMAx4xs7uovKaw2d6SKiKyqaonKHyt4akQEZFOoZ7fU7i1IxIiIiLNV88Tza+w7sdutgS2AFa7+1aNTJiIiHS8eloK/YvDZjYd/XZyu/r9j6ZUDB/0iWublBIR2dzV8yM7Fdz9KvSMgojIJqme7qPjCoPdgAnU+O1kaV83/fjoiuHDPz6vSSkRkc1JPXcfFW+2fxt4EpjWkNSIiEhT1XNNQb+rICKymWjt5zi/2sp87u7/3oD0iIhIE7XWUlhdY1xf4ExgMKCgICKyiWnt5zi/VfpsZv2BzwIfBS4DvtXSfCIi0nW1ek3BzAYBnwNOBS4G3ufuL3ZEwkREpOO1dk3hv4DjgAuB3d391Q5LlYiINEVrLYV/Jt6K+hXgy2ZWGm/EhWa95qKDzftJ5bMLR5+pZxdEpH21dk1hvZ92FhGRrk0Fv4iIlCkoiIhIWT2vuZBO7KqLjqoYnv6x65qUEhHZFKilICIiZQoKIiJSpu6jTdDlP51cMXziR69vUkpEpKtRS0FERMrUUthM/PJnR1YMn/Z3NzQpJSLSmamlICIiZQoKIiJSpu6jzdhFFx9RMfyxM27kgl+s62b65EfUxSSyuVFLQUREyhoWFMzsIjNbYWYPFsYNMrObzGxJ/h9Y+O5cM1tqZo+a2ZG1lyoiIo3UyO6jnwHnAz8vjPsiMN/dv2FmX8zhL5jZrsDJwHhgW+BmM9vR3dc2MH1Sh+/+qjI+f+ZUdSmJbMoa1lJw998Df6saPY34BTfy//TC+MvcfY27PwEsBSY2Km0iIlJbR19TGObuzwDk/21y/EhgWWG65TlOREQ6UGe50Gw1xnnNCc3OMrOFZrZw5cqVDU6WiMjmpaODwnNmNgIg/6/I8cuB0YXpRgF/rbUAd7/Q3Se4+4ShQ4c2NLEiIpubjg4Kc4Az8vMZwNWF8SebWU8zGwuMAxZ0cNpERDZ7Dbv7yMwuBQ4BhpjZcuBrwDeAy83sTOAp4AQAd19sZpcDDwFvA2frziMRkY7XsKDg7qe08NVhLUx/HnBeo9IjIiJt02suZL1989LKZxc+f4qeXRDZVCgoSLv4j19XBoqvnKRAIdIVKShIw3zpN5W/APd/TtAvwIl0dp3lOQUREekEFBRERKRMQUFERMp0TUE61GeuqLzO8N3jdJ1BpDNRS0FERMoUFEREpEzdR9J0p1xV2aV06XR1KYk0i4KCdEpHXV35lpTrpl3apJSIbF7UfSQiImUKCiIiUqbuI+kyjrrqnIrh66af36SUiGy6FBSkSzv6qnMrhudN/3qTUiKyaVD3kYiIlKmlIJuco6/83xXD82Z8tUkpEel6FBRks3D0lf9ZMTxvxhealBKRzk1BQTZbU67474rha4/7pyalRKTzUFAQKZhyxffKn6897mymzL6g4vtrZ36yo5Mk0qEUFETW0zGzf1IxPHfmmU1KiUj7U1AQaQfHzL64YnjuzDOalBKRjaOgINIgx8z6VcXw3ONPbVJKROqnoCDSgY6Z9euK4bnHn9SklIjUpqAg0mTHzppdMXzN8TOblBIRBQWRTmnqrKsrhuccP42ps66tGjelI5MkmwkFBZEubNqsGyuGrz7+iCalRDYVeveRiIiUqaUgsomZMft3FcNXzvwgx83+Y8W4K2ZOYubsuyrGzZ65b8PTJp2fgoKIlJ0w+4Hy59/M3IOTZj9W8f2vZ+7Y0UmSDqagICLr5RNXPFUx/KPjtuMrVz5dMe4/ZozsyCRJO1JQEJGG+NaVz1YM//OM4U1KiawPBQUR6TAXXrGiYvis47ZpUkqkJQoKItJUl8xeWTH84ZlDuXLW8xXjZhw/hOt+XTnuqJOG8NtfVc576KlDG5PIzYiCgohsUv7488pAMen0oSy8qLKFMuFjaqG0REFBRDZLiy6oDBS7f1KBAhQURETKlpz/XPnzuHOG8dS3Ky+Wb/e5Tf9ieacLCmY2GfgO0B34sbt/o8lJEhEpe+b/Lq8YHvG/RvHsN5dWjBv++ffy7LcXV4773Hie++97K8YN+6e9G5PIjdCpgoKZdQe+BxwOLAfuMrM57v5Qc1MmItIYz33nzorhYZ99f5NSEjpVUAAmAkvd/c8AZnYZMA1QUBCRzcZz//OHiuFh//ABVnx3fsW4bT5zGCvOv75y3DmTWfG9OZXjzp66XuvubEFhJLCsMLwcaG7YFBHp4lZ8f1bd05q7NzAp68fMTgCOdPeP5/BHgInu/pnCNGcBZ+XgTsCj+XkIUHkj84aPa89ldcVxnSUdzRrXWdLRrHGdJR2daVxnSUd7jdve3Ws/1OHuneYP2B+4oTB8LnBunfMubK9x7bmsrjius6RD26/t7yzjOks6GjGu+q+z/Z7CXcA4MxtrZlsCJwNz2phHRETaSae6puDub5vZOcANxC2pF7n74jZmExGRdtKpggKAu88D5m3ArBe247j2XFZXHNdZ0tGscZ0lHc0a11nS0ZnGdZZ0NGJchU51oVlERJqrs11TEJHNjJlZs9Mg63TJoGBm/c1sWGG4ZqYysy3NbJSZtbmdZjbMzAYUhseZ2VYtrLt7G8vqVjVsrQ2vLzPrs77La4d1trSP616umfWsGt7bzIZsTLrWV97E0D0/N6Uwqne9rU3X3sfczHqbWUO6k82sm5ltYWZbVI3f1swGu7u3kr96mdlgM+vfyvLfNe96bvuW1WnrCGbWz8z61RjfavlSx3I3Kl93yaAAbAnsUSrEPfvAiid8GgTsC4wojTCzgdWBwsx6AzOAozMTDgUmABPNbOucZiczGwfMzHVbjh+VhdtYS8ChVYXdljlt96r07lw8gGY21Mz2qlF47mVmvQqj3mdme5QGCsvrX5inv5mNMbM9i9PUmK5vpn3b2ru6rFSQDsv/lifSITXS27d65pxvqpltn8MDgAOAw4rTt3CSDK0aHlBaTkty+wfWCOwjgRlm1sNb6TttYRsmmtmI/Fw8bluU/pvZIWb2ntLxqjWOdfuyVmHWszC+NN2+Vfm1HzCp6lhsXaMwGWZmB1pUcIrprT5e2wInAoNzeOvSeVI1XWmbisvaoVaBXbWO/sBkYF/LCk0elxOBz5rZkKr82TP/9yfO35nAB/I8fZdC/j+0sA+6m9lWZjawsNwBeU58oGoRuwFH5vldmnaQmY2vtb6q7exXtT/65j4fUTpmZjY691MxLSOBTwFHWJYxOX4IsEdheFT1ccjxPQqft7IIsLtW7Y/xtc6ntnSZoFB1Au0E7APsmSfEMDPbnyhkiid6L2AMkaG650l0OpFBS4X9cOJhuNXAk+7+BvA28BrQGygFkLeBHXKZawqZ+DDiIPYFSgVNL+DDZjYop9nXzKYBMwsZ/v3AacCOOdwr0z8ZKL/D18xOyW14s7BdfYnCeOfMhAeZ2enAtEJh9hbxNPgky8CQyxsAfNLMPpij3sl1zjCzvXKa0Wb2fjObbGZ9stA72Mw+AZxpZsM9vJXb+v4szAaY2SQiuO5bWOdWwH7AcKBbZvLDgRXAI+6+2qKm+vHcR4MK8+4PHJfLKBVgRwIn54lV3k9mtnshn/QBjsu0jCltFzAW2BnYq1QA50m8W2FZw4ETzezgHB5qZqcSr1zZ1syscOIdDhxuZt0L++N0oHSCvkXko/I44AAzOym3q1S52NbMjgE+TwT9AZmGScDf5XZ0y+0/hngIqVRodSMKzsPNbDDrTM7xu5I3lZjZEbnvSoGsL/CBnKZUuPcknhmaamY75XTDgI+a2bDCtk8GJgHTrTJoHQIcX8iLB2eaxwGDcr4PA68TbzA4Nretb+7ngy1uSR9HnOergbfc/fVc/lQz+3RhO8l0vg/YO+edAvwjUdj3ynEjgGFEpW5IztcLOATYncJ5B4wGdqrKxzOLgcLMtgE+C+xT2n53X53zfgjY0SIIHgjsCYw0sx657gOBVcD97v5SLm8Qcax3sahM9GZdwOqVZdgWZnZUHpti6+ag3G/jc1kfJs659S7jO93dRy0pZMSJRMF/K/BwNj2nAFsBzwBbAGssno5+C7gJeBYonUz9gGfd/cU8CLsCLxAPdTySGfkAoqC/3t0Xm9lxufzewB+AR8zsQzkf+f9Rd38rM/Uw4CVgbX6/FVFDfZGowZyS2/BT4FmL2ueOOf1d7r4st/VY4qS+BOhjZqtZV4t4Obe3P1FAvA1Y/gcYBQwAHgOW5PKGA8cDTxKtoFWZpkU5z5L8/zaRsYcSJ+1yoPRO4WW5bsxsKlF4PEY+JUkUxg8CT2em3ZoodN4Bnsr1zcjt+KG7P5rH93Uzu5nIyG/k8o8kCvHFeUy7AR8kguKt7v50TlcKIqOAJ7Og3YM4yR8DnreoMe2X6Zvn7vdY1Nw+ncfnt7n8/kSgLx1Xcl9sCTycy+th0VLrx7qn6s3MdicC3QLgiUzb7kT+WwA8kYWEE62AVbncNTnNC7nt77j7qtyOkcD5wF+Jwnrv/Puxu5eO13uIgnM18GZux2653IeBP2TePCWPx+NEcB5K5IengBvdfWkG3wNzH68BXsr9NDHT2h94LgumHYCrgBfc/Z3c3iOIc+1xYGczex+wkjhfryPy1vZEXn3E3W/JQDYSeG+mrxeRpyfmsbmulE8y4LxAPM/Uz91fNbOdieB0C5H31gKPEPn/KXd/IysEuxDn5Y3u/nwG2PGZvuvc/flcxylE3r8L+HPuzx1zPw82syWZzsOIc2FtYfv3Iypw3Yieij0zj8zPfDKaCF7DiXNhTc63F5E3lxHnY0/gzMwjz+Q29Mp5hxD5rDtRxu1DBNA7iPNkpzxOf8nv10unDwpVtbIZwNHAzcBsYAczK2WkN4iDvSaj5SQig1xHHJxS03AB0M+iqbUbMBC4HViZhf82xEG9zt0X5oHYncg4i9z9wUzLO0Rt4E9EQHnLzCYQ+3QhUZC9kge7N1EregGYSmT6N4Fl7v5mnpxHAPcAj5tZT3dfk2kdBPwul3EmceJcQpwAfXJ4DHESzs/tP43IlH/L9PXIgvO9xAn7aO6/gUSBfR8R1N7MAmBSTnc38TLCCUQme4Io/N/I7epJFFbXE7XSnYkM/SRx0u9OZP6dgMuIwuAYYClxspZeUQKAuz9pZsvcfW0WJu8lMv6jRIG/d27TwkIhcXSudwlxUvQCPkKc/H8kTuw+wAm5T/7k7vfkKlflfpgE3E+cbAfldIvc/UGLlsqemYYFWRMstfSeB57OfbwLUZguJvJnqevuAOA2Im++ZdH62J0oiG/L47U1EVyXAddm5eSwPKbbEvnmwPzcJ5dfemnkiUSB9QpRAeoFfJQIptcAV2XAHUO0fp8nAtuAPC7PA0vc/bEs/I7O7f0Ncf7slPv9LeD2DBx75L5aBfwtW9dk/h+e6Vzq7neb2T65jAsyT0BU3Ba7+y2Fw38cEWTuJgrK3YhzeiURGErHeiSR97+f23V0rm9rIt/1IPLiCuBmd3868/6k/JsPvJVlxHtKx7YQEN5HnHPP5rHsCXySOFeeJG6XH0CcE0Nyv6/KecflsVhFlDu75LY8CNxLnMOjc/sXZFpfzJbDh3L5pYrBQUSQv5Z1lZODiHJiLpGP38hKx3uIAHBPHr/XiHP6gVLran10+qBQCAgfICLrfCLzdyO6IPoRB+/erP2XmsKzc3xvoqm6jKgxPZg1iwOIKP8nd1+U67gNOJU4aRZlJu9LHJyn3f0POd0eOe4Bcsdn4XFIruc+IjOdQ5ycc4lMMhj4GFG4XuXub+Y2PmlmX3f3V6u2/Tkz+6W7r831Xkk85f0UEVT2JDLYIqJQW52ZfUV+f3fWpA7LdQ8jHghcatFVMZA4ya7OgDCI6JoZkst8DPgMcVI/TAS7nkQTt1ToXOvur5nZfKLAmpv74z3ESVia7y+5f/Yhuunm5zaVg35u89osTHcjCoR5mfYDiJPs94WAsDdRkLxAFACl7oXZRNfZPUTtdiJRe7yvEBBKeWuZmf0mj/O43NYbskIwk8hfTxGBqBQQjiFqiRcRFYqTiJbQm0QQ9Vz/c0SBcH8GhKOIc24JcEd2mw3O4zieqCQ8kgX9/sRDnHOIgDwgl/d0FraHmdkbRD6clfOW0ndVpum+PBYH5fAi4MeZtmFE0J3r7o/l/njHzO7I4/94prXU7Tfb3R/KY/PhPPb3Ffb5JCJv3k4UeCMsul5+lPuwP7AXUfD9uBRIcr0v5DGbQRSC3YjKyj25fc/m+XU0cDnwhMeDrvsQlbhXcvyWud/GEq2Xpy26vT5GtKovz2UbEUjuJ1qNb+c2nEoEyR8CVhg/j6hU3JDH9hQizy0gzocVmb6euY+X5fR9icrEn4iguiNxjqx09zty2YPzOKzMeUuV0N2JwPm8RRfuzEzbI0SZ80a2Snrk9lxBVLp2JQLrfVmxXG+d+jmF3GE7EgdiEnFyXlOombyXqBVcSNQW9yOidG+iEN4/h48gCqIfWfTVnkwcwBuJgq8vcaCvJ2o+r2TTfThx8t9Zaqpnl8ZU4JdE99WqzBBjidrp7939lZx2O6IveS5RsG1J1GhucvdS9K/e5opCssb3PYjC/GSiQLwO+D1x0n2YKNh+QRRiay26w04iuoBecvc7Lfp0pxKB844s1IcQ/bB9iBrx3bm+UTntpe7+YmG7TgJ+4u5/q0rbyFxX6frII0TtfkUei12IFs3qFrbvcLIbiAj0q7Lp/3GihnwHcWLvQBSkLxNdbquqlrNDLmcH4qS5290faGGd2xK16/uILo3HM5DuQ5zQdxVrXBZ9yUcTwfTFrIXvQQSTUnfATkS++ylRq3w198mLRC28O+u6qhYT+fxNIsh3y314P1EoTsttXeLul+fyjyWCyT1EfqroJrC4kF4q1IYTeeOW7DY7kMir/d399hr7owfRfXIg0d3zOhHY9yJqvmNz218qzDMYOIrIj3sSQWw1UavfgmghvgCscPf7a6yzd+6T3XO7uxfOo9Ny3/QlgtPLFt1Uh+T2X5/78kiiBfU6cQ3ivszXHycCx68yLTOI8/GGUiUh9+cZRIv2ilJ3UNU+GU0U4LvlPi/1GuxLBLGlucw1eY7sT1Rq+mVa7wP+XMgjI4iL7YuJQLIqp1uT05Xzq5mNzX3zOyJoH0yUed8izq8PZrpW5bF5sXof16tTBoVSwWjRfziR2Kl3EjX9l6um3SJrYbsThRrAD9z9pSxMTiOuA2xDRO3HiKbqWnefZXGNolSLft7d52YGP4c4oZcVWisfJCL+1kTXxPas62t8mKitVhR2WVM5mThB7iCa1RsUwauWux2RoS4qFcw57mSiJva3HB5P1H7vcvc/5nQjiJP0CuJkGkYUoKOBx939mqp1lQuYNsZtTwTvpcSJPZFoig8jTuhriQy/vNT6qZr/RKIr7CGiAHu18F0pfc8Qmd+IAvYed3+txrL6EwX3NkRN/fet7MsRxEXJ+4g+8plERWQJWSurMU/34jZUD+e4Ut6cAhwKXE10I6wFjiUqJsOIgmNVbt9kooC4hMhnfYmujP2Igra0D0cQheBV9RQA2TV0TG7XQHf/eeG7ioqIRRfioUQF4U6PrqWJRIG4BHioRj7vS3R77E30e0Psy1Jg2Z5old9VI23b5TxriBbXktzm14lgOyHH3ZMt39L1jAeJfPYW0f3Sh6g43g+8WWqJW1zH6ZPL35fY/w8WK2Z5nk4G5tTanxY3NfyaOGcWEMHodSKojyS7b4rlk8WdUIOJ83Qb4hrl7fldb6Ic6kX0fvQgAvdaoqL22xppKN2M8hJxbvXMNB1OVIRXEmXQy9Xzro9OGRRKsnk0kjg5rirWSqum+wBxcK4janrFGszArM1NIWoJ9xEFy35EpppCnJB3ZnfNKCKD7kHU7C/y6N7pw7prE3/M7papZFOVaMGU11uVvm2JKP+HWgXYhmqtsLZ1d2BsQ2TWu6qm654tiW2Igrw7cX2gB9EH7q21WFpIz3Bi3/UkCvY3iNrlK8DbrRXMOf8oovZzW0sFMevuHnqeaCLXmm5rIhAeQQSEa6qnqTHPYKIAXk4UOLsRXQtvtjpj7WUVr4NNJfLRWCIvvZDjRxO3JF7r7rdn10UvsoVEBL39iTx3KVGYe1uBqIX0bE8EkV2Jc6A3cc1kfo1phwPTidrn7e6+PGvqA4gupVu8Rj+1RZfqTkQF6QHiGL6cx/QwovX5cI35BhIF3VAiv9yTQWgo61q+C4FZGRA+QLTgtyRaCCOIlslLRGH6rha4xQ0PxxL78jWioJ7t7n+tmq7F/Zl56u+By9z9iUzfKURQX0i0HIqVmG5EkN+PCKZ/ISpmj2YAGkhUdtcQefmg3Mf3Az3d/dYawbofcWyGEGXOtVn5257oEryu1LraGJ02KGRzaQxx10Lpdr/Sd8WTbjKxM98iouS7CmaLOyrOIQ7Mox79xX2IbpGxwJUefbnbEk3Lp1l3PeIlomCbSNwF8iwRqEp356wiuiZqdocU0tCtuknaCFljOyrT+A5xYpe6gqozWV+iJjee7FrYiPX2I2pr44ng/BbRxB0M/MLX/ZpeW91jLe6nDOwDiRbCzdUtrtyeKUTQ+D1RsG3t7n9oa7011lVXgdvGMkrXJO4kWkevVn0/KE/q/Yla4y1EF9lWRN5cTRTeG/VSyCzAphHH4g8ZhLYsBrxC98huRCH7iLvPyu/eQ9Swry0WOjWC31hin8+pqpj1aakylMHxKKJ1tMTdlxa+247om7+IdV1LuxD76WHivJ9KXB+41Wt0hZXSSQSRycR5/RLwevXxaIuZbV21XdsT+/WiGsd2WyJAbkN0Bd2bwWkLIkhuTxTij2eQOJWowMyvlVezQjSeaNG/TrS6il2aG51fSzrlhWaLC3kDiKhuXtVfWsiIuxE76U0iI9bqRhjIugs+T2VAGEw0pZcQB+yRzJw7E7XcZ3zdNYS9iP20lGiSusUdJKOB39JC90K1jggIuZ7VZnY7kVkfLAWE/K6030qBY1ei4Lnb1/Wt1l145sk2mGjeb0t0E8wnAufOuex7SgGhmIZW0t/aflpEFE53VAeEnHe1md1JdKv0JyoUPepZb41lbWxA+DCxjy8lugzfrp6m0PJdQtQQbyNqxqU+/3u96g6tDUjHQKIW2ouorZcKzmIlaxRR6B5AXPtYAAzPQLE2C64nq/dJIT8dRhz70gXO6hsmWgoIfYkAtE9u69Kq+Z4ys+9ky/cYooB9kWjNDSVq4auJWvrCFtYxmOhGvJVozW/weVhd4XT3v5jZD6rLp1QKYgvd/d4cN564ZvAykS8Pzv36nJldWCtIFc7HA4nz9R6iolDRWmuvgACdtKWQNYR9gd95y11GJxCZ+FHgN7WajTldqe/1Go+7HIYRUfpDRA3uCaJWPZIIRIt93V1GHyIy7d1EE3FbovVQuo5w/vrWNjqKmfX2dXeG1Kp5jAQ+R1wYu3Ej1jOUqM31ILrmHiS6bZ4n7pR5uKU0bOD62mxxlWrgG7uujZEF7aFUXZBtZfrSheePEBea57r74xux/h5EV94uRGt2UTE45zSla3dHEMHoZeIW3keJBzHbvMfd4m6304lAcm09FaRM2wSiBb4EWFWsvNSY/kSiFfJHomttIHGjw2qiS+ZdLeHsHehPtCTeILqzltRYfMOUzsHs9fgb0cUznmg5P295zamNZfQhniXpTRyfud4OXUStrrMzBgVo++TPptsU4BKvuvOkxrTlZl8GnClEV9JjRBfLkUTNbEGptlLoqlhO3IHyDnHC7kzcjndPZw0I9SoWnhtSaGdA2IU4YRcR12r2J1pRd7j7gnZO8nppr0C0Eetfrya9xS3BnycKsA15fXz18kYR/fK/8xoXeHOabYk7Z94gum8eqmO5fYkujEOIStOLrOcdL1kpOZXoQindEl7zeOV27EJ0fb2RaZ5GdCG+q6DPgnQ6cS2rP9EKedf1jI5icYPKGPLOqPU9L2zdNYN5HVHmdNqgUA+rcaG1zvl6+brbWocSD4UtIE4ezz7eccSdDAtKzd8MKNOISL+qnTaj6da3yyj3UT/iVr+3Wdc3OprYl/OaHRC6qvZu5Vg+9dvK92OJu2Mudfen6lzmUOLa2zvErca3+Abc8WJm/eut9VZXElu7TpHfjyJ6AxZ63jraDBlAP0Z0S99Z6oXYgOW02zWDNtfVlYNCeyllzqxhnEB0I91JtAberJp2gwLRpsTiTozSw29L3P2GwnejgNHufkeza+pdWUfuu/UpnAvzjCXywM8a3Z2xodoKHB2w/qHE7aJ9iOeXHmtWWtaHgkJBXjgdRVyruLardw+1t6z1HAGUHux7wN1vqzFdh9VqpHnq6RPfXOW1zP2J8+RBb+N27M6ky7wltSN4WEbeE93s9HQ2Hrfd3k08oLSgFBAymBanU0DYDCggtKoHeZNKVwoIoJaCbACLH0apebeXiISu2pWqoCAbrCtldJFm6IpdqQoKIiJSpmsKIiJSpqAgIiJlCgoirbD44fSrzWyJmT1uZt8xsy3N7O/M7PwW5qn5YrY61jXd8sfXRZpFQUGkBXmr7RXEa9tLP7XYDzivtfnc/YANXOV04qVnIk2joCDSskOBN9z9p1B+/uKfiNcW9AFGm9n1ZvaomX2tNJOZFd+r/y9mdpeZPWBm/1YYf3qOu9/MfmFmBxAvb/svM7vP4nXVIh2uU746W6STGE88rFfm8cMxTxHnzkTi9wdeA+4ys2u98ArnwttHJxJPgc+x+L3kF4AvA5PybZml31WYQ7wFc1ZHbJxILQoKIi0z4hfPWhpf/q1tM7uCeOd98b3+R+Rf6X36/YggsSfx1PzzUPG7CiJNp+4jkZYtJt77X5bv6R9N/JZudcCoHjbg6+6+V/69191/QsvBRqTpFBREWjYf6GNmp0P5h3C+BfyM6DI63MwGWfwI+3TiR2CKbgA+lq8Zx8xGWvwm9nzgRItfBSv9jgKse/+/SNMoKIi0IF/hMQM4wcyWED/K9AbwpZzkNuAXxC/OzS5cT/Cc/0bgEuAOM1sEzAL6e/zm8nnArWZ2P/DtnO8y4F/M7F5daJZm0WsuRNpR1v7vcfftm50WkQ2hloJIO8mfibwD+Gaz0yKyodRSEBGRMrUURESkTEFBRETKFBRERKRMQUFERMoUFEREpExBQUREyv4/+QaYIWylDGsAAAAASUVORK5CYII=\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {
+      "needs_background": "light"
+     },
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names = dict(sorted(objectwise_img_names.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# Extract object names and image counts\n",
+    "obj_names = list(objectwise_img_names.keys())\n",
+    "img_counts = [len(objectwise_img_names[obj]) for obj in objectwise_img_names]\n",
+    "print(sum(img_counts))\n",
+    "\n",
+    "# Create bar plot\n",
+    "sns.barplot(x=obj_names, y=img_counts)\n",
+    "\n",
+    "# Add x-axis and y-axis labels\n",
+    "plt.xlabel('Object')\n",
+    "plt.ylabel('Number of Images')\n",
+    "\n",
+    "plt.xticks(rotation=45, ha='right', fontsize=3)\n",
+    "\n",
+    "# Save the plot as a high-resolution image file\n",
+    "out_path = osp.join(out_dir, 'image_per_object_category.png')\n",
+    "plt.savefig(out_path, dpi=300)\n",
+    "\n",
+    "# Show plot\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

hot_analysis/agniv_pose_filter/hico.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b01acf6e31dad34eb1e48277e227ea0668f17905f55eae44b640cc1d14a9e41c
+size 520245

hot_analysis/agniv_pose_filter/hot.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64872b84670cb2ec643f1649e1a7926aed950adcb8f6f952916924bc4ab10f37
+size 731363

hot_analysis/agniv_pose_filter/hot_dict.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0c0a6e35bf87e98327189ef8d732aa19b0d1c85c963c1ee348e1e466864845e
+size 3679000

hot_analysis/agniv_pose_filter/pq_wnp.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cad59d5aaf077141d60a492c3784d8994d53f4dfd020f9eb4e11b3e430ef65d
+size 25433

hot_analysis/agniv_pose_filter/vcoco.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14cc34940ff4a61cfc952d8c5b98e0e7b1a58392177198fa4589f8b2bcead02a
+size 184801

hot_analysis/count_objects_per_img.py

@@ -0,0 +1,35 @@
+# get average number of objects per image
+import os.path as osp
+import json
+import plotly.express as px
+import plotly.io as pio
+
+version = '1'
+dir = '/is/cluster/work/stripathi/pycharm_remote/dca_contact/hot_analysis/'
+out_dir_hico = osp.join(dir, f'filtered_data/v_{version}/hico')
+out_dir_vcoco = osp.join(dir, f'filtered_data/v_{version}/vcoco')
+
+imgwise_obj_dict_hico = osp.join(out_dir_hico, 'object_per_image_dict.json')
+imgwise_obj_dict_vcoco = osp.join(out_dir_vcoco, 'object_per_image_dict.json')
+
+with open(imgwise_obj_dict_hico, 'r') as fp:
+    imgwise_obj_dict_hico = json.load(fp)
+with open(imgwise_obj_dict_vcoco, 'r') as fp:
+    imgwise_obj_dict_vcoco = json.load(fp)
+
+# combine the dicts
+imgwise_obj_dict = imgwise_obj_dict_hico.copy()
+imgwise_obj_dict.update(imgwise_obj_dict_vcoco)
+
+# get average length of object in the object per image key
+avg_obj_per_img = sum([len(v) for v in imgwise_obj_dict.values()]) / len(imgwise_obj_dict)
+print(f'Average number of objects per image: {avg_obj_per_img}')
+
+# get average searately for hico and vcoco
+avg_obj_per_img_hico = sum([len(v) for v in imgwise_obj_dict_hico.values()]) / len(imgwise_obj_dict_hico)
+print(f'Average number of objects per image in hico: {avg_obj_per_img_hico}')
+
+avg_obj_per_img_vcoco = sum([len(v) for v in imgwise_obj_dict_vcoco.values()])  / len(imgwise_obj_dict_vcoco)
+print(f'Average number of objects per image in vcoco: {avg_obj_per_img_vcoco}')
+
+

hot_analysis/create_combined_objectwise_plots.ipynb

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "1738c3a5",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "import os.path as osp\n",
+    "import json\n",
+    "import os\n",
+    "\n",
+    "version = '1'\n",
+    "out_dir_hico = f'./filtered_data/v_{version}/hico'\n",
+    "out_dir_vcoco = f'./filtered_data/v_{version}/vcoco'\n",
+    "out_dir_phosa = f'./filtered_data/v_{version}/phosa_split'\n",
+    "out_dir_behave = f'./filtered_data/v_{version}/behave_split'\n",
+    "\n",
+    "objectwise_img_dict_hico = osp.join(out_dir_hico, 'imgnames_per_object_dict.json')\n",
+    "objectwise_img_dict_vcoco = osp.join(out_dir_vcoco, 'imgnames_per_object_dict.json')\n",
+    "\n",
+    "with open(objectwise_img_dict_hico, 'r') as fp:\n",
+    "    objectwise_img_dict_hico = json.load(fp)\n",
+    "with open(objectwise_img_dict_vcoco, 'r') as fp:\n",
+    "    objectwise_img_dict_vcoco = json.load(fp)\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names1 = dict(sorted(objectwise_img_dict_hico.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# # Extract object names and image counts\n",
+    "# obj_names1 = list(objectwise_img_names_hico.keys())\n",
+    "# print(sorted(obj_names1))\n",
+    "# image_count1 = [len(objectwise_img_names_hico[obj]) for obj in objectwise_img_names_hico]\n",
+    "# print(sum(image_count1))\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names2 = dict(sorted(objectwise_img_dict_vcoco.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# # Extract object names and image counts\n",
+    "# obj_names2 = list(objectwise_img_names_vcoco.keys())\n",
+    "# print(sorted(obj_names2))\n",
+    "# image_count2 = [len(objectwise_img_names_vcoco[obj]) for obj in objectwise_img_names_vcoco]\n",
+    "# print(sum(image_count2))\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "2d6ad5ed",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2119\n",
+      "saved at ./filtered_data/v_1/phosa_split/imgnames_per_object_dict.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Extract image list for object in PHOSA\n",
+    "def merge_dictionaries(dict1, dict2):\n",
+    "    merged_dict = {}\n",
+    "\n",
+    "    # Merge values for common keys\n",
+    "    for key in set(dict1.keys()) | set(dict2.keys()):\n",
+    "        merged_dict[key] = list(set(dict1.get(key, [])) | set(dict2.get(key, [])))\n",
+    "\n",
+    "    return merged_dict\n",
+    "\n",
+    "def filter_dictionary(dictionary, keys):\n",
+    "    filtered_dict = {key: dictionary[key] for key in keys if key in dictionary}\n",
+    "    return filtered_dict\n",
+    "\n",
+    "phosa_object_names = [\"bat\", \"bench\", \"bicycle\", \"laptop\", \"motorcycle\", \"skateboard\", \"surfboard\", \"tennis\"]\n",
+    "combined_objectwise_img_names = merge_dictionaries(objectwise_img_names1, objectwise_img_names2)\n",
+    "phosa_objectwise_img_names = filter_dictionary(combined_objectwise_img_names, phosa_object_names)\n",
+    "net_img_count = 0\n",
+    "for v in phosa_objectwise_img_names.values():\n",
+    "    net_img_count += len(v)\n",
+    "print(net_img_count)\n",
+    "# Save the object_wise dict\n",
+    "os.makedirs(out_dir_phosa, exist_ok=True)\n",
+    "out_path_phosa = osp.join(out_dir_phosa, 'imgnames_per_object_dict.json')\n",
+    "with open(out_path_phosa, 'w') as fp:\n",
+    "    json.dump(phosa_objectwise_img_names, fp)\n",
+    "    print(f'saved at {out_path_phosa}')\n",
+    "    \n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "3f2a5e62",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "saved at ./filtered_data/v_1/behave_split/imgnames_per_object_dict.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Extract image list for object in CHORE\n",
+    "\n",
+    "behave_object_names_original = [\"backpack\", \"basketball\", \"boxlarge\", \"boxlong\", \"boxmedium\", \"boxsmall\", \"boxtiny\", \"chairblack\", \"chairwood\", \"keyboard\", \"monitor\", \"plasticcontainer\", \"stool\", \"suitcase\", \"tablesmall\", \"tablesquare\", \"toolbox\", \"trashbin\", \"yogaball\", \"yogamat\"]\n",
+    "behave_object_names_in_hot = [\"backpack\", \"chair\", \"keyboard\", \"suitcase\"]\n",
+    "\n",
+    "combined_objectwise_img_names = merge_dictionaries(objectwise_img_names1, objectwise_img_names2)\n",
+    "behave_objectwise_img_names = filter_dictionary(combined_objectwise_img_names, behave_object_names_in_hot)\n",
+    "net_img_count = 0\n",
+    "for v in behave_objectwise_img_names.values():\n",
+    "    net_img_count += len(v)\n",
+    "\n",
+    "# Save the object_wise dict\n",
+    "os.makedirs(out_dir_behave, exist_ok=True)\n",
+    "out_path_behave = osp.join(out_dir_behave, 'imgnames_per_object_dict.json')\n",
+    "with open(out_path_behave, 'w') as fp:\n",
+    "    json.dump(behave_objectwise_img_names, fp)\n",
+    "    print(f'saved at {out_path_behave}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "730e9ac2",
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "ModuleNotFoundError",
+     "evalue": "No module named 'seaborn'",
+     "output_type": "error",
+     "traceback": [
+      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
+      "\u001B[0;31mModuleNotFoundError\u001B[0m                       Traceback (most recent call last)",
+      "Cell \u001B[0;32mIn[3], line 2\u001B[0m\n\u001B[1;32m      1\u001B[0m \u001B[38;5;28;01mimport\u001B[39;00m \u001B[38;5;21;01mmatplotlib\u001B[39;00m\u001B[38;5;21;01m.\u001B[39;00m\u001B[38;5;21;01mpyplot\u001B[39;00m \u001B[38;5;28;01mas\u001B[39;00m \u001B[38;5;21;01mplt\u001B[39;00m\n\u001B[0;32m----> 2\u001B[0m \u001B[38;5;28;01mimport\u001B[39;00m \u001B[38;5;21;01mseaborn\u001B[39;00m \u001B[38;5;28;01mas\u001B[39;00m \u001B[38;5;21;01msns\u001B[39;00m\n\u001B[1;32m      4\u001B[0m \u001B[38;5;66;03m# sort the dictionary \u001B[39;00m\n\u001B[1;32m      5\u001B[0m objectwise_img_names1 \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mdict\u001B[39m(\u001B[38;5;28msorted\u001B[39m(objectwise_img_names1\u001B[38;5;241m.\u001B[39mitems(), key\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mlambda\u001B[39;00m x: \u001B[38;5;28mlen\u001B[39m(x[\u001B[38;5;241m1\u001B[39m]), reverse\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mTrue\u001B[39;00m))\n",
+      "\u001B[0;31mModuleNotFoundError\u001B[0m: No module named 'seaborn'"
+     ]
+    }
+   ],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names1 = dict(sorted(objectwise_img_names1.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# Extract object names and image counts\n",
+    "obj_names1 = list(objectwise_img_names1.keys())\n",
+    "print(obj_names1)\n",
+    "img_counts1 = [len(objectwise_img_names1[obj]) for obj in objectwise_img_names1]\n",
+    "print(sum(img_counts1))\n",
+    "\n",
+    "# sort the dictionary \n",
+    "objectwise_img_names2 = dict(sorted(objectwise_img_names2.items(), key=lambda x: len(x[1]), reverse=True))\n",
+    "\n",
+    "# Extract object names and image counts\n",
+    "obj_names2 = list(objectwise_img_names2.keys())\n",
+    "print(obj_names2)\n",
+    "img_counts2 = [len(objectwise_img_names2[obj]) for obj in objectwise_img_names2]\n",
+    "print(sum(img_counts2))\n",
+    "\n",
+    "# Create a figure with two subplots\n",
+    "fig, axs = plt.subplots(nrows=2, sharex=True, figsize=(10, 8))\n",
+    "\n",
+    "# Plot the first graph in the first subplot\n",
+    "sns.barplot(x=obj_names1, y=img_counts1, ax=axs[0])\n",
+    "\n",
+    "# Add y-axis label to the first subplot\n",
+    "axs[0].set_ylabel('Number of Images')\n",
+    "\n",
+    "# Set the rotation and alignment of x-tick labels in the first subplot\n",
+    "axs[0].tick_params(axis='x', rotation=45)\n",
+    "\n",
+    "# Plot the second graph in the second subplot\n",
+    "sns.barplot(x=obj_names2, y=img_counts2, ax=axs[1])\n",
+    "\n",
+    "# Add x-axis and y-axis labels to the second subplot\n",
+    "axs[1].set_xlabel('Object')\n",
+    "axs[1].set_ylabel('Number of Images')\n",
+    "\n",
+    "# Set the rotation and alignment of x-tick labels in the second subplot\n",
+    "axs[1].tick_params(axis='x', rotation=45)\n",
+    "\n",
+    "# Save the plot as a high-resolution image file\n",
+    "out_path = osp.join(out_dir, 'image_per_object_category.png')\n",
+    "plt.savefig(out_path, dpi=300)\n",
+    "\n",
+    "# Show the plot\n",
+    "plt.show()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "23c503dd",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_135310/1442551168.py:22: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
+      "  ax.set_xticklabels(X1)\n"
+     ]
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAALCCAYAAAA/AFnzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABUsklEQVR4nO3deVyU5f7/8fcAMiKrGioUAhJppR1PZXXcyCVtsdL6tZipaC7ZappbJ/cKMYvqmJZZaracFivrtGCZS9FipZ7UMnEBSSp32RTEuX9/9GWOyKBcOXCzvJ6Ph49Hc9/X3NfnnrkaeHPf1zUOy7IsAQAAAAAqxMfuAgAAAACgJiFEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAYCkxMRExcTEVNrxL7/8cl1++eWVdnxvmzJlihwOh/bu3VtpfWRkZMjhcGjhwoUnbbdw4UI5HA59//33lVZLTZWYmKigoKAq6Wvx4sVq1aqV6tWrp7CwsCrps7pwOByaMmWK3WUAqEYIUQBsc91116lBgwbKzc0tt02/fv3k7++vffv2nXZ/2dnZmjJlitavX3/ax7JLbTgH1DybN29WYmKi4uLi9MILL2jevHll2hw9elRt2rRRXFycDh8+XGZ/RkaGGjRooJtuukmSlJeXp8mTJ+vKK69Uo0aNKhSoq6sZM2bI4XAoNTXV4/6rr75aoaGhys7O1r59+/T444+rc+fOCg8PV1hYmC677DK98cYbVVw1gNNBiAJgm379+unw4cN69913Pe4vKCjQ0qVLdeWVV6px48an3V92dramTp3qMYC88MIL+uWXX067j8p2snMAKsvKlSvlcrn09NNPKzExUTfffHOZNvXq1dO8efO0Y8cOTZ8+vcz+e+65R/7+/nrmmWckSXv37tW0adP0888/629/+1uln0NlGj16tNq0aaO77rqrTIB866239PHHHyspKUmRkZH6+uuv9c9//lONGjXSww8/rEcffVQNGjTQrbfeqsmTJ9t0BgBMEaIA2Oa6665TcHCwXnvtNY/7ly5dqvz8fPXr1++0+ikuLlZRUdFJ29SrV09Op/O0+kHNVlBQUCnHrcj4q+52794tSae8je8f//iH7rzzTs2aNUubNm1yb1+yZIk+/PBDJSUlKSIiQpIUERGh3377TZmZmXr88ccrrfaqUBIgMzIySgXI3NxcjRw5UpdddpnuvPNOSdL555+v9PR0vffee7r//vt19913a/ny5eratauSk5OVn59v12kAMECIAmCbgIAA3XDDDVq+fLn7l7TjvfbaawoODtZ1110nSTp48KBGjhypqKgoOZ1OnX322UpOTpbL5XI/p2SezaxZs/TUU08pLi5OTqdTc+bMUbt27SRJgwYNksPhKHX7kKc5USV/eW/Tpo3q16+v8PBwXXnllaXm5ixYsEBdu3ZVkyZN5HQ6dd5552nu3Ll/+TX59NNP1bFjR4WFhSkoKEgtW7bUQw89JOnPqwEnO4cvvvhCN910k5o3by6n06moqCg98MADHm+t2rx5s26++WaFh4crICBALVu21D//+c+T1paZmamzzz5brVu31h9//CGpYu9JSbvExESFhoYqLCxMAwcO1MGDB41em4KCAg0fPlyNGzdWSEiIBgwYoAMHDpRqs3TpUl1zzTWKjIyU0+lUXFycpk+frmPHjpVqd/nll6t169b64Ycf1LlzZzVo0MD9Opfnrbfe0nnnnaf69eurdevWevfdd8uMm/LG308//aSioiJNmjRJF110kUJDQxUYGKhOnTppxYoVpfo5/hgpKSmKjo5WQECAEhIStHHjRo+17dq1S71791ZQUJDCw8P14IMPljnn8syZM0fnn3++nE6nIiMjdffdd5d6b2JiYtxXSMLDw085PygpKUlnnHGG7rzzTlmWpby8PI0cOdIdsEo4nU41a9asQjV68uOPPyoxMVEtWrRQ/fr11axZMw0ePLjMrb8l8/u2bt2qxMREhYWFKTQ0VIMGDSoTnAsLC/XAAw8oPDzc/dnz66+/VqiekqA0a9Ys/fTTT5Kkhx9+WLt379a8efPk4/Pnr1yxsbGKjo4u9VyHw6HevXursLBQ27dv/6svCYAq5Gd3AQDqtn79+mnRokV68803dc8997i379+/X6mpqerbt68CAgJUUFCghIQE7dq1S8OHD1fz5s311VdfacKECfrtt9/01FNPlTruggULdOTIEQ0bNkxOp1N9+vRRbm6uJk2apGHDhqlTp06SpPbt25db2x133KGFCxfqqquu0pAhQ1RcXKwvvvhC33zzjS6++GJJ0ty5c3X++efruuuuk5+fnz744APdddddcrlcuvvuu41ei02bNqlXr1664IILNG3aNDmdTm3dulVpaWmSpHPPPVfTpk0r9xzeeustFRQUaMSIEWrcuLHWrFmjf/3rX/r111/11ltvufv58ccf1alTJ9WrV0/Dhg1TTEyMtm3bpg8++ECPPvqox9q2bdumrl27qlGjRvr00091xhlnVPg9sSxL119/vb788kvdeeedOvfcc/Xuu+9q4MCBRq/PPffco7CwME2ZMkW//PKL5s6dq8zMTK1cuVIOh0PSn4tQBAUFadSoUQoKCtLnn3+uSZMmKScnp8zVjn379umqq67Srbfeqttvv11NmzYtt+8PP/xQt9xyi9q0aaOkpCQdOHBAd9xxh84880yP7U8cf40aNVJOTo7mz5+vvn37aujQocrNzdWLL76onj17as2aNWrbtm2pY7z88svKzc3V3XffrSNHjujpp59W165dtWHDhlK1Hjt2TD179tSll16qWbNm6bPPPtMTTzyhuLg4jRgx4qSv6ZQpUzR16lR1795dI0aMcL+u3333ndLS0lSvXj099dRTevnll/Xuu+9q7ty5CgoK0gUXXFDuMUNDQ/XMM8/opptu0vz58/XTTz/pjz/+0Mcff+x+n7zh008/1fbt2zVo0CA1a9ZMmzZt0rx587Rp0yZ98803Zfq6+eabFRsbq6SkJK1du1bz589XkyZNlJyc7G4zZMgQvfLKK7rtttvUvn17ff7557rmmmsqXFNSUpLee+89DR8+XE899ZSeffZZjRkzRm3atDnlc3///XdJ0hlnnFHh/gDYyAIAGxUXF1sRERHWP/7xj1Lbn3vuOUuSlZqaalmWZU2fPt0KDAy0tmzZUqrd+PHjLV9fX2vnzp2WZVnWjh07LElWSEiItXv37lJtv/vuO0uStWDBgjJ1DBw40IqOjnY//vzzzy1J1n333Vemrcvlcv93QUFBmf09e/a0WrRoUWpbQkKClZCQUPYFOE5KSoolydqzZ0+5bU52Dp5qSUpKshwOh5WZmene1rlzZys4OLjUNssqfV6TJ0921/Lzzz9bkZGRVrt27az9+/e721T0PXnvvfcsSdbMmTPdbYqLi61OnTqVey7HW7BggSXJuuiii6yioiL39pkzZ1qSrKVLl570NRg+fLjVoEED68iRI+5tCQkJliTrueeeO2nfJdq0aWOdddZZVm5urnvbypUrLUmlxs3Jxl9xcbFVWFhYatuBAwespk2bWoMHDy5zjICAAOvXX391b//2228tSdYDDzzg3jZw4EBLkjVt2rRSx/373/9uXXTRRSc9p927d1v+/v5Wjx49rGPHjrm3z54925JkvfTSS+5tx4+HiurVq5cVGhpq+fr6WhMmTDhp25ON6/J4eq9ff/11S5K1evXqMrUf/xpblmX16dPHaty4sfvx+vXrLUnWXXfdVardbbfdZkmyJk+eXKG63n77bUuS1ahRI6tFixYe6zzRvn37rCZNmlidOnWqUB8A7MftfABs5evrq1tvvVVff/21MjIy3Ntfe+01NW3aVN26dZP051WWTp06qWHDhtq7d6/7X/fu3XXs2DGtXr261HFvvPFGhYeH/+W6lixZIofD4XGi9/F/4Q4ICHD/96FDh7R3714lJCRo+/btOnTokFGfJfNNli5dWuZ2uIo4vpb8/Hzt3btX7du3l2VZWrdunSRpz549Wr16tQYPHqzmzZuXer6nqwQbN25UQkKCYmJi9Nlnn6lhw4bufRV9Tz766CP5+fmVuiri6+ure++91+j8hg0bpnr16rkfjxgxQn5+fvroo488vga5ubnau3evOnXqpIKCAm3evLnU8ZxOpwYNGnTKfrOzs7VhwwYNGDCg1HLiCQkJ5V5h8DT+fH195e/vL+nPW0X379+v4uJiXXzxxVq7dm2ZY/Tu3bvUla5LLrlEl156aanzLXH8bXKS1KlTp1PeFvbZZ5+pqKhII0eOdN9qJklDhw5VSEiIPvzww5M+/1SeffZZFRUVKSoqShMnTjytY3ly/Ht95MgR7d27V5dddpkkeXw9Pb1G+/btU05OjiS5X9f77ruvVLuRI0ca1XXjjTfq6quv1v79+/Xss8+WqtMTl8ulfv366eDBg/rXv/5l1BcA+xCiANiuZOGIkgUmfv31V33xxRe69dZb5evrK0lKT0/XJ598ovDw8FL/unfvLkll5lTFxsaeVk3btm1TZGSkGjVqdNJ2aWlp6t69uwIDAxUWFqbw8HD33BrTEHXLLbeoQ4cOGjJkiJo2bapbb71Vb775ZoUD1c6dO5WYmKhGjRq558YkJCSUqqXkF+vWrVtX6JjXXnutgoODlZqaqpCQkFL7KvqeZGZmKiIiosz3GbVs2bJCNZSIj48v9TgoKEgRERGlwvemTZvUp08fhYaGKiQkROHh4br99tsllX0/zjzzTHeoOZnMzExJ0tlnn11mn6dtUvnjb9GiRbrgggtUv359NW7cWOHh4frwww89jpUTz1eSzjnnnFLnK8k9X+94DRs2LDNf7EQl53Xi++Dv768WLVq49/9VzZs3V5MmTXT++eefMkj8Ffv379f999+vpk2bKiAgQOHh4e7X3dPreeIfDUr+IFDyOmVmZsrHx0dxcXGl2pmOU0nuuYslt/2ezL333qtPPvlE8+fPr/GrFAJ1CXOiANjuoosuUqtWrfT666/roYce0uuvvy7LskqtyudyuXTFFVdo7NixHo9xzjnnlHpcGb+0nWjbtm3q1q2bWrVqpSeffFJRUVHy9/fXRx99pJSUFOOrSQEBAVq9erVWrFihDz/8UJ988oneeOMNde3aVcuWLXMHSk+OHTumK664Qvv379e4cePUqlUrBQYGateuXUpMTPxLV7akP/+qvmjRIr366qsaPnx4qX2m70llO3jwoBISEhQSEqJp06YpLi5O9evX19q1azVu3Lgyr0FljhFPx37llVeUmJio3r17a8yYMWrSpIl8fX2VlJSkbdu2/eW+TjYuarObb75ZX331lcaMGaO2bdsqKChILpdLV155pcfxXt7rZFlWZZdarqlTp2rOnDmaMWOG+vfvb1sdAMwRogBUC/369dPEiRP1448/6rXXXlN8fLz7r7mSFBcXp7y8PPdVjr/CZFJ7XFycUlNTtX///nKvRn3wwQcqLCzU+++/X+qv3CeutmbCx8dH3bp1U7du3fTkk0/qscce0z//+U+tWLFC3bt3L/ccNmzYoC1btmjRokUaMGCAe/unn35aql2LFi0kqdxV3k70+OOPy8/PT3fddZeCg4N12223ufdV9D2Jjo7W8uXLlZeXV+pqlOn3cqWnp6tLly7ux3l5efrtt9909dVXS/pz9cJ9+/bpnXfeUefOnd3tduzYYdSPp/olaevWrWX2edpWnrffflstWrTQO++8U+p9LO+7gdLT08ts27JlS5lVJP+qkvP65Zdf3ONCkoqKirRjx47T+n+tsh04cEDLly/X1KlTNWnSJPd2T69ZRUVHR8vlcmnbtm2lrj5V1vfHPfvss5oyZYpGjhypcePGVUofACoPt/MBqBZKrjpNmjRJ69evL/PdUDfffLO+/vprpaamlnnuwYMHVVxcfMo+AgMD3e1P5cYbb5RlWZo6dWqZfSV/uS75y/bxf8k+dOiQFixYcMrje7J///4y20pWbCssLJRU/jl4qsWyLD399NOl2oWHh6tz58566aWXtHPnzlL7PP1F3uFwaN68efp//+//aeDAgXr//ffd+yr6nlx99dUqLi4utfT7sWPHjOd/zJs3T0ePHnU/njt3roqLi3XVVVdJ8vwaFBUVac6cOUb9nCgyMlKtW7fWyy+/rLy8PPf2VatWacOGDRU+jqf6vv32W3399dce27/33nvatWuX+/GaNWv07bffus/3dHXv3t395bfH1/Tiiy/q0KFDRqvSVTVPr6WkMqt0mih5XUu+DNgbxyzPG2+8ofvuu0/9+vXTk08+6fXjA6h8XIkCUC3Exsaqffv2Wrp0qSSVCVFjxozR+++/r169eikxMVEXXXSR8vPztWHDBr399tvKyMg45dLAcXFxCgsL03PPPafg4GAFBgbq0ksv9Th/pUuXLurfv7+eeeYZpaenu28R+uKLL9SlSxfdc8896tGjh/z9/XXttddq+PDhysvL0wsvvKAmTZrot99+M34Npk2bptWrV+uaa65RdHS0du/erTlz5uiss85Sx44dT3oOrVq1UlxcnB588EHt2rVLISEhWrJkicd5Mc8884w6duyoCy+8UMOGDVNsbKwyMjL04Ycfav369WXa+/j46JVXXlHv3r11880366OPPlLXrl0r/J5ce+216tChg8aPH6+MjAydd955euedd4znjBUVFalbt266+eab9csvv2jOnDnq2LGj+3vE2rdvr4YNG2rgwIG677775HA4tHjxYq/crvXYY4/p+uuvV4cOHTRo0CAdOHBAs2fPVuvWrUsFq5Pp1auX3nnnHfXp00fXXHONduzYoeeee07nnXeex2OcffbZ6tixo0aMGKHCwkI99dRTaty4cbm3T5oKDw/XhAkTNHXqVF155ZW67rrr3K9ru3bt3HPJKtPs2bN18OBBZWdnS/rz6m7J9zLde++9Cg0N9fi8kJAQde7cWTNnztTRo0d15plnatmyZad11bFt27bq27ev5syZo0OHDql9+/Zavny50dXGilizZo0GDBigxo0bq1u3bnr11VdL7W/fvn2pK4MAqik7lgQEAE+effZZS5J1ySWXeNyfm5trTZgwwTr77LMtf39/64wzzrDat29vzZo1y730dcny0I8//rjHYyxdutQ677zzLD8/v1JLKp+4xLll/bkk9eOPP261atXK8vf3t8LDw62rrrrK+uGHH9xt3n//feuCCy6w6tevb8XExFjJycnWSy+9ZEmyduzY4W5XkSXOly9fbl1//fVWZGSk5e/vb0VGRlp9+/Yts4R4eefw008/Wd27d7eCgoKsM844wxo6dKj13//+1+PS0Rs3brT69OljhYWFWfXr17datmxpTZw40b3f05LWBQUFVkJCghUUFGR98803FX5PLOvPJZz79+9vhYSEWKGhoVb//v2tdevWGS1xvmrVKmvYsGFWw4YNraCgIKtfv37Wvn37SrVNS0uzLrvsMisgIMCKjIy0xo4da6WmplqSrBUrVpR6P84///yT9nuif//731arVq0sp9NptW7d2nr//fetG2+80WrVqpW7zcnGn8vlsh577DErOjracjqd1t///nfrP//5T5mxd/wxnnjiCSsqKspyOp1Wp06drP/+97+ljjlw4EArMDCwTF8l719FzJ4922rVqpVVr149q2nTptaIESOsAwcOeDyeyRLnlmVZ0dHR1jXXXHPS/ZI8/jv+/x9Pfv31V/cYDg0NtW666SYrOzu7zHLk5dVeMq6O7+fw4cPWfffdZzVu3NgKDAy0rr32WisrK8toifOT9Xl8v+X9M1nmHYB9HJZl44xKAABqsLZt2yo8PLzM3LPTkZGRodjYWD3++ON68MEHvXZcAID3MCcKAIBTOHr0aJl5dytXrtR///tfXX755fYUBQCwDXOiAAA4hV27dql79+66/fbbFRkZqc2bN+u5555Ts2bNynyJKwCg9iNEAQBwCg0bNtRFF12k+fPna8+ePQoMDNQ111yjGTNmqHHjxnaXBwCoYsyJAgAAAAADzIkCAAAAAAOEKAAAAAAwUKfnRLlcLmVnZys4OFgOh8PucgAAAADYxLIs5ebmKjIyUj4+J7/WVKdDVHZ2tqKiouwuAwAAAEA1kZWVpbPOOuukbep0iAoODpb05wsVEhJiczUAAAAA7JKTk6OoqCh3RjiZOh2iSm7hCwkJIUQBAAAAqNA0HxaWAAAAAAADhCgAAAAAMECIAgAAAAADdXpOVEVYlqXi4mIdO3bM7lJs4evrKz8/P5aABwAAAP4PIeokioqK9Ntvv6mgoMDuUmzVoEEDRUREyN/f3+5SAAAAANsRosrhcrm0Y8cO+fr6KjIyUv7+/nXuaoxlWSoqKtKePXu0Y8cOxcfHn/KLxwAAAIDajhBVjqKiIrlcLkVFRalBgwZ2l2ObgIAA1atXT5mZmSoqKlL9+vXtLgkAAACwFZcVToErL7wGAAAAwPH47RgAAAAADBCiAAAAAMAAIeovsCyrVvYFAAAA4NRYWOIvcDgc+jbzgHILiyu1n2Cnny6Nbljh9pZl6YorrpCvr69SU1NL7ZszZ44eeughbdy4UQ8//LB++OEH/fzzz+rVq5fee+89L1cOAAAA1F6EqL8ot7BYBw9Xbogy5XA4tGDBArVp00bPP/+8hg8fLknasWOHxo4dq7lz56phw4YKCAjQfffdpyVLlthcMQAAAFDzEKJqmaioKD399NO655571KNHD8XExOiOO+5Qjx491L9/f0nS3LlzJUlpaWk6ePCgjdUCAAAANQ8hqhYaOHCg3n33XQ0ePFg33HCDNm7cqE2bNtldFgAAAFArEKJqqXnz5un888/X6tWrtWTJEoWHh9tdEgAAAFArsDpfLdWkSRMNHz5c5557rnr37m13OQAAAECtQYiqxfz8/OTnx8VGAAAAwJsIUQAAAABggMsUf1Gws/Jfusrq46efflJRUZH279+v3NxcrV+/XpLUtm3bSukPAAAAqE1sDVELFy7UoEGDTtqma9euWr58uftxTk6OpkyZoiVLluj3339XRESEbrrpJk2ePFlBQUGVXbKkP7/U1uRLcE+3L4fD4dVjXn311crMzHQ//vvf/+7uCwAAAMDJ2Rqi2rZtq8mTJ3vc9/bbb2vTpk3q2bOne1t+fr4SEhK0fv169ejRQ3379tW6des0a9YsrVq1SqtXr1b9+vUrvW5vh5rK6mvKlCmaMmVKme0ZGRl/vSAAAACgjrM9RHm6hayoqEizZ8+Wn5+fBg4c6N4+c+ZMrV+/XuPGjdOMGTPc28ePH6/k5GSlpKRowoQJVVE6AAAAgDqqWi4s8d5772nfvn3q1auXmjZtKunPW83mz5+voKAgTZw4sVT7iRMnKigoSPPnz7ejXAAAAAB1SLUMUSVhaMiQIe5t6enpys7OVocOHRQYGFiqfWBgoDp06KDt27crKyurSmsFAAAAULdUu9X5MjMztXz5cp111lm68sor3dvT09MlSfHx8R6fFx8fr9TUVKWnpysqKspjm8LCQhUWFrof5+TkSJJcLpdcLlepti6XS5Zluf/VZSWvgafXCQAAAKgNTH7PrXYhasGCBXK5XEpMTJSvr697+6FDhyRJoaGhHp8XEhJSqp0nSUlJmjp1apntWVlZCg4OLrXt2LFjOnbsmI4ePSofn2p5wa7KHD16VMeOHdNvv/1W6j0BAACoSi7Lkk8VLvCFqlFd3tfc3NwKt61WIcrlcmnBggVyOBwaPHiw148/YcIEjRo1yv04JydHUVFRioqKcoewEkeOHFFGRobq1asnf39/r9dSk7hcLvn6+ioiIqJKVj8EAAAoz7QPNitjf4HdZcBLYho10KRrW9ldhqT/3aVWEdUqRH322WfauXOnunXrptjY2FL7Sq5AlXelqeSky7tSJUlOp1NOp7PMdh8fnzJXm3x8fORwONz/6rKS18DT6wQAAFCVduw/rF/+yLe7DHiJJUe1+f3SpI7qUfH/8bSgRImSuVAlc6NOdKo5UwAAAADgDdUmRO3bt09Lly5Vo0aN1KdPnzL74+PjFRkZqbS0NOXnl/7rQ35+vtLS0hQbG1vuohIAAAAA4A3VJkQtXrxYRUVFuv322z3ecudwODRkyBDl5eVp+vTppfZNnz5deXl5Gjp0aJXUesxVdav1VWVfAAAAAE6t2syJevHFFyV5vpWvxNixY7V06VIlJydr3bp1uvDCC7V27VotW7ZM7dq108iRI6ukVl8fhya//7My9lXupMaYxg009bpzK9zesixdccUV8vX1VWpqaql9c+bM0UMPPaTZs2frrbfe0po1a5STk6P4+HiNGTNG/fr183b5AAAAQK1ULULUmjVrtHHjRl1yySVq06ZNue0CAwO1atUqTZkyRUuWLNGKFSsUERGh0aNHa/LkyQoICKiymjP2FeiXP/KqrL+KcDgcWrBggdq0aaPnn39ew4cPlyTt2LFDY8eO1dy5c7Vz505dcMEFGjdunJo2bar//Oc/GjBggEJDQ9WrVy+bzwAAAACo/qpFiLrkkksq/IW2oaGhSklJUUpKSiVXVTNFRUXp6aef1j333KMePXooJiZGd9xxh3r06KH+/fuXaX///fdr2bJleueddwhRAAAAQAVUixAF7xo4cKDeffddDR48WDfccIM2btyoTZs2ldv+0KFDOvfcit82CAAAANRlhKhaat68eTr//PO1evVqLVmyROHh4R7bvfnmm/ruu+/0/PPPV3GFAAAAQM1UbVbng3c1adJEw4cP17nnnqvevXt7bLNixQoNGjRIL7zwgs4///yqLRAAAACooQhRtZifn5/8/DxfbFy1apWuvfZapaSkaMCAAVVcGQAAAFBzEaLqoJUrV+qaa65RcnKyhg0bZnc5AAAAQI3CnKi/KKZxgxrZx4oVK9SrVy/df//9uvHGG/X7779Lkvz9/dWoUSOv9wcAAADUNoSov+CYyzL6EtzT7cvXx+G14y1atEgFBQVKSkpSUlKSe3tCQoJWrlzptX4AAACA2orb+f4Cb4aayuxrypQpWr9+faltCxculGVZZf4RoAAAAICKIUQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIESdgmVZdpdgO14DAAAA4H8IUeWoV6+eJKmgoMDmSuxX8hqUvCYAAABAXcb3RJXD19dXYWFh2r17tySpQYMGcjiqbmnz6sCyLBUUFGj37t0KCwuTr6+v3SUBAAAAtiNEnUSzZs0kyR2k6qqwsDD3awEAAADUdYSok3A4HIqIiFCTJk109OhRu8uxRb169bgCBQAAAByHEFUBvr6+BAkAAAAAklhYAgAAAACMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMEKIAAAAAwAAhCgAAAAAMVJsQ9e677+qKK65Q48aNVb9+fcXGxqpv377Kysoq1S4nJ0ejRo1SdHS0nE6nYmJiNGbMGOXl5dlUOQAAAIC6xM/uAizL0p133ql58+YpLi5Ot956q4KDg5Wdna1Vq1YpMzNTUVFRkqT8/HwlJCRo/fr16tGjh/r27at169Zp1qxZWrVqlVavXq369evbfEYAAAAAajPbQ9QzzzyjefPm6a677tIzzzwjX1/fUvuLi4vd/z1z5kytX79e48aN04wZM9zbx48fr+TkZKWkpGjChAlVVjsAAACAusdhWZZlV+eHDx/WmWeeqYYNG+qXX36Rn1/5mc6yLJ111lnKycnR77//rsDAQPe+/Px8NWvWTE2aNNG2bdsq3H9OTo5CQ0N16NAhhYSEnNa5AAAAoPINXPCDfvmDaRy1RcumQVo06CK7y5Bklg1svRK1bNkyHThwQIMGDdKxY8f0/vvva8uWLQoLC1P37t119tlnu9ump6crOztbPXv2LBWgJCkwMFAdOnRQamqqsrKy3Lf/AQAAAIC32RqifvjhB0mSr6+vLrjgAm3ZssW9z8fHRw888IBmzZol6c8QJUnx8fEejxUfH6/U1FSlp6eXG6IKCwtVWFjofpyTkyNJcrlccrlcp39CAAAAqFQOWfKRbTdSwcscsqrN7+Emddgaonbv3i1JevLJJ3XhhRdqzZo1Ovfcc7Vu3ToNGzZMTzzxhOLi4jRixAgdOnRIkhQaGurxWCWX3EraeZKUlKSpU6eW2Z6VlaXg4ODTPR0AAFAOl2XJx+Gwuwx4kV3vabQzX/XDiqq8X1SOps587dy50+4yJEm5ubkVbmtriCpJe/7+/nrvvfcUGRkpSerUqZPeeust/e1vf9MTTzyhESNGeKW/CRMmaNSoUe7HOTk5ioqKUlRUFHOiAACoZNM+2KyM/QV2lwEviGnUQJOubWVL35kr9mrLQa5E1RbnOAPVvHlzu8uQ9L+71CrC1hBVclXp4osvdgeoEq1bt1aLFi20detWHTx40N22vCtNJSdd3pUqSXI6nXI6nWW2+/j4yMen2nxlFgAAtdKO/Yf1yx/5dpcBL7DksO13J0sOucRVzdrCzrF0IpM6bK24ZcuWkqSwsDCP+0u2Hz582D0XqmRu1IlONWcKAAAAALzB1itRXbp0kST9/PPPZfYdPXpUW7duVWBgoMLDw9WsWTNFRkYqLS1N+fn5ZZY4T0tLU2xsLCvzAQAAAKhUtl6JiouLU48ePbR161bNnz+/1L4ZM2bo4MGD6tOnj/z8/ORwODRkyBDl5eVp+vTppdpOnz5deXl5Gjp0aFWWDwAAAKAOsvVKlCTNmTNH7du319ChQ/Xee++pVatWWrdunT7//HNFR0fr8ccfd7cdO3asli5dquTkZK1bt04XXnih1q5dq2XLlqldu3YaOXKkfScCAAAAoE6wfRZXXFycvv/+eyUmJuqHH37QM888o/T0dN19991as2aNmjVr5m4bGBioVatWaeTIkfr555/1xBNPaPPmzRo9erSWL1+ugIAAG88EAAAAQF1g+5UoSYqKitKCBQsq1DY0NFQpKSlKSUmp5KoAAAAAoCzbr0QBAAAAQE1CiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBge4iKiYmRw+Hw+O/yyy8v076wsFDTpk1TfHy86tevr8jISA0bNky7d++u+uIBAAAA1Dl+dhcgSaGhoRo5cmSZ7TExMaUeu1wuXX/99UpNTdVll12mG2+8Uenp6Zo/f76WL1+ub775RuHh4VVTNAAAAIA6qVqEqLCwME2ZMuWU7RYtWqTU1FT17dtXr776qhwOhyTpueee04gRI/Twww/r+eefr+RqAQAAANRltt/OZ+KFF16QJCUlJbkDlCQNHz5cLVq00KuvvqrDhw/bVR4AAACAOqBahKjCwkItXLhQjz32mGbPnq1vv/22TJsjR47o22+/VcuWLRUdHV1qn8Ph0BVXXKH8/Hx9//33VVU2AAAAgDqoWtzO9/vvv2vQoEGltrVr106vv/664uLiJEnbtm2Ty+VSfHy8x2OUbE9PT1enTp08tiksLFRhYaH7cU5OjqQ/51q5XK7TPg8AAFA+hyz5yLK7DHiBQ5ZtvzsxjmoXO8fSiUzqsD1EDRo0SJ06dVLr1q0VFBSkLVu26Mknn9TixYvVrVs3bdiwQcHBwTp06JCkPxeh8CQkJESS3O08SUpK0tSpU8tsz8rKUnBwsBfOBgBqF5dlyee426dR89n5nkY781U/rMiWvuFdTZ352rlzpy19M45qFzvH0olyc3Mr3Nb2EDV58uRSj9u2bauXX35ZkrR48WK98MILGjVqlFf6mjBhQqlj5eTkKCoqSlFRUe4QBgAobdoHm5Wxv8DuMuAFMY0aaNK1rWzrP3PFXm05yBWE2uAcZ6CaN29uS9+Mo9rFzrF0opK71CrC9hBVnuHDh2vx4sVKS0vTqFGj3FegyrvSVHLS5V2pkiSn0ymn01lmu4+Pj3x8qsX0MACodnbsP6xf/si3uwx4gSWHrT/vLDnkElc2awM7xxLjqHax+3PpeCZ1VI+KPTjjjDMkSfn5f/7gbtGihXx8fJSenu6xfcn28uZMAQAAAIA3VNsQVbJCX8kX7gYEBOiSSy7RL7/8oszMzFJtLcvSp59+qsDAQF188cVVXSoAAACAOsTWELV582YVFJS9z37z5s0aN26cJOm2225zbx82bJikP+c2Wdb/7oV9/vnntX37dvXr108BAQGVXDUAAACAuszWOVH//ve/9eSTT6pz586Kjo5WYGCgtmzZoo8++khHjx7VhAkT1LlzZ3f7gQMH6o033tDrr7+uHTt2KCEhQVu3btU777yj2NhYPfLIIzaeDQAAAIC6wNYQ1aVLF/38889at26dvvjiCxUUFOiMM87Q1Vdfrbvuuks9evQo1d7Hx0dLly7VjBkztHjxYqWkpKhRo0a644479Mgjjyg8PNymMwEAAABQV9gaohISEpSQkGD0HKfTqcmTJ5dZGh0AAAAAqkK1XVgCAAAAAKojQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAIABQhQAAAAAGCBEAQAAAICBahmikpOT5XA45HA49M0335TZn5OTo1GjRik6OlpOp1MxMTEaM2aM8vLybKgWAAAAQF1S7ULUxo0bNXnyZAUGBnrcn5+fr4SEBKWkpKhVq1Z64IEH1LJlS82aNUtdu3bVkSNHqrhiAAAAAHVJtQpRR48e1cCBA9W2bVv16dPHY5uZM2dq/fr1GjdunFJTUzVjxgylpqZq3Lhx+u6775SSklLFVQMAAACoS6pViHr00Ue1adMmvfTSS/L19S2z37IszZ8/X0FBQZo4cWKpfRMnTlRQUJDmz59fVeUCAAAAqIP87C6gxNq1a/Xoo49q2rRpOu+88zy2SU9PV3Z2tnr27Fnmdr/AwEB16NBBqampysrKUlRUVJnnFxYWqrCw0P04JydHkuRyueRyubx4NgBQezhkyUeW3WXACxyybP15x1iqPewcS4yj2sXuz6XjmdRRLUJUYWGhBgwYoLZt22rs2LHltktPT5ckxcfHe9wfHx+v1NRUpaenewxRSUlJmjp1apntWVlZCg4O/ovVA9WPy7Lk43DYXQa8yM73NNqZr/phRbb0De9q6szXzp07beufsVR72DmWGEe1i92fS8fLzc2tcNtqEaImTZqk9PR0/fDDDx5v4ytx6NAhSVJoaKjH/SEhIaXanWjChAkaNWqU+3FOTo6ioqIUFRXlfi5QW0z7YLMy9hfYXQa8IKZRA026tpVt/Weu2KstB/mrb21wjjNQzZs3t61/xlLtYedYYhzVLnZ/Lh2v5C61irA9RH399deaNWuWpkyZotatW1dqX06nU06ns8x2Hx8f+fhUq+lhwGnbsf+wfvkj3+4y4AWWHLZ+RllyyCWubNYGjCV4i51jiXFUu9j9uXQ8kzpsrbi4uFgDBw7UBRdcoPHjx5+yfckVqPKuNJWkx/KuVAEAAADA6bL1SlReXp57npO/v7/HNv/4xz8kSe+++657wYmS55zoVHOmAAAAAOB02RqinE6n7rjjDo/7Vq9erfT0dF133XUKDw9XTEyM4uPjFRkZqbS0NOXn55daoS8/P19paWmKjY31uKgEAAAAAHiDrSEqICCg3O91SkxMVHp6uiZMmKDLLrvMvX3IkCGaNm2apk+frhkzZri3T58+XXl5eXrooYcqvW4AAAAAdZftC0uYGjt2rJYuXark5GStW7dOF154odauXatly5apXbt2GjlypN0lAgAAAKjFqsdSGAYCAwO1atUqjRw5Uj///LOeeOIJbd68WaNHj9by5csVEBBgd4kAAAAAarFqeyVq4cKFWrhwocd9oaGhSklJUUpKStUWBQAAAKDOq3FXogAAAADAToQoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBQ4RCVnZ1dmXUAAAAAQI1Q4RB1/vnn67XXXqvMWgAAAACg2qtwiHr00Uc1fPhw3XTTTdq/f39l1gQAAAAA1VaFQ9Rdd92lH3/8Ufv27dN5552nDz74oDLrAgAAAIBqyc+kcWxsrD7//HPNnj1bN9xwg84991z5+ZU+xNq1a71aIAAAAABUJ0YhSpIyMzP1zjvvqGHDhrr++uvLhCgAAAAAqM2MEtALL7yg0aNHq3v37tq0aZPCw8Mrqy4AAAAAqJYqHKKuvPJKrVmzRrNnz9aAAQMqsyYAAAAAqLYqHKKOHTumH3/8UWeddVZl1gMAAAAA1VqFQ9Snn35amXUAAAAAQI1Q4SXOAQAAAACEKAAAAAAwQogCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwYGuIOnLkiEaNGqXOnTsrMjJS9evXV7NmzdShQwctWLBAR48eLfOcnJwcjRo1StHR0XI6nYqJidGYMWOUl5dnwxkAAAAAqGtsDVF5eXmaO3euHA6HrrnmGo0aNUp9+vTRrl27NHjwYPXq1Usul8vdPj8/XwkJCUpJSVGrVq30wAMPqGXLlpo1a5a6du2qI0eO2Hg2AAAAAOoCPzs7b9SokQ4dOiR/f/9S24uLi3XFFVdo2bJl+vjjj3XNNddIkmbOnKn169dr3LhxmjFjhrv9+PHjlZycrJSUFE2YMKFKzwEAAABA3WLrlSgfH58yAUqS/Pz81KdPH0nS1q1bJUmWZWn+/PkKCgrSxIkTS7WfOHGigoKCNH/+/MovGgAAAECdVi0XlnC5XPrkk08kSa1bt5YkpaenKzs7Wx06dFBgYGCp9oGBgerQoYO2b9+urKysKq8XAAAAQN1h6+18JYqKivTYY4/Jsizt27dPy5cv1+bNmzVo0CB169ZN0p8hSpLi4+M9HiM+Pl6pqalKT09XVFSUxzaFhYUqLCx0P87JyZH0Z2g7fu4VUBs4ZMlHlt1lwAscsmz9jGIs1R6MJXiLnWMptlGAHIyjWiOmUUC1+T3cpI5qE6KmTp3qfuxwOPTggw8qKSnJve3QoUOSpNDQUI/HCAkJKdXOk6SkpFL9lMjKylJwcPBfqt2bXJYlH4fD7jLgZXa9r9HOfNUPK6ryfuF9TZ352rlzp239M5ZqD8YSvMWuseSyLCX+rYGkBlXeNypPRmZmtfgdODc3t8Jtq0WICgoKkmX9+ReN7OxsffDBB3rooYf09ddf66OPPnIHpNM1YcIEjRo1yv04JydHUVFRioqK8lofp2vaB5uVsb/A7jLgJTGNGmjSta1s6TtzxV5tOchf6mqDc5yBat68uW39M5ZqD8YSvMXOsbTxtxwVFB2zpW94XwN/X7WOqB6/h5fcpVYR1SJElfDx8dFZZ52lESNG6IwzztDNN9+sRx99VMnJye4rUOVdaSo56fKuVEmS0+mU0+n02K+PT/WYHrZj/2H98ke+3WXASyw5bBtblhxyyf6/6uD02TmOSvpnLNUOjCV4i51jaXf+UR08XGxL3/C+sAC/avN7uEkd1aNiD3r06CFJWrlypaT/zYUqmRt1olPNmQIAAAAAb6i2ISo7O1uSVK9ePUl/hqPIyEilpaUpP7/0lZr8/HylpaUpNja23EUlAAAAAMAbbA1RP/30kwoKys7/KSgocM9duvrqqyX9udjEkCFDlJeXp+nTp5dqP336dOXl5Wno0KGVXzQAAACAOs3WOVFvvvmmnnzySXXs2FExMTEKCQnRrl279PHHH2vfvn3q1KmTHnjgAXf7sWPHaunSpUpOTta6det04YUXau3atVq2bJnatWunkSNH2ncyAAAAAOoEW0NUr169lJ2dra+++kpff/218vLyFBoaqgsuuEC33nqrBg8eLD+//5UYGBioVatWacqUKVqyZIlWrFihiIgIjR49WpMnT1ZAQICNZwMAAACgLrA1RF188cW6+OKLjZ4TGhqqlJQUpaSkVFJVAAAAAFC+aruwBAAAAABUR4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADDgZ3cBAACgbohp3MDuEuAlvJeo6whRAACg0h1zWZp63bl2lwEvOuay5OvjsLsMwBaEKAAAUOl8fRza8FuOCoqO2V0KvKCBv6/aRITYXQZgG0IUAACoEn/kFurg4WK7y4AXhAX4qU2E3VUA9mFhCQAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwQIgCAAAAAAOEKAAAAAAwYGuI2rVrl5566in16NFDzZs3l7+/v5o1a6Ybb7xR3377rcfn5OTkaNSoUYqOjpbT6VRMTIzGjBmjvLy8Kq4eAAAAQF1ka4j617/+pQceeEDbt29Xjx49NHr0aHXs2FFLly5V+/bt9cYbb5Rqn5+fr4SEBKWkpKhVq1Z64IEH1LJlS82aNUtdu3bVkSNHbDoTAAAAAHWFn52dX3LJJVq5cqUSEhJKbf/iiy/UrVs3jRgxQr1795bT6ZQkzZw5U+vXr9e4ceM0Y8YMd/vx48crOTlZKSkpmjBhQpWeAwAAAIC6xdYrUTfccEOZACVJnTp1UpcuXXTgwAFt2LBBkmRZlubPn6+goCBNnDixVPuJEycqKChI8+fPr5K6AQAAANRd1XZhiXr16kmS/Pz+vFiWnp6u7OxsdejQQYGBgaXaBgYGqkOHDtq+fbuysrKqvFYAAAAAdYett/OVZ+fOnfrss88UERGhNm3aSPozRElSfHy8x+fEx8crNTVV6enpioqK8timsLBQhYWF7sc5OTmSJJfLJZfL5c1T+MscsuQjy+4y4CUOWbaNLcZS7WHnOJKk2EYBcjCWaoWYRgG2jiXLcklW9fh5i9NjWfb97sQ4ql3sHEsnMqmj2oWoo0ePqn///iosLFRycrJ8fX0lSYcOHZIkhYaGenxeSEhIqXaeJCUlaerUqWW2Z2VlKTg4+HRL94poZ77qhxXZXQa8pKkzXzt37rSlb8ZS7WHnOHJZlhL/1kBSA1v6h/dlZGbKx+Gwpe/ig7lyHK0evyzh9BQX+mjnzsP29M04qlXsHEsnys3NrXDbahWiXC6XEhMTtXr1ag0dOlT9+/f36vEnTJigUaNGuR/n5OQoKipKUVFR7hBmt8wVe7XlIH/xrS3OcQaqefPmtvTNWKo97BxHkrTxtxwVFB2zrX94TwN/X7WOsO/nXXrhHlmHi23rH97jF+Cn5s3DbembcVS72DmWTlRyl1pFVJsQ5XK5NHjwYL322mu6/fbb9dxzz5XaX3IFqrwrTSUnXd6VKklyOp3ulf6O5+PjIx+f6jE9zJJDLtnzF0J4nyWHbWOLsVR72DmOJGl3/lEd5BeWWiEswM/WseRw+EiO6vHzFqfH4bDvdyfGUe1i51g6kUkd1SJEuVwuDRo0SC+//LL69u2rhQsXljmJkrlQJXOjTnSqOVMAAAAA4A22x77jA9Qtt9yixYsXu+dBHS8+Pl6RkZFKS0tTfn5+qX35+flKS0tTbGxsuYtKAAAAAIA32BqiSm7he/nll3XTTTfplVde8RigJMnhcGjIkCHKy8vT9OnTS+2bPn268vLyNHTo0KooGwAAAEAdZuvtfNOmTdOiRYsUFBSkc845R4888kiZNr1791bbtm0lSWPHjtXSpUuVnJysdevW6cILL9TatWu1bNkytWvXTiNHjqzaEwAAAABQ59gaojIyMiRJeXl5evTRRz22iYmJcYeowMBArVq1SlOmTNGSJUu0YsUKRUREaPTo0Zo8ebICAgKqqHIAAAAAdZWtIWrhwoVauHCh0XNCQ0OVkpKilJSUyikKAAAAAE7C9oUlAAAAAKAmIUQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAE/uwsAUDliGjewuwR4Ce8lAADVCyEKqIWOuSxNve5cu8uAFx1zWfL1cdhdBgAAECEKqJV8fRza8FuOCoqO2V0KvKCBv6/aRITYXQYAAPg/hCiglvojt1AHDxfbXQa8ICzAT20i7K4CAACUYGEJAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBAiAIAAAAAA4QoAAAAADBge4h65ZVXNHz4cF188cVyOp1yOBxauHBhue1zcnI0atQoRUdHy+l0KiYmRmPGjFFeXl7VFQ0AAACgzvKzu4CHH35YmZmZOuOMMxQREaHMzMxy2+bn5yshIUHr169Xjx491LdvX61bt06zZs3SqlWrtHr1atWvX78KqwcAAABQ19h+JWr+/PnKyMjQnj17dOedd5607cyZM7V+/XqNGzdOqampmjFjhlJTUzVu3Dh99913SklJqaKqAQAAANRVtoeo7t27Kzo6+pTtLMvS/PnzFRQUpIkTJ5baN3HiRAUFBWn+/PmVVSYAAAAASKoGIaqi0tPTlZ2drQ4dOigwMLDUvsDAQHXo0EHbt29XVlaWTRUCAAAAqAtsnxNVUenp6ZKk+Ph4j/vj4+OVmpqq9PR0RUVFeWxTWFiowsJC9+OcnBxJksvlksvl8nLFf01sowA5ZNldBrwkplGAbWPLslySVT3GNU6PZdn7GcVYqj0YS/AWO8cS46h2sftz6XgmddSYEHXo0CFJUmhoqMf9ISEhpdp5kpSUpKlTp5bZnpWVpeDgYC9UeXpclqXEvzWQ1MDuUuBFGZmZ8nE4qrzf4oO5chytHh9KOD3FhT7aufOwff0zlmoNxhK8xc6xxDiqXez+XDpebm5uhdvWmBDlDRMmTNCoUaPcj3NychQVFaWoqCh3CLPbxt9yVFB0zO4y4CUN/H3VOsKesZVeuEfW4WJb+oZ3+QX4qXnzcNv6ZyzVHowleIudY4lxVLvY/bl0vJK71CqixoSokitQ5V1pKjnp8q5USZLT6ZTT6Syz3cfHRz4+1WN62O78ozrIB0OtERbgZ9vYcjh8JEf1GNc4PQ6HvZ9RjKXag7EEb7FzLDGOahe7P5eOZ1JH9ai4AkrmQpXMjTrRqeZMAQAAAIA31KgQFRkZqbS0NOXn55fal5+fr7S0NMXGxpa7qAQAAAAAeEONCVEOh0NDhgxRXl6epk+fXmrf9OnTlZeXp6FDh9pUHQAAAIC6wvY5UfPnz9eXX34pSdqwYYN728qVKyVJHTt21JAhQyRJY8eO1dKlS5WcnKx169bpwgsv1Nq1a7Vs2TK1a9dOI0eOtOMUAAAAANQhtoeoL7/8UosWLSq1LS0tTWlpae7HJSEqMDBQq1at0pQpU7RkyRKtWLFCERERGj16tCZPnqyAgIAqrR0AAABA3WN7iFq4cKEWLlxY4fahoaFKSUlRSkpK5RUFAAAAAOWoMXOiAAAAAKA6IEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYIEQBAAAAgAFCFAAAAAAYqJEh6rvvvtPVV1+tsLAwBQYG6rLLLtObb75pd1kAAAAA6gA/uwswtWLFCvXs2VP169fXrbfequDgYC1ZskS33HKLsrKyNHr0aLtLBAAAAFCL1agrUcXFxRo6dKh8fHy0evVqzZs3T0888YT++9//6pxzztFDDz2kzMxMu8sEAAAAUIvVqBD1+eefa9u2bbrtttvUtm1b9/bQ0FA99NBDKioq0qJFi+wrEAAAAECtV6NC1MqVKyVJPXr0KLOvZ8+ekqRVq1ZVZUkAAAAA6pgaNScqPT1dkhQfH19mX7NmzRQUFORu40lhYaEKCwvdjw8dOiRJOnjwoFwul5er/WscR/LlV1xsdxnwEscRPx08WM+mvhlLtYWd4+jP/hlLtQVjCd7Czzd4i92fS8fLycmRJFmWdcq2NSpElYSe0NBQj/tDQkLcbTxJSkrS1KlTy2yPjo72ToEAAAAAarTc3Nxy80aJGhWiTteECRM0atQo92OXy6X9+/ercePGcjgcNlZWt+Tk5CgqKkpZWVkKCQmxuxzUYIwleAtjCd7AOIK3MJbsYVmWcnNzFRkZecq2NSpElSTC8q425eTkqGHDhuU+3+l0yul0ltoWFhbmtfpgJiQkhA8GeAVjCd7CWII3MI7gLYylqneqK1AlatTCEiVzoTzNe/r999+Vl5fncb4UAAAAAHhLjQpRCQkJkqRly5aV2ZeamlqqDQAAAABUhhoVorp166YWLVrotdde0/r1693bDx06pMcee0z+/v4aMGCAfQWiQpxOpyZPnlzm1krAFGMJ3sJYgjcwjuAtjKXqz2FVZA2/amTFihXq2bOn6tevr1tvvVXBwcFasmSJMjMzNWvWLI0ePdruEgEAAADUYjUuREnSmjVrNHnyZH311Vc6evSo2rRpo1GjRumWW26xuzQAAAAAtVyNDFEAAAAAYJcaNScKAAAAAOxGiAIAAAAAA4QoVKnBgwfL4XCocePGKiwstLsc1DAZGRlyOBxl/gUGBuqCCy7Q1KlTlZeXZ3eZqEF++OEH3XHHHYqPj1dgYKACAgIUFxen/v3769NPP7W7PFRznj6TGjRooMjISHXr1k2TJk3Stm3b7C4TNUR5P+OO/xcTE2N3mfg/zIlClcnNzVVERIQKCgpkWZb+/e9/sxgIjGRkZCg2NlZxcXG6/fbbJUmWZWnPnj36+OOPlZGRocsuu0xffvmlfH19ba4W1ZnL5dKDDz6olJQU+fn5qWvXrmrdurXq1aun7du367PPPtOBAwc0bdo0TZw40e5yUU15+kwqLCzU7t27tWbNGm3cuFG+vr4aO3asHn30UTkcDpsrRnXmaTydKCwsTCNHjqzawuCRn90FoO544403lJ+fr1GjRumpp57Siy++SIjCX3L22WdrypQppbYVFhbqH//4h7755hutWrVKXbt2tac41AgPP/ywUlJS1LZtW7399tuKi4srtf/w4cOaPXu29u3bZ1OFqEk8fSZJ0pdffqn+/fsrKSlJvr6+mj59etUXhxqnvPGE6oXb+VBlXnzxRfn5+Wns2LHq0qWLli9frszMTLvLQi3hdDrVpUsXSdLevXttrgbV2datWzVz5kw1btxYn3zySZkAJUkBAQEaM2aMpk6dakOFqC06duyoTz75RE6nUzNnzlRWVpbdJQHwEkIUqsRPP/2kb775Rj169FDTpk01YMAAuVwuLViwwO7SUEsUFRVp5cqVcjgcatu2rd3loBpbuHChjh07puHDh6tp06Ynbet0OquoKtRWLVu21M0336yioiK99957dpcDwEu4nQ9V4sUXX5Qk9e/fX5J0ww036K677tKCBQs0adIk+fiQ51FxW7dudd/qYFmW9u7dq9TUVO3atUszZ87UOeecY2+BqNbS0tIkiVs+UWUuv/xyLV68WN99953dpaAGOP5n3Ikuu+wyXXnllVVbEDwiRKHSHT16VIsXL1ZISIh69+4tSQoKClKfPn30yiuv6LPPPlOPHj3sLRI1yrZt2zzeZtWrVy91797dhopQk/z++++SpLPOOsvmSlBXREZGSuJWY1RMeT/jJOn+++8nRFUT/PkflW7p0qXas2ePbrrpJtWvX9+9fcCAAZL+d5UKqKiePXvKsiz3v71792rp0qXauHGjOnTooG+//dbuEgEA+EtO/Bl3/L+nnnrK7vLwfwhRqHQlIakkNJXo1q2bzjzzTC1dulT79++3ozTUEo0bN9Z1112nF154QQUFBXr44YftLgnVWLNmzSRJu3btsrkS1BXZ2dmSpPDwcJsrAeAthChUqqysLC1btkySlJCQUOoL43x9fbVr1y4VFhbqlVdesblS1AaXXnqpJDHvACfVoUMHSdLy5cttrgR1xcqVKyVJ7dq1s7cQAF7DnChUqoULF8rlcqljx45q2bJlmf3FxcVatGiRXnzxRd133302VIja5MCBA5L+/CJVoDyJiYmaMWOG5s2bp5EjR5706kBhYSEr9OG0bNmyRW+++aacTqf69OljdzkAvIQQhUpjWZYWLFggh8OhRYsWqUWLFh7bbdmyRV9//bW+//57XXzxxVVcJWqTJ598UpLUuXNnmytBdXb22Wdr7NixSkpK0lVXXaW33npLsbGxpdocOXJEc+bM0Z49e5SUlGRTpajp0tLSdPvtt6uwsFCTJ0/WmWeeaXdJALyEEIVK8/nnn2vHjh1KSEgoN0BJ0qBBg/T111/rxRdfJEShQk5c/nX//v1KS0vT2rVr1bBhQyUnJ9tXHGqERx55REeOHFFKSopatmyprl27qnXr1qpXr5527Nihzz77TPv27dMjjzxid6moAY7/TCoqKtLu3bu1Zs0abdiwQb6+vnr44Yc1efJke4tEjXGyJc4lafz48aUW6oI9HJZlWXYXgdrptttu0+uvv64FCxYoMTGx3HY5OTlq1qyZ/P399dtvvykgIKDqikSNkpGRUeaKgfTnF6KeddZZ6tGjh8aPH6/mzZvbUB1qou+//15z587V6tWrtWvXLrlcLkVERKh9+/YaNGgQS+bjpDx9JgUEBCgsLEytWrVSx44dNXDgQMXFxdlUIWqS8n7GnejAgQMKCwur/IJwUoQoAAAAADDA6nwAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAgFrv2LFjat++vW644YZS2w8dOqSoqCj985//lCTdd999uuiii+R0OtW2bVsbKgUA1ASEKABArefr66uFCxfqk08+0auvvurefu+996pRo0aaPHmye9vgwYN1yy232FEmAKCG8LO7AAAAqsI555yjGTNm6N5771XXrl21Zs0a/fvf/9Z3330nf39/SdIzzzwjSdqzZ49+/PFHO8sFAFRjhCgAQJ1x77336t1331X//v21YcMGTZo0SX/729/sLgsAUMMQogAAdYbD4dDcuXN17rnnqk2bNho/frzdJQEAaiDmRAEA6pSXXnpJDRo00I4dO/Trr7/aXQ4AoAYiRAEA6oyvvvpKKSkp+s9//qNLLrlEd9xxhyzLsrssAEANQ4gCANQJBQUFSkxM1IgRI9SlSxe9+OKLWrNmjZ577jm7SwMA1DCEKABAnTBhwgRZlqUZM2ZIkmJiYjRr1iyNHTtWGRkZkqStW7dq/fr1+v3333X48GGtX79e69evV1FRkY2VAwCqG4fFfQwAgFpu1apV6tatm1auXKmOHTuW2tezZ08VFxfrs88+U5cuXbRq1aoyz9+xY4diYmKqqFoAQHVHiAIAAAAAA9zOBwAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAGCFEAAAAAYIAQBQAAAAAG/j+Bbsc5cxhiTQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<Figure size 1000x800 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import matplotlib\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "import seaborn as sns\n",
+    "\n",
+    "# Create data\n",
+    "X1 = ['A', 'B', 'C', 'D', 'E']\n",
+    "Y1 = [10, 15, 20, 25, 30]\n",
+    "Y2 = [20, 25, 30, 35, 40]\n",
+    "\n",
+    "# Set color scheme and font\n",
+    "colors = sns.color_palette(\"Blues\", 2)\n",
+    "\n",
+    "# Create bar plot\n",
+    "fig, ax = plt.subplots(figsize=(10, 8))\n",
+    "ax.bar(X1, Y1, color=colors[0], edgecolor='white', linewidth=1)\n",
+    "ax.bar(X1, Y2, bottom=Y1, color=colors[1], edgecolor='white', linewidth=1)\n",
+    "\n",
+    "# Set axis labels and ticks\n",
+    "ax.set_xlabel('X1')\n",
+    "ax.set_ylabel('Y')\n",
+    "ax.set_xticklabels(X1)\n",
+    "ax.tick_params(axis='both', which='major', labelsize=14)\n",
+    "ax.grid(axis='y', alpha=0.4)\n",
+    "\n",
+    "# Add legend and title\n",
+    "ax.legend(['Y1', 'Y2'], loc='upper left')\n",
+    "ax.set_title('Vertical stacked bar graph of Y1 and Y2')\n",
+    "\n",
+    "# Save the graph as PDF or PNG\n",
+    "plt.savefig('vertical_stacked_bar_graph.png', dpi=300, bbox_inches='tight')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4c055ec7",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

hot_analysis/create_part_probability_mesh.py

@@ -0,0 +1,86 @@
+import numpy as np
+import os
+import json
+import trimesh
+import seaborn as sns
+
+
+# Load the combined dca train, val and test npzs
+dir = '/is/cluster/work/stripathi/pycharm_remote/dca_contact/data/dataset_extras'
+trainval_npz = np.load(os.path.join(dir, 'hot_dca_trainval.npz'), allow_pickle=True)
+test_npz = np.load(os.path.join(dir, 'hot_dca_test.npz'), allow_pickle=True)
+
+# combine the two npz
+combined_npz = {}
+for key in trainval_npz.keys():
+    combined_npz[key] = np.concatenate([trainval_npz[key], test_npz[key]], axis=0)
+
+segmentation_path = 'data/smpl_vert_segmentation.json'
+with open(segmentation_path, 'rb') as f:
+    part_segmentation = json.load(f)
+
+combine_keys = {'leftFoot': ['leftToeBase'],
+                'rightFoot': ['rightToeBase'],
+                'leftHand': ['leftHandIndex1'],
+                'rightHand': ['rightHandIndex1'],
+                'spine': ['spine1', 'spine2'],
+                'head': ['neck'],}
+
+for key in combine_keys:
+    for subkey in combine_keys[key]:
+        part_segmentation[key] += part_segmentation[subkey]
+        del part_segmentation[subkey]
+
+# reverse the part segmentation
+part_segmentation_rev = {}
+for part in part_segmentation:
+    for vert in part_segmentation[part]:
+        part_segmentation_rev[vert] = part
+
+# count the number of contact instances per vertex
+per_vert_contact_count = np.zeros(6890)
+for cls in combined_npz['contact_label']:
+    per_vert_contact_count += cls
+
+# calculate the maximum contact count per part
+part_contact_max = {}
+for part in part_segmentation:
+    part_contact_max[part] = np.max(per_vert_contact_count[part_segmentation[part]])
+
+# calculate the contact probability globally
+contact_prob = np.zeros(6890)
+for vid in range(6890):
+    contact_prob[vid] = (per_vert_contact_count[vid] / max(per_vert_contact_count)) ** 0.3
+
+# save the contact probability mesh
+outdir = "/is/cluster/work/stripathi/pycharm_remote/dca_contact/hot_analysis"
+
+# load template smpl mesh
+mesh = trimesh.load_mesh('data/smpl/smpl_neutral_tpose.ply')
+vertex_colors = trimesh.visual.interpolate(contact_prob, 'jet')
+# set the vertex colors of the mesh
+mesh.visual.vertex_colors = vertex_colors
+# save the mesh
+out_path = os.path.join(outdir, "contact_probability_mesh.obj")
+mesh.export(out_path)
+
+# # calculate the contact probability per part
+# contact_prob = np.zeros(6890)
+# for vid in range(6890):
+#     if 'Hand' in part_segmentation_rev[vid]:
+#     contact_prob[vid] = (per_vert_contact_count[vid] / part_contact_max[part_segmentation_rev[vid]]) ** 0.4 if 'Hand' not in part_segmentation_rev[vid] else (per_vert_contact_count[vid] / part_contact_max[part_segmentation_rev[vid]]) ** 0.8
+#
+# # save the contact probability mesh
+# outdir = "/is/cluster/work/stripathi/pycharm_remote/dca_contact/hot_analysis"
+#
+# # load template smpl mesh
+# mesh = trimesh.load_mesh('data/smpl/smpl_neutral_tpose.ply')
+# vertex_colors = trimesh.visual.interpolate(contact_prob, 'jet')
+# # set the vertex colors of the mesh
+# mesh.visual.vertex_colors = vertex_colors
+# # save the mesh
+# out_path = os.path.join(outdir, "contact_probability_mesh_part.obj")
+# mesh.export(out_path)
+
+
+

hot_analysis/damon_qc_stats/compute_accuracy_iou_damon.py

@@ -0,0 +1,59 @@
+# load amt csv, go through each line in vertices, combine the vertices for each object label and then compute the iou with GT from RICH and PROX
+import pandas as pd
+import numpy as np
+
+# load csv
+csv_path = './quality_assurance_accuracy.csv'
+df = pd.read_csv(csv_path)
+
+# load gt npz
+gt_path = './qa_accuracy_gt_contact_combined.npz'
+gt = np.load(gt_path)
+
+def compute_iou(pred_verts, gt_verts):
+    if len(pred_verts) != 0:
+        intersect = list(set(pred_verts) & set(gt_verts))
+        iou = len(intersect) / (len(pred_verts) + len(gt_verts) - len(intersect))
+    else:
+        iou = 0
+    return iou
+
+all_ious = []
+# for loop each row in df
+for index, row in df.iterrows():
+    combined_annotation_ids = []
+    imgname = []
+    # get vertices
+    annotation_dict = eval(row['vertices'])
+    worker_id = row['WorkerId']
+    # single for loop in the dictionary
+    for im, anno in annotation_dict.items():
+        imgname.append(im)
+        for ann in anno:
+            # single for loop in the dict
+            for k, v in ann.items():
+                combined_annotation_ids.extend(v)
+    # remove repeated values
+    combined_annotation_ids = list(set(combined_annotation_ids))
+
+    assert len(imgname) == 1
+    imgname = imgname[0]
+
+    # get gt for the imgname
+    gt_ids = gt[imgname]
+    if 'prox' in imgname:
+        continue
+
+    # compute iou
+    iou = compute_iou(combined_annotation_ids, gt_ids)
+    print('worker id: ', worker_id, 'imgname: ', imgname, 'iou: ', iou)
+    all_ious.append(iou)
+
+# compute mean iou
+mean_iou = np.mean(all_ious)
+print('mean iou: ', mean_iou)
+
+
+
+
+

hot_analysis/damon_qc_stats/compute_fleiss_kappa_damon.py

@@ -0,0 +1,111 @@
+import numpy as np
+import pandas as pd
+from scipy.stats import chi2
+
+
+def correct_supporting(vertices):
+    # Copy vertices from objects to supporting since the dropdown option was missing in the QA app
+    def add_supporting(verts, object_name):
+        temp_supporting_vids = []
+        supporting_vid = -1
+        for id, v_i in enumerate(verts):
+            # single key dict
+            for k, v in v_i.items():
+                if k == object_name:
+                    temp_supporting_vids = v
+                if k == 'SUPPORTING':
+                    supporting_vid = id
+        if supporting_vid != -1:
+            # append to supporting
+            verts[supporting_vid]['SUPPORTING'] += temp_supporting_vids
+        return verts
+
+    # correct supporting contacts
+    for i, vert in enumerate(vertices):
+        for k, v in vert.items():
+            if k == 'hot/training/hake_train2015_HICO_train2015_00000019.jpg':
+                # copy bicycle contacts to supporting
+                v = add_supporting(v, 'BICYCLE')
+            if k == 'hot/training/hake_train2015_HICO_train2015_00000020.jpg':
+                # copy skateboard contacts to supporting
+                v = add_supporting(v, 'SKATEBOARD')
+            if k == 'hot/training/hake_train2015_HICO_train2015_00000942':
+                # copy bench contacts to supporting
+                v = add_supporting(v, 'BENCH')
+
+            # combine all vert_ids into a single list no matter the object
+            v = {ki: vi for d in v for ki, vi in d.items()}
+            v = [vi for k, vi in v.items()]
+            v = [item for sublist in v for item in sublist]
+            v = list(set(v))
+            # binarize the list to a numpy array
+            v_np = np.zeros(6890)
+            v_np[v] = 1
+            vert[k] = v_np
+        vertices[i] = vert
+    return vertices
+
+def fleiss_kappa_per_img(vertices):
+    """
+    Compute Fleiss' kappa per imagename
+    Parameters
+    ----------
+    vertices : list of np arrays where each array is of shape (6890,) and 1 indicates a vertex is selected
+    """
+    n = len(vertices)  # number of raters
+    N = 6890  # number of images
+    k = 2  # number of categories
+
+    # compute the observed agreement
+    M = np.zeros((N, k))
+
+    for i in range(k):
+        M[:, i] = np.sum(vertices == i, axis=0)
+
+    assert np.sum(M) == N * n
+
+    # compute the expected agreement
+    p = np.sum(M, axis=0) / (N * n)
+    P = (np.sum(M * M, axis=1) - n) / (n * (n - 1))
+    Pbar = np.mean(P)
+    PbarE = np.sum(p * p)
+
+    # compute Fleiss' kappa
+    kappa = (Pbar - PbarE) / (1 - PbarE)
+    return kappa
+
+def fleiss_kappa(data):
+    """
+    Compute Fleiss' kappa per imagename
+    Parameters
+    ----------
+    data : list of dicts where keys are imgnames
+    """
+    imgnames = sorted(data[0].keys())
+    kappas = []
+    for img in imgnames:
+        kappa_data = []
+        for d in data:
+            kappa_data.append(d[img])
+        kappa_data = np.array(kappa_data)
+        kappa_img = fleiss_kappa_per_img(kappa_data)
+        print(f'Fleiss\' Kappa for {img}: {kappa_img}')
+        kappas.append(kappa_img)
+
+    # computer mean kappa
+    kappa = np.mean(kappas)
+    return kappa
+
+
+# Load the combined qa csv file
+csv_file = 'quality_assurance_fleiss.csv'
+df = pd.read_csv(csv_file)
+
+vertices = df['vertices'].values
+vertices = [eval(v) for v in vertices]
+
+vertices = correct_supporting(vertices)
+
+kappa = fleiss_kappa(vertices)
+
+print('Fleiss\' Kappa:', kappa)

hot_analysis/damon_qc_stats/qa_accuracy_gt_contact_combined.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b34393328dae058aa12245f547ee954f48508bf0c4ed7ecbe40a674245ba5717
+size 456742