Upload 1031 files

README.md

@@ -0,0 +1,108 @@
+ <div align="center">
+ 
+## Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation
+<div align="center">
+    <img src="./docs/pipeline.png" width="100%">
+</div>
+
+</div>
+
+<div align="center">
+<a href="">Mohamed El Amine Boudjoghra</a><sup>1</sup>, <a href="">Angela Dai</a><sup>2</sup>, <a href=""> Jean Lahoud</a><sup>1</sup>, <a href="">Hisham Cholakkal</a><sup>1</sup>, <a href="">Rao Muhammad Anwer</a><sup>1,3</sup>,  <a href="">Salman Khan</a><sup>1,4</sup>, <a href="">Fahad Khan</a><sup>1,5</sup>
+
+<sup>1</sup>Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) <sup>2</sup>Technical University of Munich (TUM) <sup>3</sup>Aalto University <sup>4</sup>Australian National University <sup>5</sup>Linköping University
+</div>
+
+
+<div align="center">
+ 
+<a href='https://arxiv.org/abs/2406.02548' target="_blank">![paper](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)</a> 
+
+
+ </div>
+
+
+
+### News
+
+* **30 May 2024**: [Open-YOLO 3D](https://arxiv.org/abs/2406.02548) released on arXiv. 📝
+* **30 May 2024**: Code released. 💻
+
+### Abstract
+
+ Recent works on open-vocabulary 3D instance segmentation show strong promise, but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on 3D clip features, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a consequence, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a fast yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that effectively leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. 
+ We address this task by generating class-agnostic 3D masks for objects in the scene and associating them with text prompts.
+ We observe that the projection of class-agnostic 3D point cloud instances already holds instance information; thus, using SAM might only result in redundancy that unnecessarily increases the inference time.
+We empirically find that a better performance of matching text prompts to 3D masks can be achieved in a faster fashion with a 2D object detector.  We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, 
+ under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network. Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to 16x speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene.
+
+### Qualitative results
+<br>
+
+<div align="center">
+    <img src="./docs/qualitatives.png" width="100%">
+</div>
+
+
+## Installation guide
+
+Kindly check [Installation guide](./docs/Installation.md) on how to setup the Conda environment and to download the checkpoints, the pre-computed class agnostic masks, and the ground truth masks.
+
+## Data Preparation
+
+Kindly check [Data Preparation guide](./docs/Data_prep.md) on how to prepare ScanNet200 and Replica datasets.
+
+## Results reproducibility
+
+Kindly use the pre-computed class agnostic masks we shared to reproduce the exact numbers we reported in the paper.
+
+**Reproduce the results of ScanNet200 with precomputed-masks (using Mask3D)**
+```
+python run_evaluation.py --dataset_name scannet200 --path_to_3d_masks "./output/scannet200/scannet200_masks"
+```
+**Reproduce the results of ScanNet200 with oracle 3D masks (ground truth 3D masks)**
+```
+python run_evaluation.py --dataset_name scannet200 --path_to_3d_masks "./output/scannet200/scannet200_ground_truth_masks" --is_gt
+```
+**Reproduce the results of Replica with precomputed-masks (using Mask3D)**
+```
+python run_evaluation.py --dataset_name replica --path_to_3d_masks "./output/replica/replica_masks"
+```
+**Reproduce the results of Replica with oracle 3D masks (ground truth 3D masks)**
+```
+python run_evaluation.py --dataset_name replica --path_to_3d_masks "./output/replica/replica_ground_truth_masks" --is_gt
+```
+
+You can evaluate without our 3D class-agnostic masks, but this may lead to variability in results due to elements like furthest point sampling that cause randomness in predictions from Mask3D. For consistent results with the ones we report in the paper, we recommend using our pre-computed masks. 
+
+**Reproduce the results of Replica or ScanNet200 without using our pre-computed masks**
+```
+python run_evaluation.py --dataset_name $DATASET_NAME
+```
+
+## Single scene inference
+
+```
+from utils import OpenYolo3D
+
+openyolo3d = OpenYolo3D("$(pwd)/pretrained/config.yaml") #Initialize the model, define the text prompts in the config.
+prediction = openyolo3d.predict("$(pwd)/data/replica/office0", 6553.5) #Predict the instance masks and labels (takes around 20 seconds in total).
+openyolo3d.save_output_as_ply("$(pwd)/sample/output.ply", True) # Save the ply file for visualization, you can use meshlab to visualize the output scene
+```
+
+## Acknoledgments
+We would like to thank the authors of <a href="https://github.com/cvg/Mask3D">Mask3D</a> and <a href="https://github.com/AILab-CVC/YOLO-World">YoloWorld</a> for their works which were used for our model.
+</div>
+
+## BibTeX :pray:
+```
+@misc{boudjoghra2024openyolo,
+      title={Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation}, 
+      author={Mohamed El Amine Boudjoghra and Angela Dai and Jean Lahoud and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan and Fahad Shahbaz Khan},
+      year={2024},
+      eprint={2406.02548},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+

environment.yml

@@ -0,0 +1,216 @@
+name: openyolo3d
+channels:
+  - anaconda
+  - defaults
+dependencies:
+  - _libgcc_mutex=0.1=main
+  - _openmp_mutex=5.1=1_gnu
+  - blas=1.0=openblas
+  - boltons=23.0.0=py310h06a4308_0
+  - brotlipy=0.7.0=py310h7f8727e_1002
+  - bzip2=1.0.8=h7b6447c_0
+  - ca-certificates=2023.01.10=h06a4308_0
+  - certifi=2022.12.7=py310h06a4308_0
+  - cffi=1.15.1=py310h5eee18b_3
+  - charset-normalizer=2.0.4=pyhd3eb1b0_0
+  - conda=23.3.1=py310h06a4308_0
+  - conda-content-trust=0.1.3=py310h06a4308_0
+  - conda-package-handling=2.0.2=py310h06a4308_0
+  - conda-package-streaming=0.7.0=py310h06a4308_0
+  - cryptography=39.0.1=py310h9ce1e76_0
+  - idna=3.4=py310h06a4308_0
+  - jsonpatch=1.32=pyhd3eb1b0_0
+  - jsonpointer=2.1=pyhd3eb1b0_0
+  - ld_impl_linux-64=2.38=h1181459_1
+  - libffi=3.4.2=h6a678d5_6
+  - libgcc-ng=11.2.0=h1234567_1
+  - libgfortran-ng=11.2.0=h00389a5_1
+  - libgfortran5=11.2.0=h1234567_1
+  - libgomp=11.2.0=h1234567_1
+  - libopenblas=0.3.21=h043d6bf_0
+  - libstdcxx-ng=11.2.0=h1234567_1
+  - libuuid=1.41.5=h5eee18b_0
+  - ncurses=6.4=h6a678d5_0
+  - nomkl=3.0=0
+  - openblas-devel=0.3.21=h06a4308_0
+  - openssl=1.1.1s=h7f8727e_0
+  - packaging=23.0=py310h06a4308_0
+  - pluggy=1.0.0=py310h06a4308_1
+  - pycosat=0.6.4=py310h5eee18b_0
+  - pycparser=2.21=pyhd3eb1b0_0
+  - pyopenssl=23.0.0=py310h06a4308_0
+  - pysocks=1.7.1=py310h06a4308_0
+  - python=3.10.9=h7a1cb2a_0
+  - readline=8.2=h5eee18b_0
+  - requests=2.28.1=py310h06a4308_1
+  - ruamel.yaml=0.17.21=py310h5eee18b_0
+  - ruamel.yaml.clib=0.2.6=py310h5eee18b_1
+  - setuptools=65.6.3=py310h06a4308_0
+  - six=1.16.0=pyhd3eb1b0_1
+  - sqlite=3.41.2=h5eee18b_0
+  - tk=8.6.12=h1ccaba5_0
+  - toolz=0.12.0=py310h06a4308_0
+  - tqdm=4.65.0=py310h2f386ee_0
+  - urllib3=1.26.15=py310h06a4308_0
+  - wheel=0.37.1=pyhd3eb1b0_0
+  - xz=5.2.10=h5eee18b_1
+  - zlib=1.2.13=h5eee18b_0
+  - zstandard=0.19.0=py310h5eee18b_0
+  - pip
+  - pip:
+      - absl-py==1.4.0
+      - addict==2.4.0
+      - aiohttp==3.8.4
+      - aiosignal==1.3.1
+      # - albumentations==1.2.1 #manual
+      - antlr4-python3-runtime==4.8
+      - anyio==3.6.2
+      - appdirs==1.4.4
+      - asttokens==2.2.1
+      - async-timeout==4.0.2
+      - attrs==23.1.0
+      - backcall==0.2.0
+      - black==21.4b2
+      - cachetools==5.3.0
+      - click==8.1.3
+      - cloudpickle==2.1.0
+      - comm==0.1.3
+      - configargparse==1.5.3
+      - contourpy==1.0.7
+      - cycler==0.11.0
+      - dash==2.9.3
+      - dash-core-components==2.0.0
+      - dash-html-components==2.0.0
+      - dash-table==5.0.0
+      - debugpy==1.6.7
+      - decorator==5.1.1
+      # - detectron2==0.6
+      - docker-pycreds==0.4.0
+      - executing==1.2.0
+      - fastapi==0.95.1
+      - fastjsonschema==2.16.3
+      - fire==0.4.0
+      - flake8==6.0.0
+      - flask==2.2.3
+      - fonttools==4.39.3
+      - frozenlist==1.3.3
+      - fsspec==2023.4.0
+      # - fvcore==0.1.5.post20220512 #manual
+      - gitdb==4.0.10
+      - gitpython==3.1.31
+      - google-auth==2.17.3
+      - google-auth-oauthlib==1.0.0
+      - grpcio==1.54.0
+      - h11==0.14.0
+      - hydra-core==1.0.5
+      - imageio==2.21.1
+      - importlib-metadata==3.10.1
+      - iopath==0.1.10
+      - ipykernel==6.22.0
+      - ipython==8.12.0
+      - ipywidgets==8.0.6
+      - itsdangerous==2.1.2
+      - jedi==0.18.2
+      - jinja2==3.1.2
+      - joblib==1.2.0
+      - jsonschema==4.17.3
+      - jupyter-client==8.2.0
+      - jupyter-core==5.3.0
+      - jupyterlab-widgets==3.0.7
+      - kiwisolver==1.4.4
+      - lazy-loader==0.2
+      - loguru==0.6.0
+      - markdown==3.4.3
+      - markupsafe==2.1.2
+      - matplotlib==3.7.1
+      - matplotlib-inline==0.1.6
+      # - minkowskiengine==0.5.4
+      - multidict==6.0.4
+      - mypy-extensions==1.0.0
+      - natsort==8.3.1
+      - nbformat==5.7.0
+      - nest-asyncio==1.5.6
+      - networkx==3.1
+      - ninja==1.10.2.3
+      - numpy==1.24.2
+      - oauthlib==3.2.2
+      # - omegaconf==2.0.6 #manual
+      # - open3d==0.17.0 #manual
+      - opencv-python-headless==4.7.0.72
+      - pandas==2.0.0
+      - parso==0.8.3
+      - pathspec==0.11.1
+      - pathtools==0.1.2
+      - pexpect==4.8.0
+      - pickleshare==0.7.5
+      - pillow==9.5.0
+      - pip==23.1
+      - platformdirs==3.2.0
+      - plotly==5.14.1
+      - plyfile==0.7.4
+      # - pointnet2==0.0.0
+      - portalocker==2.7.0
+      - prompt-toolkit==3.0.38
+      - protobuf==4.22.3
+      - psutil==5.9.5
+      - ptyprocess==0.7.0
+      - pure-eval==0.2.2
+      - pyasn1==0.5.0
+      - pyasn1-modules==0.3.0
+      - pycocotools==2.0.4
+      - pydantic==1.10.7
+      - pydeprecate==0.3.2
+      - pygments==2.15.1
+      - pyparsing==3.0.9
+      - pyquaternion==0.9.9
+      - pyrsistent==0.19.3
+      - python-dateutil==2.8.2
+      - python-dotenv==0.20.0
+      - python-multipart==0.0.6
+      # - pytorch-lightning==1.7.2
+      - pytz==2023.3
+      - pyviz3d==0.2.28
+      - pywavelets==1.4.1
+      - pyyaml==5.3.1
+      - pyzmq==25.0.2
+      - qudida==0.0.4
+      - regex==2023.3.23
+      - requests-oauthlib==1.3.1
+      - rsa==4.9
+      - scikit-image==0.20.0
+      - scikit-learn==1.1.2
+      - scipy==1.9.0
+      - sentry-sdk==1.20.0
+      - setproctitle==1.3.2
+      - smmap==5.0.0
+      - sniffio==1.3.0
+      - stack-data==0.6.2
+      - starlette==0.26.1
+      - tabulate==0.9.0
+      - tenacity==8.2.2
+      - tensorboard==2.12.2
+      - tensorboard-data-server==0.7.0
+      - tensorboard-plugin-wit==1.8.1
+      - termcolor==2.2.0
+      - threadpoolctl==3.1.0
+      - tifffile==2023.4.12
+      - toml==0.10.2
+      # - torch==1.12.1+cu113
+      # - torch-scatter==2.1.1
+      # - torchmetrics==0.11.4
+      # - torchvision==0.13.1+cu113
+      - tornado==6.3
+      - traitlets==5.9.0
+      - trimesh==3.14.0
+      - typing-extensions==4.5.0
+      - tzdata==2023.3
+      - uvicorn==0.21.1
+      - volumentations==0.1.8
+      - wandb==0.15.0
+      - wcwidth==0.2.6
+      - werkzeug==2.2.3
+      - widgetsnbextension==4.0.7
+      - yacs==0.1.8
+      - yarl==1.8.2
+      - zipp==3.15.0
+prefix: /opt/conda

models/Mask3D/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2022
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

models/Mask3D/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include mask3d/conf *.yaml

models/Mask3D/README.md

@@ -0,0 +1,289 @@
+# Packaged version of Mask3D to be used in LabelMaker
+
+## Installation
+
+```
+# Some users experienced issues on Ubuntu with an AMD CPU
+# Install libopenblas-dev (issue #115, thanks WindWing)
+# sudo apt-get install libopenblas-dev
+
+export TORCH_CUDA_ARCH_LIST="6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6"
+
+conda env create -f environment.yml
+
+conda activate mask3d_cuda113
+
+pip3 install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
+pip3 install torch-scatter -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
+pip3 install 'git+https://github.com/facebookresearch/detectron2.git@710e7795d0eeadf9def0e7ef957eea13532e34cf' --no-deps
+
+mkdir third_party
+cd third_party
+
+git clone --recursive "https://github.com/NVIDIA/MinkowskiEngine"
+cd MinkowskiEngine
+git checkout 02fc608bea4c0549b0a7b00ca1bf15dee4a0b228
+python setup.py install --force_cuda --blas=openblas
+
+cd ..
+git clone https://github.com/ScanNet/ScanNet.git
+cd ScanNet/Segmentator
+git checkout 3e5726500896748521a6ceb81271b0f5b2c0e7d2
+make
+
+cd third_party/pointnet2
+python setup.py install
+
+cd ../../
+pip3 install pytorch-lightning==1.7.2
+
+pip install .
+
+```
+
+To use the model in your code you need to download a checkpoint from the list below.
+Afterwards, the basic model can be used like:
+
+
+```python
+from mask3d import get_model
+
+model = get_model(checkpoint_path='checkpoints/scannet200/scannet200_benchmark.ckpt')
+```
+
+
+Here is a minimal example assuming you have a pointcloud in the folder data.
+
+```python
+
+from mask3d import get_model, load_mesh, prepare_data, map_output_to_pointcloud, save_colorized_mesh 
+
+model = get_model('checkpoints/scannet200/scannet200_benchmark.ckpt')
+model.eval()
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model.to(device)
+
+# load input data
+pointcloud_file = 'data/pcl.ply'
+mesh = load_mesh(pointcloud_file)
+
+# prepare data
+data, points, colors, features, unique_map, inverse_map = prepare_data(mesh, device)
+
+# run model
+with torch.no_grad():
+    outputs = model(data, raw_coordinates=features)
+    
+# map output to point cloud
+labels = map_output_to_pointcloud(mesh, outputs, inverse_map)
+
+# save colorized mesh
+save_colorized_mesh(mesh, labels, 'data/pcl_labelled.ply', colormap='scannet200')
+```
+
+So far, only Scannet200 checkpoints are supported. We are working on the ScanNet checkpoints. 
+
+# Original Information
+
+## Mask3D: Mask Transformer for 3D Instance Segmentation
+<div align="center">
+<a href="https://jonasschult.github.io/">Jonas Schult</a><sup>1</sup>, <a href="https://francisengelmann.github.io/">Francis Engelmann</a><sup>2,3</sup>, <a href="https://www.vision.rwth-aachen.de/person/10/">Alexander Hermans</a><sup>1</sup>, <a href="https://orlitany.github.io/">Or Litany</a><sup>4</sup>, <a href="https://inf.ethz.ch/people/person-detail.MjYyNzgw.TGlzdC8zMDQsLTg3NDc3NjI0MQ==.html">Siyu Tang</a><sup>3</sup>,  <a href="https://www.vision.rwth-aachen.de/person/1/">Bastian Leibe</a><sup>1</sup>
+
+<sup>1</sup>RWTH Aachen University <sup>2</sup>ETH AI Center <sup>3</sup>ETH Zurich <sup>4</sup>NVIDIA
+
+Mask3D predicts accurate 3D semantic instances achieving state-of-the-art on ScanNet, ScanNet200, S3DIS and STPLS3D.
+
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/mask3d-for-3d-semantic-instance-segmentation/3d-instance-segmentation-on-scannetv2)](https://paperswithcode.com/sota/3d-instance-segmentation-on-scannetv2?p=mask3d-for-3d-semantic-instance-segmentation)
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/mask3d-for-3d-semantic-instance-segmentation/3d-instance-segmentation-on-scannet200)](https://paperswithcode.com/sota/3d-instance-segmentation-on-scannet200?p=mask3d-for-3d-semantic-instance-segmentation)
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/mask3d-for-3d-semantic-instance-segmentation/3d-instance-segmentation-on-s3dis)](https://paperswithcode.com/sota/3d-instance-segmentation-on-s3dis?p=mask3d-for-3d-semantic-instance-segmentation)
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/mask3d-for-3d-semantic-instance-segmentation/3d-instance-segmentation-on-stpls3d)](https://paperswithcode.com/sota/3d-instance-segmentation-on-stpls3d?p=mask3d-for-3d-semantic-instance-segmentation)
+
+<a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white"></a>
+<a href="https://pytorchlightning.ai/"><img alt="Lightning" src="https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning&logoColor=white"></a>
+<a href="https://hydra.cc/"><img alt="Config: Hydra" src="https://img.shields.io/badge/Config-Hydra-89b8cd"></a>
+
+![teaser](./docs/teaser.jpg)
+
+</div>
+<br><br>
+
+[[Project Webpage](https://jonasschult.github.io/Mask3D/)]
+[[Paper](https://arxiv.org/abs/2210.03105)]
+[[Demo](https://francisengelmann.github.io/mask3d/)]
+
+
+## News
+
+* **17. January 2023**: Mask3D is accepted at ICRA 2023. :fire:
+* **14. October 2022**: STPLS3D support added.
+* **10. October 2022**: Mask3D ranks 2nd on the [STPLS3D Challenge](https://codalab.lisn.upsaclay.fr/competitions/4646#results) hosted by the [Urban3D Workshop](https://urban3dchallenge.github.io/) at ECCV 2022.
+* **6. October 2022**: [Mask3D preprint](https://arxiv.org/abs/2210.03105) released on arXiv.
+* **25. September 2022**: Code released.
+
+## Code structure
+We adapt the codebase of [Mix3D](https://github.com/kumuji/mix3d) which provides a highly modularized framework for 3D Semantic Segmentation based on the MinkowskiEngine.
+
+```
+├── mix3d
+│   ├── main_instance_segmentation.py <- the main file
+│   ├── conf                          <- hydra configuration files
+│   ├── datasets
+│   │   ├── preprocessing             <- folder with preprocessing scripts
+│   │   ├── semseg.py                 <- indoor dataset
+│   │   └── utils.py        
+│   ├── models                        <- Mask3D modules
+│   ├── trainer
+│   │   ├── __init__.py
+│   │   └── trainer.py                <- train loop
+│   └── utils
+├── data
+│   ├── processed                     <- folder for preprocessed datasets
+│   └── raw                           <- folder for raw datasets
+├── scripts                           <- train scripts
+├── docs
+├── README.md
+└── saved                             <- folder that stores models and logs
+```
+
+### Dependencies :memo:
+The main dependencies of the project are the following:
+```yaml
+python: 3.10.9
+cuda: 11.3
+```
+You can set up a conda environment as follows
+```
+# Some users experienced issues on Ubuntu with an AMD CPU
+# Install libopenblas-dev (issue #115, thanks WindWing)
+# sudo apt-get install libopenblas-dev
+
+export TORCH_CUDA_ARCH_LIST="6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6"
+
+conda env create -f environment.yml
+
+conda activate mask3d_cuda113
+
+pip3 install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
+pip3 install torch-scatter -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
+pip3 install 'git+https://github.com/facebookresearch/detectron2.git@710e7795d0eeadf9def0e7ef957eea13532e34cf' --no-deps
+
+mkdir third_party
+cd third_party
+
+git clone --recursive "https://github.com/NVIDIA/MinkowskiEngine"
+cd MinkowskiEngine
+git checkout 02fc608bea4c0549b0a7b00ca1bf15dee4a0b228
+python setup.py install --force_cuda --blas=openblas
+
+cd ..
+git clone https://github.com/ScanNet/ScanNet.git
+cd ScanNet/Segmentator
+git checkout 3e5726500896748521a6ceb81271b0f5b2c0e7d2
+make
+
+cd ../../pointnet2
+python setup.py install
+
+cd ../../
+pip3 install pytorch-lightning==1.7.2
+```
+
+### Data preprocessing :hammer:
+After installing the dependencies, we preprocess the datasets.
+
+#### ScanNet / ScanNet200
+First, we apply Felzenswalb and Huttenlocher's Graph Based Image Segmentation algorithm to the test scenes using the default parameters.
+Please refer to the [original repository](https://github.com/ScanNet/ScanNet/tree/master/Segmentator) for details.
+Put the resulting segmentations in `./data/raw/scannet_test_segments`.
+```
+python -m datasets.preprocessing.scannet_preprocessing preprocess \
+--data_dir="PATH_TO_RAW_SCANNET_DATASET" \
+--save_dir="data/processed/scannet" \
+--git_repo="PATH_TO_SCANNET_GIT_REPO" \
+--scannet200=false/true
+```
+
+#### S3DIS
+The S3DIS dataset contains some smalls bugs which we initially fixed manually. We will soon release a preprocessing script which directly preprocesses the original dataset. For the time being, please follow the instructions [here](https://github.com/JonasSchult/Mask3D/issues/8#issuecomment-1279535948) to fix the dataset manually. Afterwards, call the preprocessing script as follows:
+
+```
+python -m datasets.preprocessing.s3dis_preprocessing preprocess \
+--data_dir="PATH_TO_Stanford3dDataset_v1.2" \
+--save_dir="data/processed/s3dis"
+```
+
+#### STPLS3D
+```
+python -m datasets.preprocessing.stpls3d_preprocessing preprocess \
+--data_dir="PATH_TO_STPLS3D" \
+--save_dir="data/processed/stpls3d"
+```
+
+### Training and testing :train2:
+Train Mask3D on the ScanNet dataset:
+```bash
+python main_instance_segmentation.py
+```
+Please refer to the [config scripts](https://github.com/JonasSchult/Mask3D/tree/main/scripts) (for example [here](https://github.com/JonasSchult/Mask3D/blob/main/scripts/scannet/scannet_val.sh#L15)) for detailed instructions how to reproduce our results.
+In the simplest case the inference command looks as follows:
+```bash
+python main_instance_segmentation.py \
+general.checkpoint='PATH_TO_CHECKPOINT.ckpt' \
+general.train_mode=false
+```
+
+## Trained checkpoints :floppy_disk:
+We provide detailed scores and network configurations with trained checkpoints.
+
+### [S3DIS](http://buildingparser.stanford.edu/dataset.html) (pretrained on ScanNet train+val)
+Following PointGroup, HAIS and SoftGroup, we finetune a model pretrained on ScanNet ([config](./scripts/scannet/scannet_pretrain_for_s3dis.sh) and [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/scannet_pretrained.ckpt)).
+| Dataset | AP | AP_50 | AP_25 | Config | Checkpoint :floppy_disk: | Scores :chart_with_upwards_trend: | Visualizations :telescope:
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| Area 1 | 69.3 | 81.9 | 87.7 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area1_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area1_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_1/)
+| Area 2 | 44.0 | 59.5 | 66.5 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area2_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area2_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_2/)
+| Area 3 | 73.4 | 83.2 | 88.2 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area3_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area3_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_3/)
+| Area 4 | 58.0 | 69.5 | 74.9 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area4_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area4_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_4/)
+| Area 5 | 57.8 | 71.9 | 77.2 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area5_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area5_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_5/)
+| Area 6 | 68.4 | 79.9 | 85.2 | [config](scripts/s3dis/s3dis_pretrained.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/scannet_pretrained/area6_scannet_pretrained.ckpt) | [scores](./docs/detailed_scores/s3dis/scannet_pretrained/s3dis_area6_scannet_pretrained.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/scannet_pretrained/area_6/)
+
+### [S3DIS](http://buildingparser.stanford.edu/dataset.html) (from scratch)
+
+| Dataset | AP | AP_50 | AP_25 | Config | Checkpoint :floppy_disk: | Scores :chart_with_upwards_trend: | Visualizations :telescope:
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| Area 1 | 74.1 | 85.1 | 89.6 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area1_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area1_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_1/)
+| Area 2 | 44.9 | 57.1 | 67.9 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area2_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area2_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_2/)
+| Area 3 | 74.4 | 84.4 | 88.1 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area3_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area3_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_3/)
+| Area 4 | 63.8 | 74.7 | 81.1 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area4_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area4_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_4/)
+| Area 5 | 56.6 | 68.4 | 75.2 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area5_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area5_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_5/)
+| Area 6 | 73.3 | 83.4 | 87.8 | [config](scripts/s3dis/s3dis_from_scratch.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/s3dis/from_scratch/area6_from_scratch.ckpt) | [scores](./docs/detailed_scores/s3dis/from_scratch/s3dis_area6_from_scratch.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/s3dis/from_scratch/area_6/)
+
+### [ScanNet v2](https://kaldir.vc.in.tum.de/scannet_benchmark/semantic_instance_3d?metric=ap)
+
+| Dataset | AP | AP_50 | AP_25 | Config | Checkpoint :floppy_disk: | Scores :chart_with_upwards_trend: | Visualizations :telescope:
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| ScanNet val  | 55.2 | 73.7 | 83.5 | [config](scripts/scannet/scannet_val.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/scannet/scannet_val.ckpt) | [scores](./docs/detailed_scores/scannet_val.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/scannet/val/)
+| ScanNet test | 56.6 | 78.0 | 87.0 | [config](scripts/scannet/scannet_benchmark.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/scannet/scannet_benchmark.ckpt) | [scores](http://kaldir.vc.in.tum.de/scannet_benchmark/result_details?id=1081) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/scannet/test/)
+
+### [ScanNet 200](https://kaldir.vc.in.tum.de/scannet_benchmark/scannet200_semantic_instance_3d)
+
+| Dataset | AP | AP_50 | AP_25 | Config | Checkpoint :floppy_disk: | Scores :chart_with_upwards_trend: | Visualizations :telescope:
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| ScanNet200 val | 27.4 | 37.0 | 42.3 | [config](scripts/scannet200/scannet200_val.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/scannet200/scannet200_val.ckpt) | [scores](./docs/detailed_scores/scannet200_val.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/scannet200/val/)
+| ScanNet200 test | 27.8 | 38.8 | 44.5 | [config](scripts/scannet200/scannet200_benchmark.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/scannet200/scannet200_benchmark.ckpt) | [scores](https://kaldir.vc.in.tum.de/scannet_benchmark/result_details?id=1242) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/scannet200/test/)
+
+### [STPLS3D](https://www.stpls3d.com/)
+
+| Dataset | AP | AP_50 | AP_25 | Config | Checkpoint :floppy_disk: | Scores :chart_with_upwards_trend: | Visualizations :telescope:
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| STPLS3D val | 57.3 | 74.3 | 81.6 | [config](scripts/stpls3d/stpls3d_val.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/stpls3d/stpls3d_val.ckpt) | [scores](./docs/detailed_scores/stpls3d.txt) | [visualizations](https://omnomnom.vision.rwth-aachen.de/data/mask3d/visualizations/stpls3d/)
+| STPLS3D test | 63.4 | 79.2 | 85.6 | [config](scripts/stpls3d/stpls3d_benchmark.sh) | [checkpoint](https://omnomnom.vision.rwth-aachen.de/data/mask3d/checkpoints/stpls3d/stpls3d_benchmark.zip) | [scores](https://codalab.lisn.upsaclay.fr/competitions/4646#results) | visualizations
+
+## BibTeX :pray:
+```
+@article{Schult23ICRA,
+  title     = {{Mask3D: Mask Transformer for 3D Semantic Instance Segmentation}},
+  author    = {Schult, Jonas and Engelmann, Francis and Hermans, Alexander and Litany, Or and Tang, Siyu and Leibe, Bastian},
+  booktitle = {{International Conference on Robotics and Automation (ICRA)}},
+  year      = {2023}
+}
+```

models/Mask3D/build/lib/mask3d/__init__.py

@@ -0,0 +1,216 @@
+import hydra
+import torch
+
+from mask3d.models.mask3d import Mask3D
+from mask3d.utils.utils import (
+    load_checkpoint_with_missing_or_exsessive_keys,
+    load_backbone_checkpoint_with_missing_or_exsessive_keys,
+)
+
+class InstanceSegmentation(torch.nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.model = hydra.utils.instantiate(cfg.model)
+
+
+    def forward(self, x, raw_coordinates=None, point2segment=None):
+        return self.model(x, raw_coordinates=raw_coordinates, point2segment=point2segment)
+    
+
+from omegaconf import OmegaConf, DictConfig
+import hydra
+from hydra.core.global_hydra import GlobalHydra
+from hydra.experimental import initialize, compose
+
+# imports for input loading
+import albumentations as A
+import MinkowskiEngine as ME
+import numpy as np
+import open3d as o3d
+
+# imports for output
+from mask3d.datasets.scannet200.scannet200_constants import (VALID_CLASS_IDS_20, VALID_CLASS_IDS_200, SCANNET_COLOR_MAP_20, SCANNET_COLOR_MAP_200)
+
+def get_model(checkpoint_path=None, dataset_name = "scannet200"):
+
+
+    # Initialize the directory with config files
+    with initialize(config_path="conf"):
+        # Compose a configuration
+        cfg = compose(config_name="config_base_instance_segmentation.yaml")
+
+    cfg.general.checkpoint = checkpoint_path
+
+    # would be nicd to avoid this hardcoding below
+    # dataset_name = checkpoint_path.split('/')[-1].split('_')[0]
+    if dataset_name == 'scannet200':
+        cfg.general.num_targets = 201
+        cfg.general.train_mode = False
+        cfg.general.eval_on_segments = True
+        cfg.general.topk_per_image = 300
+        cfg.general.use_dbscan = True
+        cfg.general.dbscan_eps = 0.95
+        cfg.general.export_threshold = 0.001
+
+        # # data
+        cfg.data.num_labels = 200
+        cfg.data.test_mode = "validation"
+
+        # # model
+        cfg.model.num_queries = 150
+        
+    if dataset_name == 'scannet':
+        cfg.general.num_targets = 19
+        cfg.general.train_mode = False
+        cfg.general.eval_on_segments = True
+        cfg.general.topk_per_image = 300
+        cfg.general.use_dbscan = True
+        cfg.general.dbscan_eps = 0.95
+        cfg.general.export_threshold = 0.001
+
+        # # data
+        cfg.data.num_labels = 20
+        cfg.data.test_mode = "test"
+
+        # # model
+        cfg.model.num_queries = 150
+        
+        #TODO: this has to be fixed and discussed with Jonas
+        # cfg.model.scene_min = -3.
+        # cfg.model.scene_max = 3.
+
+    # # Initialize the Hydra context
+    # hydra.core.global_hydra.GlobalHydra.instance().clear()
+    # hydra.initialize(config_path="conf")
+
+    # Load the configuration
+    # cfg = hydra.compose(config_name="config_base_instance_segmentation.yaml")
+    model = InstanceSegmentation(cfg)
+
+    if cfg.general.backbone_checkpoint is not None:
+        cfg, model = load_backbone_checkpoint_with_missing_or_exsessive_keys(
+            cfg, model
+        )
+    if cfg.general.checkpoint is not None:
+        cfg, model = load_checkpoint_with_missing_or_exsessive_keys(cfg, model)
+
+    return model
+
+
+def load_mesh(pcl_file):
+    
+    # load point cloud
+    input_mesh_path = pcl_file
+    mesh = o3d.io.read_triangle_mesh(input_mesh_path)
+    return mesh
+
+def prepare_data(mesh, device):
+    
+    # normalization for point cloud features
+    color_mean = (0.47793125906962, 0.4303257521323044, 0.3749598901421883)
+    color_std = (0.2834475483823543, 0.27566157565723015, 0.27018971370874995)
+    normalize_color = A.Normalize(mean=color_mean, std=color_std)
+
+    
+    points = np.asarray(mesh.vertices)
+    colors = np.asarray(mesh.vertex_colors)
+    colors = colors * 255.
+
+    pseudo_image = colors.astype(np.uint8)[np.newaxis, :, :]
+    colors = np.squeeze(normalize_color(image=pseudo_image)["image"])
+
+    coords = np.floor(points / 0.02)
+    _, _, unique_map, inverse_map = ME.utils.sparse_quantize(
+        coordinates=coords,
+        features=colors,
+        return_index=True,
+        return_inverse=True,
+    )
+
+    sample_coordinates = coords[unique_map]
+    coordinates = [torch.from_numpy(sample_coordinates).int()]
+    sample_features = colors[unique_map]
+    features = [torch.from_numpy(sample_features).float()]
+
+    coordinates, _ = ME.utils.sparse_collate(coords=coordinates, feats=features)
+    features = torch.cat(features, dim=0)
+    data = ME.SparseTensor(
+        coordinates=coordinates,
+        features=features,
+        device=device,
+    )
+    
+    
+    return data, points, colors, features, unique_map, inverse_map
+
+
+def map_output_to_pointcloud(mesh, 
+                             outputs, 
+                             inverse_map):
+    
+    # parse predictions
+    logits = outputs["pred_logits"]
+    masks = outputs["pred_masks"]
+
+    # reformat predictions
+    logits = logits[0]
+    masks = masks[0]
+
+    labels = []
+    confidences = []
+    masks_binary = []
+
+    for i in range(len(logits)):
+        p_labels = torch.softmax(logits[i], dim=-1)
+        p_masks = torch.sigmoid(masks[:, i])
+        l = torch.argmax(p_labels, dim=-1)
+        c_label = torch.max(p_labels)
+        m = p_masks > 0.5
+        c_m = p_masks[m].sum() / (m.sum() + 1e-8)
+        c = c_label * c_m
+        labels.append(l.item())
+        confidences.append(c.item())
+        masks_binary.append(m[inverse_map])  # mapping the mask back to the original point cloud 
+    return (torch.stack(masks_binary), torch.tensor(confidences))
+
+def save_colorized_mesh(mesh, labels_mapped, output_file, colormap='scannet'):
+    
+    # colorize mesh
+    colors = np.zeros((len(mesh.vertices), 3))
+    for li in np.unique(labels_mapped):
+        if colormap == 'scannet':
+            raise ValueError('Not implemented yet')
+        elif colormap == 'scannet200':
+            v_li = VALID_CLASS_IDS_200[int(li)]
+            colors[(labels_mapped == li)[:, 0], :] = SCANNET_COLOR_MAP_200[v_li]
+        else:
+            raise ValueError('Unknown colormap - not supported')
+    
+    colors = colors / 255.
+    mesh.vertex_colors = o3d.utility.Vector3dVector(colors)
+    o3d.io.write_triangle_mesh(output_file, mesh)
+
+if __name__ == '__main__':
+    
+    model = get_model('checkpoints/scannet200/scannet200_benchmark.ckpt')
+    model.eval()
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    model.to(device)
+    
+    # load input data
+    pointcloud_file = 'data/pcl.ply'
+    mesh = load_mesh(pointcloud_file)
+    
+    # prepare data
+    data, points, colors, features, unique_map, inverse_map = prepare_data(mesh, device)
+    
+    # run model
+    with torch.no_grad():
+        outputs = model(data, raw_coordinates=features)
+        
+    # map output to point cloud
+    labels = map_output_to_pointcloud(mesh, outputs, inverse_map)
+    
+    # save colorized mesh
+    save_colorized_mesh(mesh, labels, 'data/pcl_labelled.ply', colormap='scannet200')
+    

models/Mask3D/build/lib/mask3d/benchmark/evaluate_semantic_instance.py

@@ -0,0 +1,1141 @@
+# Evaluates semantic instance task
+# Adapted from the CityScapes evaluation: https://github.com/mcordts/cityscapesScripts/tree/master/cityscapesscripts/evaluation
+# Input:
+#   - path to .txt prediction files
+#   - path to .txt ground truth files
+#   - output file to write results to
+# Each .txt prediction file look like:
+#    [(pred0) rel. path to pred. mask over verts as .txt] [(pred0) label id] [(pred0) confidence]
+#    [(pred1) rel. path to pred. mask over verts as .txt] [(pred1) label id] [(pred1) confidence]
+#    [(pred2) rel. path to pred. mask over verts as .txt] [(pred2) label id] [(pred2) confidence]
+#    ...
+#
+# NOTE: The prediction files must live in the root of the given prediction path.
+#       Predicted mask .txt files must live in a subfolder.
+#       Additionally, filenames must not contain spaces.
+# The relative paths to predicted masks must contain one integer per line,
+# where each line corresponds to vertices in the *_vh_clean_2.ply (in that order).
+# Non-zero integers indicate part of the predicted instance.
+# The label ids specify the class of the corresponding mask.
+# Confidence is a float confidence score of the mask.
+#
+# Note that only the valid classes are used for evaluation,
+# i.e., any ground truth label not in the valid label set
+# is ignored in the evaluation.
+#
+# example usage: evaluate_semantic_instance.py --scan_path [path to scan data] --output_file [output file]
+
+# python imports
+import math
+import os, sys, argparse
+import inspect
+from copy import deepcopy
+from uuid import uuid4
+
+import torch
+
+try:
+    import numpy as np
+except:
+    print("Failed to import numpy package.")
+    sys.exit(-1)
+
+from scipy import stats
+
+# currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
+# parentdir = os.path.dirname(currentdir)
+# sys.path.insert(0,parentdir)
+import benchmark.util as util
+import benchmark.util_3d as util_3d
+
+# parser = argparse.ArgumentParser()
+# parser.add_argument('--gt_path', default='', help='path to directory of gt .txt files')
+# parser.add_argument('--output_file', default='', help='output file [default: ./semantic_instance_evaluation.txt]')
+# opt = parser.parse_args()
+
+# if opt.output_file == '':
+#    opt.output_file = os.path.join(os.getcwd(), 'semantic_instance_evaluation.txt')
+
+
+# ---------- Label info ---------- #
+CLASS_LABELS = [
+    "cabinet",
+    "bed",
+    "chair",
+    "sofa",
+    "table",
+    "door",
+    "window",
+    "bookshelf",
+    "picture",
+    "counter",
+    "desk",
+    "curtain",
+    "refrigerator",
+    "shower curtain",
+    "toilet",
+    "sink",
+    "bathtub",
+    "otherfurniture",
+]
+VALID_CLASS_IDS = np.array(
+    [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28, 33, 34, 36, 39]
+)
+ID_TO_LABEL = {}
+LABEL_TO_ID = {}
+for i in range(len(VALID_CLASS_IDS)):
+    LABEL_TO_ID[CLASS_LABELS[i]] = VALID_CLASS_IDS[i]
+    ID_TO_LABEL[VALID_CLASS_IDS[i]] = CLASS_LABELS[i]
+# ---------- Evaluation params ---------- #
+# overlaps for evaluation
+opt = {}
+opt["overlaps"] = np.append(np.arange(0.5, 0.95, 0.05), 0.25)
+# minimum region size for evaluation [verts]
+opt["min_region_sizes"] = np.array([100])  # 100 for s3dis, scannet
+# distance thresholds [m]
+opt["distance_threshes"] = np.array([float("inf")])
+# distance confidences
+opt["distance_confs"] = np.array([-float("inf")])
+
+
+def evaluate_matches(matches):
+    overlaps = opt["overlaps"]
+    min_region_sizes = [opt["min_region_sizes"][0]]
+    dist_threshes = [opt["distance_threshes"][0]]
+    dist_confs = [opt["distance_confs"][0]]
+
+    # results: class x overlap
+    ap = np.zeros(
+        (len(dist_threshes), len(CLASS_LABELS), len(overlaps)), float
+    )
+    for di, (min_region_size, distance_thresh, distance_conf) in enumerate(
+        zip(min_region_sizes, dist_threshes, dist_confs)
+    ):
+        for oi, overlap_th in enumerate(overlaps):
+            pred_visited = {}
+            for m in matches:
+                for p in matches[m]["pred"]:
+                    for label_name in CLASS_LABELS:
+                        for p in matches[m]["pred"][label_name]:
+                            if "uuid" in p:
+                                pred_visited[p["uuid"]] = False
+            for li, label_name in enumerate(CLASS_LABELS):
+                y_true = np.empty(0)
+                y_score = np.empty(0)
+                hard_false_negatives = 0
+                has_gt = False
+                has_pred = False
+                for m in matches:
+                    pred_instances = matches[m]["pred"][label_name]
+                    gt_instances = matches[m]["gt"][label_name]
+                    # filter groups in ground truth
+                    gt_instances = [
+                        gt
+                        for gt in gt_instances
+                        if gt["instance_id"] >= 1000
+                        and gt["vert_count"] >= min_region_size
+                        and gt["med_dist"] <= distance_thresh
+                        and gt["dist_conf"] >= distance_conf
+                    ]
+                    if gt_instances:
+                        has_gt = True
+                    if pred_instances:
+                        has_pred = True
+
+                    cur_true = np.ones(len(gt_instances))
+                    cur_score = np.ones(len(gt_instances)) * (-float("inf"))
+                    cur_match = np.zeros(len(gt_instances), dtype=bool)
+                    # collect matches
+                    for (gti, gt) in enumerate(gt_instances):
+                        found_match = False
+                        num_pred = len(gt["matched_pred"])
+                        for pred in gt["matched_pred"]:
+                            # greedy assignments
+                            if pred_visited[pred["uuid"]]:
+                                continue
+                            overlap = float(pred["intersection"]) / (
+                                gt["vert_count"]
+                                + pred["vert_count"]
+                                - pred["intersection"]
+                            )
+                            if overlap > overlap_th:
+                                confidence = pred["confidence"]
+                                # if already have a prediction for this gt,
+                                # the prediction with the lower score is automatically a false positive
+                                if cur_match[gti]:
+                                    max_score = max(cur_score[gti], confidence)
+                                    min_score = min(cur_score[gti], confidence)
+                                    cur_score[gti] = max_score
+                                    # append false positive
+                                    cur_true = np.append(cur_true, 0)
+                                    cur_score = np.append(cur_score, min_score)
+                                    cur_match = np.append(cur_match, True)
+                                # otherwise set score
+                                else:
+                                    found_match = True
+                                    cur_match[gti] = True
+                                    cur_score[gti] = confidence
+                                    pred_visited[pred["uuid"]] = True
+                        if not found_match:
+                            hard_false_negatives += 1
+                    # remove non-matched ground truth instances
+                    cur_true = cur_true[cur_match == True]
+                    cur_score = cur_score[cur_match == True]
+
+                    # collect non-matched predictions as false positive
+                    for pred in pred_instances:
+                        found_gt = False
+                        for gt in pred["matched_gt"]:
+                            overlap = float(gt["intersection"]) / (
+                                gt["vert_count"]
+                                + pred["vert_count"]
+                                - gt["intersection"]
+                            )
+                            if overlap > overlap_th:
+                                found_gt = True
+                                break
+                        if not found_gt:
+                            num_ignore = pred["void_intersection"]
+                            for gt in pred["matched_gt"]:
+                                # group?
+                                if gt["instance_id"] < 1000:
+                                    num_ignore += gt["intersection"]
+                                # small ground truth instances
+                                if (
+                                    gt["vert_count"] < min_region_size
+                                    or gt["med_dist"] > distance_thresh
+                                    or gt["dist_conf"] < distance_conf
+                                ):
+                                    num_ignore += gt["intersection"]
+                            proportion_ignore = (
+                                float(num_ignore) / pred["vert_count"]
+                            )
+                            # if not ignored append false positive
+                            if proportion_ignore <= overlap_th:
+                                cur_true = np.append(cur_true, 0)
+                                confidence = pred["confidence"]
+                                cur_score = np.append(cur_score, confidence)
+
+                    # append to overall results
+                    y_true = np.append(y_true, cur_true)
+                    y_score = np.append(y_score, cur_score)
+
+                # compute average precision
+                if has_gt and has_pred:
+                    # compute precision recall curve first
+
+                    # sorting and cumsum
+                    score_arg_sort = np.argsort(y_score)
+                    y_score_sorted = y_score[score_arg_sort]
+                    y_true_sorted = y_true[score_arg_sort]
+                    y_true_sorted_cumsum = np.cumsum(y_true_sorted)
+
+                    # unique thresholds
+                    (thresholds, unique_indices) = np.unique(
+                        y_score_sorted, return_index=True
+                    )
+                    num_prec_recall = len(unique_indices) + 1
+
+                    # prepare precision recall
+                    num_examples = len(y_score_sorted)
+                    # https://github.com/ScanNet/ScanNet/pull/26
+                    # all predictions are non-matched but also all of them are ignored and not counted as FP
+                    # y_true_sorted_cumsum is empty
+                    # num_true_examples = y_true_sorted_cumsum[-1]
+                    num_true_examples = (
+                        y_true_sorted_cumsum[-1]
+                        if len(y_true_sorted_cumsum) > 0
+                        else 0
+                    )
+                    precision = np.zeros(num_prec_recall)
+                    recall = np.zeros(num_prec_recall)
+
+                    # deal with the first point
+                    y_true_sorted_cumsum = np.append(y_true_sorted_cumsum, 0)
+                    # deal with remaining
+                    for idx_res, idx_scores in enumerate(unique_indices):
+                        cumsum = y_true_sorted_cumsum[idx_scores - 1]
+                        tp = num_true_examples - cumsum
+                        fp = num_examples - idx_scores - tp
+                        fn = cumsum + hard_false_negatives
+                        p = float(tp) / (tp + fp)
+                        r = float(tp) / (tp + fn)
+                        precision[idx_res] = p
+                        recall[idx_res] = r
+
+                    # first point in curve is artificial
+                    precision[-1] = 1.0
+                    recall[-1] = 0.0
+
+                    # compute average of precision-recall curve
+                    recall_for_conv = np.copy(recall)
+                    recall_for_conv = np.append(
+                        recall_for_conv[0], recall_for_conv
+                    )
+                    recall_for_conv = np.append(recall_for_conv, 0.0)
+
+                    stepWidths = np.convolve(
+                        recall_for_conv, [-0.5, 0, 0.5], "valid"
+                    )
+                    # integrate is now simply a dot product
+                    ap_current = np.dot(precision, stepWidths)
+
+                elif has_gt:
+                    ap_current = 0.0
+                else:
+                    ap_current = float("nan")
+                ap[di, li, oi] = ap_current
+    return ap
+
+
+def compute_averages(aps):
+    d_inf = 0
+    o50 = np.where(np.isclose(opt["overlaps"], 0.5))
+    o25 = np.where(np.isclose(opt["overlaps"], 0.25))
+    oAllBut25 = np.where(np.logical_not(np.isclose(opt["overlaps"], 0.25)))
+    avg_dict = {}
+    # avg_dict['all_ap']     = np.nanmean(aps[ d_inf,:,:  ])
+    avg_dict["all_ap"] = np.nanmean(aps[d_inf, :, oAllBut25])
+    avg_dict["all_ap_50%"] = np.nanmean(aps[d_inf, :, o50])
+    avg_dict["all_ap_25%"] = np.nanmean(aps[d_inf, :, o25])
+    avg_dict["classes"] = {}
+    for (li, label_name) in enumerate(CLASS_LABELS):
+        avg_dict["classes"][label_name] = {}
+        # avg_dict["classes"][label_name]["ap"]       = np.average(aps[ d_inf,li,  :])
+        avg_dict["classes"][label_name]["ap"] = np.average(
+            aps[d_inf, li, oAllBut25]
+        )
+        avg_dict["classes"][label_name]["ap50%"] = np.average(
+            aps[d_inf, li, o50]
+        )
+        avg_dict["classes"][label_name]["ap25%"] = np.average(
+            aps[d_inf, li, o25]
+        )
+    return avg_dict
+
+
+def make_pred_info(pred: dict):
+    # pred = {'pred_scores' = 100, 'pred_classes' = 100 'pred_masks' = Nx100}
+    pred_info = {}
+    assert (
+        pred["pred_classes"].shape[0]
+        == pred["pred_scores"].shape[0]
+        == pred["pred_masks"].shape[1]
+    )
+    for i in range(len(pred["pred_classes"])):
+        info = {}
+        info["label_id"] = pred["pred_classes"][i]
+        info["conf"] = pred["pred_scores"][i]
+        info["mask"] = pred["pred_masks"][:, i]
+        pred_info[uuid4()] = info  # we later need to identify these objects
+    return pred_info
+
+
+def assign_instances_for_scan(pred: dict, gt_file: str):
+    pred_info = make_pred_info(pred)
+    try:
+        gt_ids = util_3d.load_ids(gt_file)
+    except Exception as e:
+        util.print_error("unable to load " + gt_file + ": " + str(e))
+
+    # get gt instances
+    gt_instances = util_3d.get_instances(
+        gt_ids, VALID_CLASS_IDS, CLASS_LABELS, ID_TO_LABEL
+    )
+    # associate
+    gt2pred = deepcopy(gt_instances)
+    for label in gt2pred:
+        for gt in gt2pred[label]:
+            gt["matched_pred"] = []
+    pred2gt = {}
+    for label in CLASS_LABELS:
+        pred2gt[label] = []
+    num_pred_instances = 0
+    # mask of void labels in the groundtruth
+    bool_void = np.logical_not(np.in1d(gt_ids // 1000, VALID_CLASS_IDS))
+    # go thru all prediction masks
+    for uuid in pred_info:
+        label_id = int(pred_info[uuid]["label_id"])
+        conf = pred_info[uuid]["conf"]
+        if not label_id in ID_TO_LABEL:
+            continue
+        label_name = ID_TO_LABEL[label_id]
+        # read the mask
+        pred_mask = pred_info[uuid]["mask"]
+        assert len(pred_mask) == len(gt_ids)
+        # convert to binary
+        pred_mask = np.not_equal(pred_mask, 0)
+        num = np.count_nonzero(pred_mask)
+        if num < opt["min_region_sizes"][0]:
+            continue  # skip if empty
+
+        pred_instance = {}
+        pred_instance["uuid"] = uuid
+        pred_instance["pred_id"] = num_pred_instances
+        pred_instance["label_id"] = label_id
+        pred_instance["vert_count"] = num
+        pred_instance["confidence"] = conf
+        pred_instance["void_intersection"] = np.count_nonzero(
+            np.logical_and(bool_void, pred_mask)
+        )
+
+        # matched gt instances
+        matched_gt = []
+        # go thru all gt instances with matching label
+        for (gt_num, gt_inst) in enumerate(gt2pred[label_name]):
+            intersection = np.count_nonzero(
+                np.logical_and(gt_ids == gt_inst["instance_id"], pred_mask)
+            )
+            if intersection > 0:
+                gt_copy = gt_inst.copy()
+                pred_copy = pred_instance.copy()
+                gt_copy["intersection"] = intersection
+                pred_copy["intersection"] = intersection
+                matched_gt.append(gt_copy)
+                gt2pred[label_name][gt_num]["matched_pred"].append(pred_copy)
+        pred_instance["matched_gt"] = matched_gt
+        num_pred_instances += 1
+        pred2gt[label_name].append(pred_instance)
+
+    return gt2pred, pred2gt
+
+
+def print_results(avgs):
+    sep = ""
+    col1 = ":"
+    lineLen = 64
+
+    print("")
+    print("#" * lineLen)
+    line = ""
+    line += "{:<15}".format("what") + sep + col1
+    line += "{:>15}".format("AP") + sep
+    line += "{:>15}".format("AP_50%") + sep
+    line += "{:>15}".format("AP_25%") + sep
+    print(line)
+    print("#" * lineLen)
+
+    for (li, label_name) in enumerate(CLASS_LABELS):
+        ap_avg = avgs["classes"][label_name]["ap"]
+        ap_50o = avgs["classes"][label_name]["ap50%"]
+        ap_25o = avgs["classes"][label_name]["ap25%"]
+        line = "{:<15}".format(label_name) + sep + col1
+        line += sep + "{:>15.3f}".format(ap_avg) + sep
+        line += sep + "{:>15.3f}".format(ap_50o) + sep
+        line += sep + "{:>15.3f}".format(ap_25o) + sep
+        print(line)
+
+    all_ap_avg = avgs["all_ap"]
+    all_ap_50o = avgs["all_ap_50%"]
+    all_ap_25o = avgs["all_ap_25%"]
+
+    print("-" * lineLen)
+    line = "{:<15}".format("average") + sep + col1
+    line += "{:>15.3f}".format(all_ap_avg) + sep
+    line += "{:>15.3f}".format(all_ap_50o) + sep
+    line += "{:>15.3f}".format(all_ap_25o) + sep
+    print(line)
+    print("")
+
+
+def write_result_file(avgs, filename):
+    _SPLITTER = ","
+    with open(filename, "w") as f:
+        f.write(
+            _SPLITTER.join(["class", "class id", "ap", "ap50", "ap25"]) + "\n"
+        )
+        for i in range(len(VALID_CLASS_IDS)):
+            class_name = CLASS_LABELS[i]
+            class_id = VALID_CLASS_IDS[i]
+            ap = avgs["classes"][class_name]["ap"]
+            ap50 = avgs["classes"][class_name]["ap50%"]
+            ap25 = avgs["classes"][class_name]["ap25%"]
+            f.write(
+                _SPLITTER.join(
+                    [str(x) for x in [class_name, class_id, ap, ap50, ap25]]
+                )
+                + "\n"
+            )
+
+
+def evaluate(
+    preds: dict, gt_path: str, output_file: str, dataset: str = "scannet"
+):
+    global CLASS_LABELS
+    global VALID_CLASS_IDS
+    global ID_TO_LABEL
+    global LABEL_TO_ID
+    global opt
+
+    if dataset == "stpls3d":
+        # global CLASS_LABELS
+        # global VALID_CLASS_IDS
+        # global ID_TO_LABEL
+        # global LABEL_TO_ID
+
+        opt["min_region_sizes"] = np.array([10])
+
+        CLASS_LABELS = [
+            "Build",
+            "LowVeg",
+            "MediumVeg",
+            "HighVeg",
+            "Vehicle",
+            "Truck",
+            "Aircraft",
+            "MilitaryVeh",
+            "Bike",
+            "Motorcycle",
+            "LightPole",
+            "StreetSign",
+            "Clutter",
+            "Fence",
+        ]
+        VALID_CLASS_IDS = np.array(
+            [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
+        )
+
+        ID_TO_LABEL = {}
+        LABEL_TO_ID = {}
+        for i in range(len(VALID_CLASS_IDS)):
+            LABEL_TO_ID[CLASS_LABELS[i]] = VALID_CLASS_IDS[i]
+            ID_TO_LABEL[VALID_CLASS_IDS[i]] = CLASS_LABELS[i]
+
+    if dataset == "s3dis":
+        # global CLASS_LABELS
+        # global VALID_CLASS_IDS
+        # global ID_TO_LABEL
+        # global LABEL_TO_ID
+
+        CLASS_LABELS = [
+            "ceiling",
+            "floor",
+            "wall",
+            "beam",
+            "column",
+            "window",
+            "door",
+            "table",
+            "chair",
+            "sofa",
+            "bookcase",
+            "board",
+            "clutter",
+        ]
+        VALID_CLASS_IDS = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13])
+        ID_TO_LABEL = {}
+        LABEL_TO_ID = {}
+        for i in range(len(VALID_CLASS_IDS)):
+            LABEL_TO_ID[CLASS_LABELS[i]] = VALID_CLASS_IDS[i]
+            ID_TO_LABEL[VALID_CLASS_IDS[i]] = CLASS_LABELS[i]
+
+    if dataset == "scannet200":
+        CLASS_LABELS = (
+            "chair",
+            "table",
+            "door",
+            "couch",
+            "cabinet",
+            "shelf",
+            "desk",
+            "office chair",
+            "bed",
+            "pillow",
+            "sink",
+            "picture",
+            "window",
+            "toilet",
+            "bookshelf",
+            "monitor",
+            "curtain",
+            "book",
+            "armchair",
+            "coffee table",
+            "box",
+            "refrigerator",
+            "lamp",
+            "kitchen cabinet",
+            "towel",
+            "clothes",
+            "tv",
+            "nightstand",
+            "counter",
+            "dresser",
+            "stool",
+            "cushion",
+            "plant",
+            "ceiling",
+            "bathtub",
+            "end table",
+            "dining table",
+            "keyboard",
+            "bag",
+            "backpack",
+            "toilet paper",
+            "printer",
+            "tv stand",
+            "whiteboard",
+            "blanket",
+            "shower curtain",
+            "trash can",
+            "closet",
+            "stairs",
+            "microwave",
+            "stove",
+            "shoe",
+            "computer tower",
+            "bottle",
+            "bin",
+            "ottoman",
+            "bench",
+            "board",
+            "washing machine",
+            "mirror",
+            "copier",
+            "basket",
+            "sofa chair",
+            "file cabinet",
+            "fan",
+            "laptop",
+            "shower",
+            "paper",
+            "person",
+            "paper towel dispenser",
+            "oven",
+            "blinds",
+            "rack",
+            "plate",
+            "blackboard",
+            "piano",
+            "suitcase",
+            "rail",
+            "radiator",
+            "recycling bin",
+            "container",
+            "wardrobe",
+            "soap dispenser",
+            "telephone",
+            "bucket",
+            "clock",
+            "stand",
+            "light",
+            "laundry basket",
+            "pipe",
+            "clothes dryer",
+            "guitar",
+            "toilet paper holder",
+            "seat",
+            "speaker",
+            "column",
+            "bicycle",
+            "ladder",
+            "bathroom stall",
+            "shower wall",
+            "cup",
+            "jacket",
+            "storage bin",
+            "coffee maker",
+            "dishwasher",
+            "paper towel roll",
+            "machine",
+            "mat",
+            "windowsill",
+            "bar",
+            "toaster",
+            "bulletin board",
+            "ironing board",
+            "fireplace",
+            "soap dish",
+            "kitchen counter",
+            "doorframe",
+            "toilet paper dispenser",
+            "mini fridge",
+            "fire extinguisher",
+            "ball",
+            "hat",
+            "shower curtain rod",
+            "water cooler",
+            "paper cutter",
+            "tray",
+            "shower door",
+            "pillar",
+            "ledge",
+            "toaster oven",
+            "mouse",
+            "toilet seat cover dispenser",
+            "furniture",
+            "cart",
+            "storage container",
+            "scale",
+            "tissue box",
+            "light switch",
+            "crate",
+            "power outlet",
+            "decoration",
+            "sign",
+            "projector",
+            "closet door",
+            "vacuum cleaner",
+            "candle",
+            "plunger",
+            "stuffed animal",
+            "headphones",
+            "dish rack",
+            "broom",
+            "guitar case",
+            "range hood",
+            "dustpan",
+            "hair dryer",
+            "water bottle",
+            "handicap bar",
+            "purse",
+            "vent",
+            "shower floor",
+            "water pitcher",
+            "mailbox",
+            "bowl",
+            "paper bag",
+            "alarm clock",
+            "music stand",
+            "projector screen",
+            "divider",
+            "laundry detergent",
+            "bathroom counter",
+            "object",
+            "bathroom vanity",
+            "closet wall",
+            "laundry hamper",
+            "bathroom stall door",
+            "ceiling light",
+            "trash bin",
+            "dumbbell",
+            "stair rail",
+            "tube",
+            "bathroom cabinet",
+            "cd case",
+            "closet rod",
+            "coffee kettle",
+            "structure",
+            "shower head",
+            "keyboard piano",
+            "case of water bottles",
+            "coat rack",
+            "storage organizer",
+            "folded chair",
+            "fire alarm",
+            "power strip",
+            "calendar",
+            "poster",
+            "potted plant",
+            "luggage",
+            "mattress",
+        )
+
+        VALID_CLASS_IDS = np.array(
+            (
+                2,
+                4,
+                5,
+                6,
+                7,
+                8,
+                9,
+                10,
+                11,
+                13,
+                14,
+                15,
+                16,
+                17,
+                18,
+                19,
+                21,
+                22,
+                23,
+                24,
+                26,
+                27,
+                28,
+                29,
+                31,
+                32,
+                33,
+                34,
+                35,
+                36,
+                38,
+                39,
+                40,
+                41,
+                42,
+                44,
+                45,
+                46,
+                47,
+                48,
+                49,
+                50,
+                51,
+                52,
+                54,
+                55,
+                56,
+                57,
+                58,
+                59,
+                62,
+                63,
+                64,
+                65,
+                66,
+                67,
+                68,
+                69,
+                70,
+                71,
+                72,
+                73,
+                74,
+                75,
+                76,
+                77,
+                78,
+                79,
+                80,
+                82,
+                84,
+                86,
+                87,
+                88,
+                89,
+                90,
+                93,
+                95,
+                96,
+                97,
+                98,
+                99,
+                100,
+                101,
+                102,
+                103,
+                104,
+                105,
+                106,
+                107,
+                110,
+                112,
+                115,
+                116,
+                118,
+                120,
+                121,
+                122,
+                125,
+                128,
+                130,
+                131,
+                132,
+                134,
+                136,
+                138,
+                139,
+                140,
+                141,
+                145,
+                148,
+                154,
+                155,
+                156,
+                157,
+                159,
+                161,
+                163,
+                165,
+                166,
+                168,
+                169,
+                170,
+                177,
+                180,
+                185,
+                188,
+                191,
+                193,
+                195,
+                202,
+                208,
+                213,
+                214,
+                221,
+                229,
+                230,
+                232,
+                233,
+                242,
+                250,
+                261,
+                264,
+                276,
+                283,
+                286,
+                300,
+                304,
+                312,
+                323,
+                325,
+                331,
+                342,
+                356,
+                370,
+                392,
+                395,
+                399,
+                408,
+                417,
+                488,
+                540,
+                562,
+                570,
+                572,
+                581,
+                609,
+                748,
+                776,
+                1156,
+                1163,
+                1164,
+                1165,
+                1166,
+                1167,
+                1168,
+                1169,
+                1170,
+                1171,
+                1172,
+                1173,
+                1174,
+                1175,
+                1176,
+                1178,
+                1179,
+                1180,
+                1181,
+                1182,
+                1183,
+                1184,
+                1185,
+                1186,
+                1187,
+                1188,
+                1189,
+                1190,
+                1191,
+            )
+        )
+
+        ID_TO_LABEL = {}
+        LABEL_TO_ID = {}
+        for i in range(len(VALID_CLASS_IDS)):
+            LABEL_TO_ID[CLASS_LABELS[i]] = VALID_CLASS_IDS[i]
+            ID_TO_LABEL[VALID_CLASS_IDS[i]] = CLASS_LABELS[i]
+
+    total_true = 0
+    total_seen = 0
+    NUM_CLASSES = len(VALID_CLASS_IDS)
+
+    true_positive_classes = np.zeros(NUM_CLASSES)
+    positive_classes = np.zeros(NUM_CLASSES)
+    gt_classes = np.zeros(NUM_CLASSES)
+
+    # precision & recall
+    total_gt_ins = np.zeros(NUM_CLASSES)
+    at = 0.5
+    tpsins = [[] for _ in range(NUM_CLASSES)]
+    fpsins = [[] for _ in range(NUM_CLASSES)]
+    # mucov and mwcov
+    all_mean_cov = [[] for _ in range(NUM_CLASSES)]
+    all_mean_weighted_cov = [[] for _ in range(NUM_CLASSES)]
+
+    print("evaluating", len(preds), "scans...")
+    matches = {}
+    for i, (k, v) in enumerate(preds.items()):
+        gt_file = os.path.join(gt_path, k + ".txt")
+        if not os.path.isfile(gt_file):
+            util.print_error(
+                "Scan {} does not match any gt file".format(k), user_fault=True
+            )
+
+        if dataset == "s3dis":
+            gt_ids = util_3d.load_ids(gt_file)
+            gt_sem = (gt_ids // 1000) - 1
+            gt_ins = gt_ids - (gt_ids // 1000) * 1000
+
+            # pred_sem = v['pred_classes'] - 1
+            pred_sem = np.zeros(v["pred_masks"].shape[0], dtype=np.int)
+            # TODO CONTINUE HERE!!!!!!!!!!!!!
+            pred_ins = np.zeros(v["pred_masks"].shape[0], dtype=np.int)
+
+            for inst_id in reversed(range(v["pred_masks"].shape[1])):
+                point_ids = np.argwhere(v["pred_masks"][:, inst_id] == 1.0)[
+                    :, 0
+                ]
+                pred_ins[point_ids] = inst_id + 1
+                pred_sem[point_ids] = v["pred_classes"][inst_id] - 1
+
+            # semantic acc
+            total_true += np.sum(pred_sem == gt_sem)
+            total_seen += pred_sem.shape[0]
+
+            # TODO PARALLELIZ THIS!!!!!!!
+            # pn semantic mIoU
+            """
+            for j in range(gt_sem.shape[0]):
+                gt_l = int(gt_sem[j])
+                pred_l = int(pred_sem[j])
+                gt_classes[gt_l] += 1
+                positive_classes[pred_l] += 1
+                true_positive_classes[gt_l] += int(gt_l == pred_l)
+            """
+
+            uniq, counts = np.unique(pred_sem, return_counts=True)
+            positive_classes[uniq] += counts
+
+            uniq, counts = np.unique(gt_sem, return_counts=True)
+            gt_classes[uniq] += counts
+
+            uniq, counts = np.unique(
+                gt_sem[pred_sem == gt_sem], return_counts=True
+            )
+            true_positive_classes[uniq] += counts
+
+            # instance
+            un = np.unique(pred_ins)
+            pts_in_pred = [[] for _ in range(NUM_CLASSES)]
+            for ig, g in enumerate(un):  # each object in prediction
+                if g == -1:
+                    continue
+                tmp = pred_ins == g
+                sem_seg_i = int(stats.mode(pred_sem[tmp])[0])
+                pts_in_pred[sem_seg_i] += [tmp]
+
+            un = np.unique(gt_ins)
+            pts_in_gt = [[] for _ in range(NUM_CLASSES)]
+            for ig, g in enumerate(un):
+                tmp = gt_ins == g
+                sem_seg_i = int(stats.mode(gt_sem[tmp])[0])
+                pts_in_gt[sem_seg_i] += [tmp]
+
+            # instance mucov & mwcov
+            for i_sem in range(NUM_CLASSES):
+                sum_cov = 0
+                mean_cov = 0
+                mean_weighted_cov = 0
+                num_gt_point = 0
+                for ig, ins_gt in enumerate(pts_in_gt[i_sem]):
+                    ovmax = 0.0
+                    num_ins_gt_point = np.sum(ins_gt)
+                    num_gt_point += num_ins_gt_point
+                    for ip, ins_pred in enumerate(pts_in_pred[i_sem]):
+                        union = ins_pred | ins_gt
+                        intersect = ins_pred & ins_gt
+                        iou = float(np.sum(intersect)) / np.sum(union)
+
+                        if iou > ovmax:
+                            ovmax = iou
+                            ipmax = ip
+
+                    sum_cov += ovmax
+                    mean_weighted_cov += ovmax * num_ins_gt_point
+
+                if len(pts_in_gt[i_sem]) != 0:
+                    mean_cov = sum_cov / len(pts_in_gt[i_sem])
+                    all_mean_cov[i_sem].append(mean_cov)
+
+                    mean_weighted_cov /= num_gt_point
+                    all_mean_weighted_cov[i_sem].append(mean_weighted_cov)
+
+        if dataset == "s3dis":
+            # instance precision & recall
+            for i_sem in range(NUM_CLASSES):
+                tp = [0.0] * len(pts_in_pred[i_sem])
+                fp = [0.0] * len(pts_in_pred[i_sem])
+                gtflag = np.zeros(len(pts_in_gt[i_sem]))
+                total_gt_ins[i_sem] += len(pts_in_gt[i_sem])
+
+                for ip, ins_pred in enumerate(pts_in_pred[i_sem]):
+                    ovmax = -1.0
+
+                    for ig, ins_gt in enumerate(pts_in_gt[i_sem]):
+                        union = ins_pred | ins_gt
+                        intersect = ins_pred & ins_gt
+                        iou = float(np.sum(intersect)) / np.sum(union)
+
+                        if iou > ovmax:
+                            ovmax = iou
+                            igmax = ig
+
+                    if ovmax >= at:
+                        tp[ip] = 1  # true
+                    else:
+                        fp[ip] = 1  # false positive
+
+                tpsins[i_sem] += tp
+                fpsins[i_sem] += fp
+
+        matches_key = os.path.abspath(gt_file)
+        # assign gt to predictions
+        gt2pred, pred2gt = assign_instances_for_scan(v, gt_file)
+        matches[matches_key] = {}
+        matches[matches_key]["gt"] = gt2pred
+        matches[matches_key]["pred"] = pred2gt
+        sys.stdout.write("\rscans processed: {}".format(i + 1))
+        sys.stdout.flush()
+    print("")
+    ap_scores = evaluate_matches(matches)
+    avgs = compute_averages(ap_scores)
+
+    # print
+    print_results(avgs)
+    write_result_file(avgs, output_file)
+
+    if dataset == "s3dis":
+        MUCov = np.zeros(NUM_CLASSES)
+        MWCov = np.zeros(NUM_CLASSES)
+        for i_sem in range(NUM_CLASSES):
+            MUCov[i_sem] = np.mean(all_mean_cov[i_sem])
+            MWCov[i_sem] = np.mean(all_mean_weighted_cov[i_sem])
+
+        precision = np.zeros(NUM_CLASSES)
+        recall = np.zeros(NUM_CLASSES)
+        for i_sem in range(NUM_CLASSES):
+            tp = np.asarray(tpsins[i_sem]).astype(np.float)
+            fp = np.asarray(fpsins[i_sem]).astype(np.float)
+            tp = np.sum(tp)
+            fp = np.sum(fp)
+            rec = tp / total_gt_ins[i_sem]
+            prec = tp / (tp + fp)
+
+            precision[i_sem] = prec
+            recall[i_sem] = rec
+
+        """
+        LOG_FOUT = open(os.path.join('results_a5.txt'), 'w')
+    
+        def log_string(out_str):
+            LOG_FOUT.write(out_str + '\n')
+            LOG_FOUT.flush()
+            print(out_str)
+        """
+
+        return np.mean(precision), np.mean(recall)
+
+
+# TODO: remove this
+# import pandas as pd
+# def main():
+#    print("!!! CLI is only for debugging purposes. use `evaluate()` instead.")
+#    evaluate(pd.read_pickle("/globalwork/schult/saved_predictions.pkl"), opt.gt_path, opt.output_file)
+
+# if __name__ == '__main__':
+#    main()

models/Mask3D/build/lib/mask3d/benchmark/util.py

@@ -0,0 +1,128 @@
+import os, sys
+import csv
+
+try:
+    import numpy as np
+except:
+    print("Failed to import numpy package.")
+    sys.exit(-1)
+try:
+    import imageio
+except:
+    print("Please install the module 'imageio' for image processing, e.g.")
+    print("pip install imageio")
+    sys.exit(-1)
+
+# print an error message and quit
+def print_error(message, user_fault=False):
+    sys.stderr.write("ERROR: " + str(message) + "\n")
+    if user_fault:
+        sys.exit(2)
+    sys.exit(-1)
+
+
+# if string s represents an int
+def represents_int(s):
+    try:
+        int(s)
+        return True
+    except ValueError:
+        return False
+
+
+def read_label_mapping(
+    filename, label_from="raw_category", label_to="nyu40id"
+):
+    assert os.path.isfile(filename)
+    mapping = dict()
+    with open(filename) as csvfile:
+        reader = csv.DictReader(csvfile, delimiter="\t")
+        for row in reader:
+            mapping[row[label_from]] = int(row[label_to])
+    # if ints convert
+    if represents_int(list(mapping.keys())[0]):
+        mapping = {int(k): v for k, v in mapping.items()}
+    return mapping
+
+
+# input: scene_types.txt or scene_types_all.txt
+def read_scene_types_mapping(filename, remove_spaces=True):
+    assert os.path.isfile(filename)
+    mapping = dict()
+    lines = open(filename).read().splitlines()
+    lines = [line.split("\t") for line in lines]
+    if remove_spaces:
+        mapping = {x[1].strip(): int(x[0]) for x in lines}
+    else:
+        mapping = {x[1]: int(x[0]) for x in lines}
+    return mapping
+
+
+# color by label
+def visualize_label_image(filename, image):
+    height = image.shape[0]
+    width = image.shape[1]
+    vis_image = np.zeros([height, width, 3], dtype=np.uint8)
+    color_palette = create_color_palette()
+    for idx, color in enumerate(color_palette):
+        vis_image[image == idx] = color
+    imageio.imwrite(filename, vis_image)
+
+
+# color by different instances (mod length of color palette)
+def visualize_instance_image(filename, image):
+    height = image.shape[0]
+    width = image.shape[1]
+    vis_image = np.zeros([height, width, 3], dtype=np.uint8)
+    color_palette = create_color_palette()
+    instances = np.unique(image)
+    for idx, inst in enumerate(instances):
+        vis_image[image == inst] = color_palette[inst % len(color_palette)]
+    imageio.imwrite(filename, vis_image)
+
+
+# color palette for nyu40 labels
+def create_color_palette():
+    return [
+        (0, 0, 0),
+        (174, 199, 232),  # wall
+        (152, 223, 138),  # floor
+        (31, 119, 180),  # cabinet
+        (255, 187, 120),  # bed
+        (188, 189, 34),  # chair
+        (140, 86, 75),  # sofa
+        (255, 152, 150),  # table
+        (214, 39, 40),  # door
+        (197, 176, 213),  # window
+        (148, 103, 189),  # bookshelf
+        (196, 156, 148),  # picture
+        (23, 190, 207),  # counter
+        (178, 76, 76),
+        (247, 182, 210),  # desk
+        (66, 188, 102),
+        (219, 219, 141),  # curtain
+        (140, 57, 197),
+        (202, 185, 52),
+        (51, 176, 203),
+        (200, 54, 131),
+        (92, 193, 61),
+        (78, 71, 183),
+        (172, 114, 82),
+        (255, 127, 14),  # refrigerator
+        (91, 163, 138),
+        (153, 98, 156),
+        (140, 153, 101),
+        (158, 218, 229),  # shower curtain
+        (100, 125, 154),
+        (178, 127, 135),
+        (120, 185, 128),
+        (146, 111, 194),
+        (44, 160, 44),  # toilet
+        (112, 128, 144),  # sink
+        (96, 207, 209),
+        (227, 119, 194),  # bathtub
+        (213, 92, 176),
+        (94, 106, 211),
+        (82, 84, 163),  # otherfurn
+        (100, 85, 144),
+    ]

models/Mask3D/build/lib/mask3d/benchmark/util_3d.py

@@ -0,0 +1,177 @@
+import os, sys
+import json
+
+try:
+    import numpy as np
+except:
+    print("Failed to import numpy package.")
+    sys.exit(-1)
+
+try:
+    from plyfile import PlyData, PlyElement
+except:
+    print("Please install the module 'plyfile' for PLY i/o, e.g.")
+    print("pip install plyfile")
+    sys.exit(-1)
+
+import benchmark.util as util
+
+
+# matrix: 4x4 np array
+# points Nx3 np array
+def transform_points(matrix, points):
+    assert len(points.shape) == 2 and points.shape[1] == 3
+    num_points = points.shape[0]
+    p = np.concatenate([points, np.ones((num_points, 1))], axis=1)
+    p = np.matmul(matrix, np.transpose(p))
+    p = np.transpose(p)
+    p[:, :3] /= p[:, 3, None]
+    return p[:, :3]
+
+
+def export_ids(filename, ids):
+    with open(filename, "w") as f:
+        for id in ids:
+            f.write("%d\n" % id)
+
+
+def load_ids(filename):
+    ids = open(filename).read().splitlines()
+    ids = np.array(ids, dtype=np.int64)
+    return ids
+
+
+def read_mesh_vertices(filename):
+    assert os.path.isfile(filename)
+    with open(filename, "rb") as f:
+        plydata = PlyData.read(f)
+        num_verts = plydata["vertex"].count
+        vertices = np.zeros(shape=[num_verts, 3], dtype=np.float32)
+        vertices[:, 0] = plydata["vertex"].data["x"]
+        vertices[:, 1] = plydata["vertex"].data["y"]
+        vertices[:, 2] = plydata["vertex"].data["z"]
+    return vertices
+
+
+# export 3d instance labels for instance evaluation
+def export_instance_ids_for_eval(filename, label_ids, instance_ids):
+    assert label_ids.shape[0] == instance_ids.shape[0]
+    output_mask_path_relative = "pred_mask"
+    name = os.path.splitext(os.path.basename(filename))[0]
+    output_mask_path = os.path.join(
+        os.path.dirname(filename), output_mask_path_relative
+    )
+    if not os.path.isdir(output_mask_path):
+        os.mkdir(output_mask_path)
+    insts = np.unique(instance_ids)
+    zero_mask = np.zeros(shape=(instance_ids.shape[0]), dtype=np.int32)
+    with open(filename, "w") as f:
+        for idx, inst_id in enumerate(insts):
+            if inst_id == 0:  # 0 -> no instance for this vertex
+                continue
+            output_mask_file = os.path.join(
+                output_mask_path_relative, name + "_" + str(idx) + ".txt"
+            )
+            loc = np.where(instance_ids == inst_id)
+            label_id = label_ids[loc[0][0]]
+            f.write("%s %d %f\n" % (output_mask_file, label_id, 1.0))
+            # write mask
+            mask = np.copy(zero_mask)
+            mask[loc[0]] = 1
+            export_ids(output_mask_file, mask)
+
+
+# ------------ Instance Utils ------------ #
+
+
+class Instance(object):
+    instance_id = 0
+    label_id = 0
+    vert_count = 0
+    med_dist = -1
+    dist_conf = 0.0
+
+    def __init__(self, mesh_vert_instances, instance_id):
+        if instance_id == -1:
+            return
+        self.instance_id = int(instance_id)
+        self.label_id = int(self.get_label_id(instance_id))
+        self.vert_count = int(
+            self.get_instance_verts(mesh_vert_instances, instance_id)
+        )
+
+    def get_label_id(self, instance_id):
+        return int(instance_id // 1000)
+
+    def get_instance_verts(self, mesh_vert_instances, instance_id):
+        return (mesh_vert_instances == instance_id).sum()
+
+    def to_json(self):
+        return json.dumps(
+            self, default=lambda o: o.__dict__, sort_keys=True, indent=4
+        )
+
+    def to_dict(self):
+        dict = {}
+        dict["instance_id"] = self.instance_id
+        dict["label_id"] = self.label_id
+        dict["vert_count"] = self.vert_count
+        dict["med_dist"] = self.med_dist
+        dict["dist_conf"] = self.dist_conf
+        return dict
+
+    def from_json(self, data):
+        self.instance_id = int(data["instance_id"])
+        self.label_id = int(data["label_id"])
+        self.vert_count = int(data["vert_count"])
+        if "med_dist" in data:
+            self.med_dist = float(data["med_dist"])
+            self.dist_conf = float(data["dist_conf"])
+
+    def __str__(self):
+        return "(" + str(self.instance_id) + ")"
+
+
+def read_instance_prediction_file(filename, pred_path):
+    lines = open(filename).read().splitlines()
+    instance_info = {}
+    abs_pred_path = os.path.abspath(pred_path)
+    for line in lines:
+        parts = line.split(" ")
+        if len(parts) != 3:
+            util.print_error(
+                "invalid instance prediction file. Expected (per line): [rel path prediction] [label id prediction] [confidence prediction]"
+            )
+        if os.path.isabs(parts[0]):
+            util.print_error(
+                "invalid instance prediction file. First entry in line must be a relative path"
+            )
+        mask_file = os.path.join(os.path.dirname(filename), parts[0])
+        mask_file = os.path.abspath(mask_file)
+        # check that mask_file lives inside prediction path
+        if os.path.commonprefix([mask_file, abs_pred_path]) != abs_pred_path:
+            util.print_error(
+                "predicted mask {} in prediction text file {} points outside of prediction path.".format(
+                    mask_file, filename
+                )
+            )
+
+        info = {}
+        info["label_id"] = int(float(parts[1]))
+        info["conf"] = float(parts[2])
+        instance_info[mask_file] = info
+    return instance_info
+
+
+def get_instances(ids, class_ids, class_labels, id2label):
+    instances = {}
+    for label in class_labels:
+        instances[label] = []
+    instance_ids = np.unique(ids)
+    for id in instance_ids:
+        if id == 0:
+            continue
+        inst = Instance(ids, id)
+        if inst.label_id in class_ids:
+            instances[id2label[inst.label_id]].append(inst.to_dict())
+    return instances

models/Mask3D/build/lib/mask3d/conf/augmentation/albumentations_aug.yaml

@@ -0,0 +1,30 @@
+__version__: 0.4.5
+transform:
+  __class_fullname__: albumentations.core.composition.Compose
+  additional_targets: {}
+  bbox_params: null
+  keypoint_params: null
+  p: 1.0
+  transforms:
+    - __class_fullname__: albumentations.augmentations.transforms.RandomBrightnessContrast
+      always_apply: true
+      brightness_by_max: true
+      brightness_limit:
+        - -0.2
+        - 0.2
+      contrast_limit:
+        - -0.2
+        - 0.2
+      p: 0.5
+    - __class_fullname__: albumentations.augmentations.transforms.RGBShift
+      always_apply: true
+      b_shift_limit:
+        - -20
+        - 20
+      g_shift_limit:
+        - -20
+        - 20
+      p: 0.5
+      r_shift_limit:
+        - -20
+        - 20

models/Mask3D/build/lib/mask3d/conf/augmentation/volumentations_aug.yaml

@@ -0,0 +1,53 @@
+# pi   = 3.14159265358979
+# pi/2 = 1.57079632679489
+# pi/3 = 1.04719755119659
+# pi/6 = 0.52359877559829
+# pi/12 = 0.26179938779914
+# pi/24 = 0.13089969389957
+#
+__version__: 0.1.6
+transform:
+  __class_fullname__: volumentations.core.composition.Compose
+  additional_targets: {}
+  p: 1.0
+  transforms:
+    - __class_fullname__: volumentations.augmentations.transforms.Scale3d
+      always_apply: true
+      p: 0.5
+      scale_limit:
+        - - -0.1
+          - 0.1
+        - - -0.1
+          - 0.1
+        - - -0.1
+          - 0.1
+    - __class_fullname__: volumentations.augmentations.transforms.RotateAroundAxis3d
+      always_apply: true
+      axis:
+        - 0
+        - 0
+        - 1
+      p: 0.5
+      rotation_limit:
+        - -3.141592653589793
+        - 3.141592653589793
+    - __class_fullname__: volumentations.augmentations.transforms.RotateAroundAxis3d
+      always_apply: true
+      axis:
+        - 0
+        - 1
+        - 0
+      p: 0.5
+      rotation_limit:
+        - -0.13089969389957
+        - 0.13089969389957
+    - __class_fullname__: volumentations.augmentations.transforms.RotateAroundAxis3d
+      always_apply: true
+      axis:
+        - 1
+        - 0
+        - 0
+      p: 0.5
+      rotation_limit:
+        - -0.13089969389957
+        - 0.13089969389957

models/Mask3D/build/lib/mask3d/conf/callbacks/callbacks_instance_segmentation.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+- _target_: pytorch_lightning.callbacks.ModelCheckpoint
+  monitor: val_mean_ap_50
+  save_last: true
+  save_top_k: 1
+  mode: max
+  dirpath: ${general.save_dir}
+  filename: "{epoch}-{val_mean_ap_50:.3f}"
+  every_n_epochs: 1
+
+- _target_: pytorch_lightning.callbacks.LearningRateMonitor

models/Mask3D/build/lib/mask3d/conf/config_base_instance_segmentation.yaml

@@ -0,0 +1,75 @@
+general:
+  train_mode: true
+  task: "instance_segmentation"
+  seed: null
+  checkpoint: null
+  backbone_checkpoint: null
+  freeze_backbone: false # train only last layer
+  linear_probing_backbone: false
+  train_on_segments: false
+  eval_on_segments: false
+  filter_out_instances: false
+  save_visualizations: false
+  visualization_point_size: 20
+  decoder_id: -1
+  export: false
+  use_dbscan: false
+  ignore_class_threshold: 100
+  project_name: scannet
+  workspace: jonasschult
+  experiment_name: DEBUG_ABLATION
+  num_targets: 19
+  add_instance: true
+  dbscan_eps: 0.95
+  dbscan_min_points: 1
+
+
+  export_threshold: 0.0001
+
+  reps_per_epoch: 1
+
+  on_crops: false
+
+  scores_threshold: 0.0
+  iou_threshold: 1.0
+
+  area: 5
+
+  eval_inner_core: -1 # disabled
+
+  topk_per_image: 100
+
+  ignore_mask_idx: []
+
+  max_batch_size: 99999999
+
+  save_dir: saved/${general.experiment_name}
+  # time/commit/md5(config)_uuid
+  # time/experiment_id/version_uuid
+  # experiment_id: 1 # commit[:8], or unique from logger
+  # version: 1 # md5[:8] of config
+
+  gpus: 1
+
+defaults:
+  - data: indoor
+  - data/data_loaders: simple_loader
+  - data/datasets: scannet
+  - data/collation_functions: voxelize_collate
+  - logging: full
+  - model: mask3d
+  - metrics: miou
+  - optimizer: adamw
+  - scheduler: onecyclelr
+  - trainer: trainer600
+  - callbacks: callbacks_instance_segmentation
+  - matcher: hungarian_matcher
+  - loss: set_criterion
+
+hydra:
+  run:
+    dir: saved/hydra_logs/${now:%Y-%m-%d}/${now:%H-%M-%S}
+  sweep:
+    dir: saved/hydra_logs/${now:%Y-%m-%d}/${now:%H-%M-%S}
+    # dir: ${general.save_dir}
+    subdir: ${hydra.job.num}_${hydra.job.id}

models/Mask3D/build/lib/mask3d/conf/data/collation_functions/voxelize_collate.yaml

@@ -0,0 +1,42 @@
+# @package data
+
+train_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollate
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.train_mode}
+  small_crops: false
+  very_small_crops: false
+  batch_instance: false
+  probing: ${general.linear_probing_backbone}
+  task: ${general.task}
+  ignore_class_threshold: ${general.ignore_class_threshold}
+  filter_out_classes: ${data.train_dataset.filter_out_classes}
+  label_offset: ${data.train_dataset.label_offset}
+  num_queries: ${model.num_queries}
+
+validation_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollate
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.validation_mode}
+  batch_instance: false
+  probing: ${general.linear_probing_backbone}
+  task: ${general.task}
+  ignore_class_threshold: ${general.ignore_class_threshold}
+  filter_out_classes: ${data.validation_dataset.filter_out_classes}
+  label_offset: ${data.validation_dataset.label_offset}
+  num_queries: ${model.num_queries}
+
+test_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollate
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.test_mode}
+  batch_instance: false
+  probing: ${general.linear_probing_backbone}
+  task: ${general.task}
+  ignore_class_threshold: ${general.ignore_class_threshold}
+  filter_out_classes: ${data.test_dataset.filter_out_classes}
+  label_offset: ${data.test_dataset.label_offset}
+  num_queries: ${model.num_queries}

models/Mask3D/build/lib/mask3d/conf/data/collation_functions/voxelize_collate_merge.yaml

@@ -0,0 +1,36 @@
+# @package data
+
+train_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollateMerge
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.train_mode}
+  small_crops: false
+  very_small_crops: false
+  scenes: 2
+  batch_instance: false
+  make_one_pc_noise: false
+  place_nearby: false
+  place_far: false
+  proba: 1
+  probing: ${general.linear_probing_backbone}
+  include_ignore: ${general.include_ignore}
+  task: ${general.task}
+
+validation_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollate
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.validation_mode}
+  probing: ${general.linear_probing_backbone}
+  include_ignore: ${general.include_ignore}
+  task: ${general.task}
+
+test_collation:
+  _target_: mask3d.datasets.utils.VoxelizeCollate
+  ignore_label: ${data.ignore_label}
+  voxel_size: ${data.voxel_size}
+  mode: ${data.test_mode}
+  probing: ${general.linear_probing_backbone}
+  include_ignore: ${general.include_ignore}
+  task: ${general.task}

models/Mask3D/build/lib/mask3d/conf/data/data_loaders/simple_loader.yaml

@@ -0,0 +1,22 @@
+# @package data
+
+train_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: true
+  pin_memory: ${data.pin_memory}
+  num_workers: ${data.num_workers}
+  batch_size: ${data.batch_size}
+
+validation_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: false
+  pin_memory: ${data.pin_memory}
+  num_workers: ${data.num_workers}
+  batch_size: ${data.test_batch_size}
+
+test_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: false
+  pin_memory: ${data.pin_memory}
+  num_workers: ${data.num_workers}
+  batch_size: ${data.test_batch_size}

models/Mask3D/build/lib/mask3d/conf/data/data_loaders/simple_loader_save_memory.yaml

@@ -0,0 +1,22 @@
+# @package data
+
+train_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: true
+  pin_memory: ${data.pin_memory}
+  num_workers: ${data.num_workers}
+  batch_size: ${data.batch_size}
+
+validation_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: false
+  pin_memory: ${data.pin_memory}
+  num_workers: 1
+  batch_size: ${data.test_batch_size}
+
+test_dataloader:
+  _target_: torch.utils.data.DataLoader
+  shuffle: false
+  pin_memory: ${data.pin_memory}
+  num_workers: 1
+  batch_size: ${data.test_batch_size}

models/Mask3D/build/lib/mask3d/conf/data/datasets/matterport.yaml

@@ -0,0 +1,48 @@
+# @package data
+train_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/matterport
+  image_augmentations_path: mix3d/conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: mix3d/conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/matterport/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+validation_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/scannet
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/matterport/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+test_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/matterport
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/matterport/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}

models/Mask3D/build/lib/mask3d/conf/data/datasets/matterport_scannet.yaml

@@ -0,0 +1,50 @@
+# @package data
+train_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir:
+    - data/processed/scannet
+    - data/processed/matterport
+  image_augmentations_path: mix3d/conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: mix3d/conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+validation_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/scannet
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+test_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/scannet
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}

models/Mask3D/build/lib/mask3d/conf/data/datasets/rio.yaml

@@ -0,0 +1,48 @@
+# @package data
+train_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/rio
+  image_augmentations_path: mix3d/conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: mix3d/conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+validation_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/rio
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+
+test_dataset:
+  _target_: mix3d.datasets.semseg.SemanticSegmentationDataset
+  data_dir: data/processed/rio
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}

models/Mask3D/build/lib/mask3d/conf/data/datasets/s3dis.yaml

@@ -0,0 +1,87 @@
+# @package data
+train_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "s3dis"
+  data_dir: data/processed/s3dis
+  image_augmentations_path: conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/s3dis/label_database.yaml
+  color_mean_std: data/processed/s3dis/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  # different augs experiments
+  instance_oversampling: 0.0
+  place_around_existing: False
+  point_per_cut: 0
+  max_cut_region: 0
+  flip_in_center: false
+  noise_rate: 0
+  resample_points: 0
+  cropping: ${data.cropping}
+  cropping_args: ${data.cropping_args}
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  filter_out_classes: []
+  label_offset: 0
+
+validation_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "s3dis"
+  data_dir: data/processed/s3dis
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/s3dis/label_database.yaml
+  color_mean_std: data/processed/s3dis/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  filter_out_classes: []
+  label_offset: 0
+
+test_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "s3dis"
+  data_dir: data/processed/s3dis
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/s3dis/label_database.yaml
+  color_mean_std: data/processed/s3dis/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  filter_out_classes: []
+  label_offset: 0

models/Mask3D/build/lib/mask3d/conf/data/datasets/scannet.yaml

@@ -0,0 +1,79 @@
+# @package data
+train_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet"
+  data_dir: data/processed/scannet
+  image_augmentations_path: conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  # different augs experiments
+  instance_oversampling: 0.0
+  place_around_existing: false
+  point_per_cut: 0
+  max_cut_region: 0
+  flip_in_center: false
+  noise_rate: 0
+  resample_points: 0
+  add_unlabeled_pc: false
+  cropping: ${data.cropping}
+  cropping_args: ${data.cropping_args}
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 1]
+  label_offset: 2
+
+validation_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet"
+  data_dir: data/processed/scannet
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 1]
+  label_offset: 2
+
+test_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet"
+  data_dir: data/processed/scannet
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/scannet/label_database.yaml
+  color_mean_std: data/processed/scannet/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 1]
+  label_offset: 2

models/Mask3D/build/lib/mask3d/conf/data/datasets/scannet200.yaml

@@ -0,0 +1,79 @@
+# @package data
+train_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet200"
+  data_dir: /home/weders/scratch/scratch/scannetter/arkit/raw/
+  image_augmentations_path: conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: conf/augmentation/volumentations_aug.yaml
+  # label_db_filepath: data/processed/scannet200/label_database.yaml
+  # color_mean_std: data/processed/scannet200/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  # different augs experiments
+  instance_oversampling: 0.0
+  place_around_existing: false
+  point_per_cut: 0
+  max_cut_region: 0
+  flip_in_center: false
+  noise_rate: 0
+  resample_points: 0
+  add_unlabeled_pc: false
+  cropping: ${data.cropping}
+  cropping_args: ${data.cropping_args}
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 2]
+  label_offset: 2
+
+validation_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet200"
+  data_dir: /home/weders/scratch/scratch/scannetter/arkit/raw/
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  # label_db_filepath: data/processed/scannet200/label_database.yaml
+  # color_mean_std: data/processed/scannet200/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 2]
+  label_offset: 2
+
+test_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "scannet200"
+  data_dir: /home/weders/scratch/scratch/scannetter/arkit/raw/
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  # label_db_filepath: data/processed/scannet200/label_database.yaml
+  # color_mean_std: data/processed/scannet200/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  filter_out_classes: [0, 2]
+  label_offset: 2

models/Mask3D/build/lib/mask3d/conf/data/datasets/semantic_kitti.yaml

@@ -0,0 +1,42 @@
+# @package data
+train_dataset:
+  _target_: mix3d.datasets.outdoor_semseg.LidarDataset
+  data_dir: data/processed/semantic_kitti
+  label_db_filepath: data/processed/semantic_kitti/label_database.yaml
+  mode: ${data.train_mode}
+  add_reflection: ${data.add_reflection}
+  add_distance: ${data.add_distance}
+  add_instance: ${data.add_instance}
+  num_labels: ${data.num_labels}
+  sweep: ${data.sweep}
+  data_percent: 1.0
+  ignore_label: ${data.ignore_label}
+  volume_augmentations_path: mix3d/conf/augmentation/volumentations_aug.yaml
+
+validation_dataset:
+  _target_: mix3d.datasets.outdoor_semseg.LidarDataset
+  data_dir: data/processed/semantic_kitti
+  label_db_filepath: data/processed/semantic_kitti/label_database.yaml
+  mode: ${data.validation_mode}
+  add_reflection: ${data.add_reflection}
+  add_distance: ${data.add_distance}
+  add_instance: ${data.add_instance}
+  num_labels: ${data.num_labels}
+  sweep: ${data.sweep}
+  data_percent: 1.0
+  ignore_label: ${data.ignore_label}
+  volume_augmentations_path: null
+
+test_dataset:
+  _target_: mix3d.datasets.outdoor_semseg.LidarDataset
+  data_dir: data/processed/semantic_kitti
+  label_db_filepath: data/processed/semantic_kitti/label_database.yaml
+  mode: ${data.test_mode}
+  add_reflection: ${data.add_reflection}
+  add_distance: ${data.add_distance}
+  add_instance: ${data.add_instance}
+  num_labels: ${data.num_labels}
+  sweep: ${data.sweep}
+  data_percent: 1.0
+  ignore_label: ${data.ignore_label}
+  volume_augmentations_path: null

models/Mask3D/build/lib/mask3d/conf/data/datasets/stpls3d.yaml

@@ -0,0 +1,95 @@
+# @package data
+train_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "stpls3d"
+  data_dir: data/processed/stpls3d
+  image_augmentations_path: conf/augmentation/albumentations_aug.yaml
+  volume_augmentations_path: conf/augmentation/volumentations_aug.yaml
+  label_db_filepath: data/processed/stpls3d/label_database.yaml
+  color_mean_std: data/processed/stpls3d/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.train_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  # different augs experiments
+  instance_oversampling: 0.0
+  place_around_existing: False
+  point_per_cut: 0
+  max_cut_region: 0
+  flip_in_center: false
+  noise_rate: 0
+  resample_points: 0
+  cropping: ${data.cropping}
+  cropping_args: ${data.cropping_args}
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  reps_per_epoch: ${general.reps_per_epoch}
+  eval_inner_core: ${general.eval_inner_core}
+  filter_out_classes: [0]
+  label_offset: 1
+  is_elastic_distortion: true
+  color_drop: 0.0
+
+validation_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "stpls3d"
+  data_dir: data/processed/stpls3d
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/stpls3d/label_database.yaml
+  color_mean_std: data/processed/stpls3d/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.validation_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  on_crops: ${general.on_crops}
+  eval_inner_core: ${general.eval_inner_core}
+  filter_out_classes: [0]
+  label_offset: 1
+
+test_dataset:
+  _target_: mask3d.datasets.semseg.SemanticSegmentationDataset
+  dataset_name: "stpls3d"
+  data_dir: data/processed/stpls3d
+  image_augmentations_path: null
+  volume_augmentations_path: null
+  label_db_filepath: data/processed/stpls3d/label_database.yaml
+  color_mean_std: data/processed/stpls3d/color_mean_std.yaml
+  data_percent: 1.0
+  mode: ${data.test_mode}
+  ignore_label: ${data.ignore_label}
+  num_labels: ${data.num_labels}
+  add_raw_coordinates: ${data.add_raw_coordinates}
+  add_colors: ${data.add_colors}
+  add_normals: ${data.add_normals}
+  add_instance: ${data.add_instance}
+  cache_data: ${data.cache_data}
+  cropping: false
+  is_tta: false
+  crop_min_size: ${data.crop_min_size}
+  crop_length: ${data.crop_length}
+  cropping_v1: ${data.cropping_v1}
+  area: ${general.area}
+  on_crops: ${general.on_crops}
+  eval_inner_core: ${general.eval_inner_core}
+  filter_out_classes: [0]
+  label_offset: 1

models/Mask3D/build/lib/mask3d/conf/data/indoor.yaml

@@ -0,0 +1,43 @@
+# @package _group_
+
+# these parameters are inherited by datasets, data_loaders and collators
+# but they might be overwritten
+
+# splits
+train_mode: train
+validation_mode: validation
+test_mode: validation # test  # validation
+
+# dataset
+ignore_label: 255
+add_raw_coordinates: true # 3dim
+add_colors: true # 3dim
+add_normals: false # 3dim
+in_channels: 3 # in_channels = 3 * (add_normals + add_colors + add_raw_coordinates)
+num_labels: 20
+# num_labels: 41
+add_instance: ${general.add_instance}
+task: ${general.task}
+
+# data loader
+pin_memory: false
+num_workers: 4
+batch_size: 5
+test_batch_size: 1
+cache_data: false
+
+# collation
+voxel_size: 0.02
+
+reps_per_epoch: ${general.reps_per_epoch}
+
+cropping: false
+cropping_args:
+  min_points: 30000
+  aspect: 0.8
+  min_crop: 0.5
+  max_crop: 1.0
+
+crop_min_size: 20000
+crop_length: 6.0
+cropping_v1: true

models/Mask3D/build/lib/mask3d/conf/data/outdoor.yaml

@@ -0,0 +1,26 @@
+# @package _group_
+
+# these parameters are inherited by datasets, data_loaders and collators
+# but they might be overwritten
+
+# splits
+train_mode: train
+validation_mode: validation
+test_mode: validation
+
+# dataset
+ignore_label: 255
+add_distance: true # 1dim
+add_reflection: true # 1dim
+in_channels: 2 # in_channels = add_distance + add_reflection
+num_labels: 19
+add_instance: false
+
+# data loader
+pin_memory: true
+num_workers: 4
+batch_size: 18
+sweep: 1
+
+# collation
+voxel_size: 0.15

models/Mask3D/build/lib/mask3d/conf/logging/base.yaml

@@ -0,0 +1,10 @@
+# @package _group_
+- _target_: pytorch_lightning.loggers.NeptuneLogger
+  project_name: ${general.workspace}/${general.project_name}
+  experiment_name: ${general.experiment_name}
+  offline_mode: false
+
+- _target_: pytorch_lightning.loggers.CSVLogger
+  save_dir: ${general.save_dir}
+  name: ${general.experiment_id}
+  version: ${general.version}

models/Mask3D/build/lib/mask3d/conf/logging/full.yaml

@@ -0,0 +1,8 @@
+# @package _group_
+- _target_: pytorch_lightning.loggers.WandbLogger
+  project: ${general.project_name}
+  name: ${general.experiment_name}
+  save_dir: ${general.save_dir}
+  entity: "schult"
+  resume: "allow"
+  id: ${general.experiment_name}

models/Mask3D/build/lib/mask3d/conf/logging/minimal.yaml

@@ -0,0 +1,5 @@
+# @package _group_
+- _target_: pytorch_lightning.loggers.CSVLogger
+  save_dir: ${general.save_dir}
+  name: ${general.experiment_id}
+  version: ${general.version}

models/Mask3D/build/lib/mask3d/conf/logging/offline.yaml

@@ -0,0 +1,10 @@
+# @package _group_
+- _target_: pytorch_lightning.loggers.TensorBoardLogger
+  name: ${general.experiment_id}
+  version: ${general.version}
+  save_dir: ${general.save_dir}
+
+- _target_: pytorch_lightning.loggers.CSVLogger
+  name: ${general.experiment_id}
+  version: ${general.version}
+  save_dir: ${general.save_dir}

models/Mask3D/build/lib/mask3d/conf/loss/cross_entropy.yaml

@@ -0,0 +1,3 @@
+# @package _group_
+_target_: torch.nn.CrossEntropyLoss
+ignore_index: ${data.ignore_label}

models/Mask3D/build/lib/mask3d/conf/loss/set_criterion.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+_target_: mask3d.models.criterion.SetCriterion
+num_classes: ${general.num_targets}
+eos_coef: 0.1
+losses:
+  - "labels"
+  - "masks"
+num_points: ${matcher.num_points}
+oversample_ratio: 3.0
+importance_sample_ratio: 0.75
+class_weights: -1

models/Mask3D/build/lib/mask3d/conf/loss/set_criterion_custom_weights_1.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+_target_: mask3d.models.criterion.SetCriterion
+num_classes: ${general.num_targets}
+eos_coef: 0.1
+losses:
+  - "labels"
+  - "masks"
+num_points: ${matcher.num_points}
+oversample_ratio: 3.0
+importance_sample_ratio: 0.75
+class_weights: [1.0,1.5,10.0,1.0,1.0,1.0,1.0,1.0,10.0,10.0,1.0,10.0,1.0,1.0]

models/Mask3D/build/lib/mask3d/conf/matcher/hungarian_matcher.yaml

@@ -0,0 +1,6 @@
+# @package _group_
+_target_: mask3d.models.matcher.HungarianMatcher
+cost_class: 2.
+cost_mask: 5.
+cost_dice: 2.
+num_points: -1

models/Mask3D/build/lib/mask3d/conf/metrics/miou.yaml

@@ -0,0 +1,4 @@
+# @package _group_
+_target_: mask3d.models.metrics.ConfusionMatrix
+num_classes: ${data.num_labels}
+ignore_label: ${data.ignore_label}

models/Mask3D/build/lib/mask3d/conf/model/mask3d.yaml

@@ -0,0 +1,47 @@
+# @package _group_
+_target_: mask3d.models.Mask3D
+
+# transformer parameters
+hidden_dim: 128
+dim_feedforward: 1024
+num_queries: 100
+num_heads: 8
+num_decoders: 3
+dropout: 0.0
+pre_norm: false
+use_level_embed: false
+normalize_pos_enc: true
+positional_encoding_type: "fourier"
+gauss_scale: 1.0
+hlevels: [0,1,2,3]
+
+# queries
+non_parametric_queries: true
+random_query_both: false
+random_normal: false
+random_queries: false
+use_np_features: false
+
+# sampling
+sample_sizes: [200, 800, 3200, 12800, 51200]
+max_sample_size: false # change false means sampling activated
+
+shared_decoder: true
+num_classes: ${general.num_targets}
+train_on_segments: ${general.train_on_segments}
+scatter_type: "mean"
+
+voxel_size: ${data.voxel_size}
+
+config:
+  backbone:
+    _target_: mask3d.models.Res16UNet34C
+    config:
+      dialations: [ 1, 1, 1, 1 ]
+      conv1_kernel_size: 5
+      bn_momentum: 0.02
+    # depends on normals, color, raw_coordinates
+    # varies from 3 to 9
+    in_channels: ${data.in_channels}
+    out_channels: ${data.num_labels}
+    out_fpn: true

models/Mask3D/build/lib/mask3d/conf/optimizer/adamw.yaml

@@ -0,0 +1,3 @@
+# @package _group_
+_target_: torch.optim.AdamW
+lr: 0.0001

models/Mask3D/build/lib/mask3d/conf/optimizer/adamw_lower.yaml

@@ -0,0 +1,3 @@
+# @package _group_
+_target_: torch.optim.AdamW
+lr: 0.005

models/Mask3D/build/lib/mask3d/conf/scheduler/exponentiallr.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+
+scheduler:
+  _target_: torch.optim.lr_scheduler.ExponentialLR
+  gamma: 0.99999
+  last_epoch: -1 # ${trainer.max_epochs}
+  # need to set to number because of tensorboard logger
+  # steps_per_epoch: -1
+
+pytorch_lightning_params:
+  interval: step

models/Mask3D/build/lib/mask3d/conf/scheduler/lambdalr.yaml

@@ -0,0 +1,8 @@
+# @package _group_
+
+scheduler:
+  _target_: torch.optim.lr_scheduler.StepLR
+  step_size: 99999
+
+pytorch_lightning_params:
+  interval: epoch

models/Mask3D/build/lib/mask3d/conf/scheduler/onecyclelr.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+
+scheduler:
+  _target_: torch.optim.lr_scheduler.OneCycleLR
+  max_lr: ${optimizer.lr}
+  epochs: ${trainer.max_epochs}
+  # need to set to number because of tensorboard logger
+  steps_per_epoch: -1
+
+pytorch_lightning_params:
+  interval: step

models/Mask3D/build/lib/mask3d/conf/trainer/trainer.yaml

@@ -0,0 +1,7 @@
+# @package _group_
+deterministic: false
+max_epochs: 1000
+min_epochs: 1
+resume_from_checkpoint: null
+check_val_every_n_epoch: 50
+num_sanity_val_steps: -1

models/Mask3D/build/lib/mask3d/conf/trainer/trainer600.yaml

@@ -0,0 +1,7 @@
+# @package _group_
+deterministic: false
+max_epochs: 601
+min_epochs: 1
+resume_from_checkpoint: null
+check_val_every_n_epoch: 50
+num_sanity_val_steps: 2

models/Mask3D/build/lib/mask3d/datasets/outdoor_semseg.py

@@ -0,0 +1,206 @@
+import logging
+from pathlib import Path
+from typing import List, Optional, Union, Tuple
+from random import random
+
+import numpy as np
+import volumentations as V
+import yaml
+from torch.utils.data import Dataset
+
+logger = logging.getLogger(__name__)
+
+
+class LidarDataset(Dataset):
+    def __init__(
+        self,
+        data_dir: Optional[
+            Union[str, Tuple[str]]
+        ] = "data/processed/semantic_kitti",
+        label_db_filepath: Optional[
+            str
+        ] = "./data/processed/semantic_kitti/label_database.yaml",
+        mode: Optional[str] = "train",
+        add_reflection: Optional[bool] = True,
+        add_distance: Optional[bool] = False,
+        add_instance: Optional[bool] = True,
+        num_labels: Optional[int] = -1,
+        data_percent: Optional[float] = 1.0,
+        ignore_label: Optional[Union[int, List[int]]] = 255,
+        volume_augmentations_path: Optional[str] = None,
+        sweep: Optional[int] = 1,
+    ):
+        self.mode = mode
+        self.data_dir = data_dir
+        if type(data_dir) == str:
+            self.data_dir = [self.data_dir]
+        self.ignore_label = ignore_label
+        self.add_instance = add_instance
+        self.add_distance = add_distance
+        self.add_reflection = add_reflection
+
+        # loading database files
+        self._data = []
+        for database_path in self.data_dir:
+            database_path = Path(database_path)
+            if not (database_path / f"{mode}_database.yaml").exists():
+                print(f"generate {database_path}/{mode}_database.yaml first")
+                exit()
+            self._data.extend(
+                self._load_yaml(database_path / f"{mode}_database.yaml")
+            )
+
+        labels = self._load_yaml(Path(label_db_filepath))
+        self._labels = self._select_correct_labels(labels, num_labels)
+
+        # augmentations
+        self.volume_augmentations = V.NoOp()
+        if volume_augmentations_path is not None:
+            self.volume_augmentations = V.load(
+                volume_augmentations_path, data_format="yaml"
+            )
+
+        # reformulating in sweeps
+        data = [[]]
+        last_scene = self._data[0]["scene"]
+        for x in self._data:
+            if x["scene"] == last_scene:
+                data[-1].append(x)
+            else:
+                last_scene = x["scene"]
+                data.append([x])
+        for i in range(len(data)):
+            data[i] = list(self.chunks(data[i], sweep))
+        self._data = [val for sublist in data for val in sublist]
+
+        if data_percent < 1.0:
+            self._data = self._data[: int(len(self._data) * data_percent)]
+
+    @staticmethod
+    def chunks(lst, n):
+        """Yield successive n-sized chunks from lst."""
+        for i in range(0, len(lst), n):
+            yield lst[i : i + n]
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx: int):
+        points = []
+        for sweep in self.data[idx]:
+            points.append(np.load(sweep["filepath"]))
+            # rotate
+            points[-1][:, :3] = (
+                points[-1][:, :3] @ np.array(sweep["pose"])[:3, :3]
+            )
+            # translate
+            points[-1][:, :3] += np.array(sweep["pose"])[:3, 3]
+        points = np.vstack(points)
+
+        coordinates, features, labels = (
+            points[:, :3],
+            points[:, 3:-2],
+            points[:, -2:],
+        )
+
+        if not self.add_reflection:
+            features = np.ones(np.ones((len(coordinates), 1)))
+
+        if self.add_distance:
+            center_coordinate = coordinates.mean(0)
+            features = np.hstack(
+                (
+                    features,
+                    np.linalg.norm(coordinates - center_coordinate, axis=1)[
+                        :, np.newaxis
+                    ],
+                )
+            )
+
+        # volume and image augmentations for train
+        if "train" in self.mode:
+            coordinates -= coordinates.mean(0)
+            if 0.5 > random():
+                coordinates += (
+                    np.random.uniform(coordinates.min(0), coordinates.max(0))
+                    / 2
+                )
+            aug = self.volume_augmentations(
+                points=coordinates,
+                features=features,
+                labels=labels,
+            )
+            coordinates, features, labels = (
+                aug["points"],
+                aug["features"],
+                aug["labels"],
+            )
+
+        # prepare labels and map from 0 to 20(40)
+        labels = labels.astype(np.int32)
+        if labels.size > 0:
+            labels[:, 0] = self._remap_from_zero(labels[:, 0])
+            if not self.add_instance:
+                # taking only first column, which is segmentation label, not instance
+                labels = labels[:, 0].flatten()
+
+        return coordinates, features, labels
+
+    @property
+    def data(self):
+        """database file containing information about preproscessed dataset"""
+        return self._data
+
+    @property
+    def label_info(self):
+        """database file containing information labels used by dataset"""
+        return self._labels
+
+    @staticmethod
+    def _load_yaml(filepath):
+        with open(filepath) as f:
+            file = yaml.safe_load(f)
+        return file
+
+    def _select_correct_labels(self, labels, num_labels):
+        number_of_validation_labels = 0
+        number_of_all_labels = 0
+        for (
+            k,
+            v,
+        ) in labels.items():
+            number_of_all_labels += 1
+            if v["validation"]:
+                number_of_validation_labels += 1
+
+        if num_labels == number_of_all_labels:
+            return labels
+        elif num_labels == number_of_validation_labels:
+            valid_labels = dict()
+            for (
+                k,
+                v,
+            ) in labels.items():
+                if v["validation"]:
+                    valid_labels.update({k: v})
+            return valid_labels
+        else:
+            msg = f"""not available number labels, select from:
+            {number_of_validation_labels}, {number_of_all_labels}"""
+            raise ValueError(msg)
+
+    def _remap_from_zero(self, labels):
+        labels[
+            ~np.isin(labels, list(self.label_info.keys()))
+        ] = self.ignore_label
+        # remap to the range from 0
+        for i, k in enumerate(self.label_info.keys()):
+            labels[labels == k] = i
+        return labels
+
+    def _remap_model_output(self, output):
+        output = np.array(output)
+        output_remapped = output.copy()
+        for i, k in enumerate(self.label_info.keys()):
+            output_remapped[output == i] = k
+        return output_remapped