roboception 0.21
A modular perception library for machine learning in robotics use cases
Requires Python: >=3.7

Table of contents
=================

<!--ts-->
   * [Out of box colab usage](#out-of-box-colab-usage)
   * [Getting Started](#getting-started)
   * [Supported Dataloaders](#dataloaders)
        * [CIHP](#cihp)
        * [DIODE](#diode)
        * [NYUV2](#nyuv2)
        * [ADE20K](#ade20k)
        * [Generic Image CSV](#generic)
        * [NUSCENES](#nuscenes)
   * [General Development Instructions](#general-development-instructions)
<!--te-->

Out of box colab usage
======================

1. Easy monocular depth estimation - https://colab.research.google.com/drive/1T2gONs_gst4zpdS7fBoIaQclgg3J2Jgk?usp=sharing


Getting Started
===============

1. Local system setup 

    1. Make sure system is [configured to use nvidia-gpu](https://docs.google.com/document/d/1TscBJu4nX1Q2VVBUJIEgFi6_ijH89-mzsuZ_Whd27V4/edit?usp=sharing)

    2. Make sure [nvidia docker is setup](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)

    OR

2. [Azure cloud computer setup](https://docs.google.com/document/d/1y7JMUwg69t_6jbP-GHrJHk9jQVy3mHcn/edit?usp=sharing&ouid=117914071723364752002&rtpof=true&sd=true)

    After system setup 

3. Run the command to get the environment setup (all the relevant libraries) or use the dockerfile here and build env yourself
    
    ```bash
    $ docker pull homagni/mltraining:latest
    ```

4. git clone this repo

5. Make a folder called "datasets" in the same directory where you git cloned this repo

6. Follow the instructions on how to acquire the relevant data from dataloaders instructions below and put the downloaded data in "datasets" folder

7. Run the following script to start the docker container by attaching the datasets (example if /home/homag/Desktop/ is the parent directory of the git cloned):

    ```bash
    $ xhost +

    $ sudo nvidia-docker run --rm -ti --mount type=bind,source=/home/homag/Desktop/ml,target=/ml --mount type=bind,source=/home/homag/Desktop/datasets,target=/datasets --privileged --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" homagni/mltraining:latest
    ```

8. Git ignored file extensions (found in .git/info/exclude)
    ```bash
    *.png
    *.jpg
    *.jpeg
    *.pcd
    *.dat
    *.pth
    *.pyc
    ```


Supported Dataloaders
=====================
The dataloaders folder provides high level abstractions to input dataset file names, parse and create a fast accessible numpy memory map and retrieve pytorch compatible batch in a get batch fashion for batchwise training of any network. Each python file in the dataloaders folder can be run as standalone to view a sample of the dataset youre training on. Current supported dataloaders are as follows

 CIHP
 ----------------------------------------
Its a human part segmentation dataset, I modified the loader for entire human pixel segmentation (binary segmentation) link - https://drive.google.com/drive/folders/0BzvH3bSnp3E9QjVYZlhWSjltSWM?resourcekey=0-nkS8bDVjPs3bEw3UZW-omA

 DIODE
 --------------------------------------
its a monocular depth estimation dataset, contains RGB and depth maps of indoor and outdoor scenes, download from here - https://diode-dataset.org/ , link - http://diode-dataset.s3.amazonaws.com/train.tar.gz
 
 NYUV2
 -----
The famous depth estimation dataset contains scans of indoor scenes for monocular depth estimation, download from here - https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html , download individual scenarios using links like these from the website- http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/basements.zip and place each part as a seperate folder in datasets/nyuv2/ for example, datasets/nyuv2/basements/basement_0001a/ will contain all the pgm files

 ADE20K
 ------
The famous semantic segmentation dataset. The dataloader/ade20k.py has a script that checks folders and downloads the dataset and the object class names automatically the classnames csv is downloaded from [here](https://github.com/CSAILVision/sceneparsing/blob/master/objectInfo150.csv) and then put in /datasets/ADE20K/ folder 

 Generic Image CSV
 -----------------
A generic loader for any data annotated as images and corresponding csv annotations for details see dataloaders/imgcsvloader_readme 

 NUSCENES
 --------
nuscenes lidar dataset 

(steps to acquire)

[login to nuscenes -> go to downloads

navigate to the nuScenes-lidarseg subsection

select All-> download Metadata and sensor file blobs by clicking on 'US' mirror link

extract the downloaded folder in nuscenes/nuscenes-lidarseg-all-v1.0

navigate to Full dataset-> Trainval subsection

click download Metadata by clicking on 'US' link and save it as nuscenes/v1.0-trainval_meta folder

now you can download the raw data part by part 

for example navigate to File blobs 85 scenes, part 1, expand the arrow symbol

and can download only the Lidar blobs by again clicking on 'US' link

and can save it as nuscenes/part1/v1.0-trainval01_blobs_lidar

can do similar thing for camera data as well]






General Development Instructions
================================


All projects should be in subdirectories in the applications folder as main.py
They can easily call and build upon the submodules - custom_losses, dataloaders, models, trainers, visualizers and applications



datacreators:
contains several functions that can utilize a bunch of simulators to generate data for various applications


custom_losses:
Provides high level abstractions to easily define loss functions for specific types of training

models:
Defines a wide variety of model architectures to be used for various scenarios

trainers:
Defines an extremely high level abstraction for quick training a model for any general application. Supply the appropriate model, loss function and dataloader classes to it and just call the fit function

visualizers:
For pretty visualization of outputs, defines detailed functions to take care of pointcloud visualization from depth maps as well

applications:
Here all the modular machine learning solutions are assembled. Just create a specific folder with the name of the project and create a main function that calls all the submodules to load the datasets, define losses and complete the training


misctools:

miscellaneous tools (code snippets and functions) that can help for various CV related tasks including helping generating data from custom collected data, calibrating cameras/lidars, stiching images, homography, etc. Can also contain starting point code for various applications