Joint Learning of Instance and Semantic Segmentation for Robotic PickandPlace with Heavy Occlusions in Clutter We present joint learning of instance and semantic segmentation for visible and occluded region masks. Sharing the feature extractor with instance occlusion segmentation, we introduce semantic occlusion segmentation into the instance segmentation model. This joint learning fuses the instance and imagelevel reasoning of the mask prediction on the different segmentation tasks, which was missing in the previous work of learning instance segmentation only (instanceonly). In the experiments, we evaluated the proposed joint learning comparing the instanceonly learning on the test dataset. We also applied the joint learning model to 2 different types of robotic pickandplace tasks (random and target picking) and evaluated its effectiveness to achieve realworld robotic tasks.

Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions Semantic segmentation and instance level segmentation made substantial progress in recent years due to the emergence of deep neural networks (DNNs). A number of deep architectures with Convolution Neural Networks (CNNs) were proposed that surpass the traditional machine learning approaches for segmentation by a large margin. These architectures predict the directly observable semantic category of each pixel by usually optimizing a crossentropy loss. In this work we push the limit of semantic segmentation towards predicting semantic labels of directly visible as well as occluded objects or objects parts, where the networkxe2x80x99s input is a single depth image. We group the semantic categories into one background and multiple foreground object groups, and we propose a modification of the standard crossentropy loss to cope with the settings. In our experiments we demonstrate that a CNN trained by minimizing the proposed loss is able to predict semantic categories for visible and occluded object parts without requiring to increase the network size (compared to a standard segmentation task). The results are validated on a newly generated dataset (augmented from SUNCG) dataset.