Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions Semantic segmentation and instance level segmentation made substantial progress in recent years due to the emergence of deep neural networks (DNNs). A number of deep architectures with Convolution Neural Networks (CNNs) were proposed that surpass the traditional machine learning approaches for segmentation by a large margin. These architectures predict the directly observable semantic category of each pixel by usually optimizing a crossentropy loss. In this work we push the limit of semantic segmentation towards predicting semantic labels of directly visible as well as occluded objects or objects parts, where the networkxe2x80x99s input is a single depth image. We group the semantic categories into one background and multiple foreground object groups, and we propose a modification of the standard crossentropy loss to cope with the settings. In our experiments we demonstrate that a CNN trained by minimizing the proposed loss is able to predict semantic categories for visible and occluded object parts without requiring to increase the network size (compared to a standard segmentation task). The results are validated on a newly generated dataset (augmented from SUNCG) dataset.

Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, ARVR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multitask learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection subtask is responsible for providing the boundary features and constructing crosstask joint loss functions for network training. The entire network is implemented endtoend and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multitask CNN model can effectively improve the performance of every single task.