Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, ARVR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multitask learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection subtask is responsible for providing the boundary features and constructing crosstask joint loss functions for network training. The entire network is implemented endtoend and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multitask CNN model can effectively improve the performance of every single task.

Monocular Outdoor Semantic Mapping with a Multitask Network In many robotic applications, especially for the autonomous driving, understanding the semantic information and the geometric structure of surroundings are both essential. Semantic 3D maps, as a carrier of the environmental knowledge, are then intensively studied for their abilities and applications. However, it is still challenging to produce a dense outdoor semantic map from a monocular image stream. Motivated by this target, in this paper, we propose a method for largescale 3D reconstruction from consecutive monocular images. First, with the correlation of underlying information between depth and semantic prediction, a novel multitask Convolutional Neural Network (CNN) is designed for joint prediction. Given a single image, the network learns lowlevel information with a shared encoder and separately predicts with decoders containing additional Atrous Spatial Pyramid Pooling (ASPP) layers and the residual connection which merits disparities and semantic mutually. To overcome the inconsistency of monocular depth prediction for reconstruction, postprocessing steps with the superpixelization and the effective 3D representation approach are obtained to give the final semantic map. Experiments are compared with other methods on both semantic labeling and depth prediction. We also qualitatively demonstrate the map reconstructed from largescale, difficult monocular image sequences to prove the effectiveness and superiority.