Depth Map Inpainting Using a Fully Convolutional Network Emerging consumer depth cameras, such as Microsoft Kinect and ASUS Xtion Pro, are lowcost depth sensors which can capture the depth maps of the scene, and obtaining color images of the corresponding scenes at the same time. Depth information is very important for computer vision applications, and to acquire the accurate depth information is challengeable. However, due to the limitations of imaging principle, depth maps acquired by such devices are often noisy and will lose depth values for certain areas. In this paper, we present a novel method for depth inpainting problem using a Fully Convolutional Network (FCN). The network has two input branches. One branch is a color image that has been processed by a convoluted hidden layer, and the other one is the missing depth map. Besides, we add a shortcut in order to improve the feature extraction capability of the network. We have validated our method in the Middlebury Stereo Dataset and realworld scenes captured by Kinect 2.0. Experiments show that our method has better applicability and reliability than traditional methods in depth inpainting.

Full Resolution Dense Depth Recovery by Fusing RGB Images and Sparse Depth Full resolution depth is required in many real world engineering applications. However, exist depth sensors only offer sparse depth sample points with limited resolution and noise, e.g., LiDARs. We here propose a deep learning based full resolution depth recovery method from monocular images and corresponding sparse depth measurements of target environment. The novelty of our idea is that the structure similar information between the RGB image and depth image is used to refine the dense depth estimation result. This important similar structure information can be found using a correlation layer in the regression neural network. We show that the proposed method can achieve higher estimation accuracy compared to the state of the art methods. The experiments conducted on the NYU Depth V2 prove the novelty of our idea.