Full Resolution Dense Depth Recovery by Fusing RGB Images and Sparse Depth Full resolution depth is required in many real world engineering applications. However, exist depth sensors only offer sparse depth sample points with limited resolution and noise, e.g., LiDARs. We here propose a deep learning based full resolution depth recovery method from monocular images and corresponding sparse depth measurements of target environment. The novelty of our idea is that the structure similar information between the RGB image and depth image is used to refine the dense depth estimation result. This important similar structure information can be found using a correlation layer in the regression neural network. We show that the proposed method can achieve higher estimation accuracy compared to the state of the art methods. The experiments conducted on the NYU Depth V2 prove the novelty of our idea.

SemiSupervised Monocular Depth Estimation with LeftRight Consistency Using Deep Neural Network There has been tremendous research progress in estimating the depth of a scene from a monocular camera image. Existing methods for singleimage depth prediction are exclusively based on deep neural networks, and their training can be unsupervised using stereo image pairs, supervised using LiDAR point clouds, or semisupervised using both stereo and LiDAR. In general, semisupervised training is preferred as it does not suffer from the weaknesses of either supervised training, resulting from the difference in the cameras and the LiDARs field of view, or unsupervised training, resulting from the poor depth accuracy that can be recovered from a stereo pair. In this paper, we present our research in singleimage depth prediction using semisupervised training that outperforms the stateoftheart. We achieve this through a loss function that explicitly exploits leftright consistency in a stereo reconstruction, which has not been adopted in previous semisupervised training. In addition, we describe the correct use of ground truth depth derived from LiDAR that can significantly reduce prediction error. The performance of our depth prediction model is evaluated on popular datasets, and the importance of each aspect of our semisupervised training approach is demonstrated through experimental results. Our deep neural network model has been made publicly available.1.