Abstract-Depth prediction is one of the fundamental problems in computer vision. In this paper, we propose a simple yet effective convolutional spatial propagation network (CSPN) to learn the affinity matrix for various depth estimation tasks. Specifically, it is an efficient linear propagation model, in which the propagation is performed with a manner of recurrent convolutional operation, and the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). We can append this module to any output from a state-of-the-art (SOTA) depth estimation networks to improve their performances. In practice, we further extend CSPN in two aspects: 1) allow it to take sparse depth map as additional input, which is useful for the task of depth completion [2], [3]; 2) extend to 3D CSPN to handle features with one additional dimension, similar to the commonly used 3D convolution operation. It is effective in the task of stereo matching using 3D cost volume [4] . For the task of single image depth estimation and depth completion, we experiment the proposed CPSN conjunct algorithms over the popular NYU v2 [5] and KITTI [6] datasets, where we show that our proposed algorithms not only produce high quality (e.g., 30% more reduction in depth error), but also run faster (e.g., 2 to 5 × faster) than previous SOTA spatial propagation network [7] . We also evaluated our stereo matching algorithm on the Scene Flow [8] and KITTI Stereo datasets [6] , [9] , and rank 1st on both the KITTI Stereo 2012 and 2015 benchmarks , which demonstrates the effectiveness of the proposed module. The code of CSPN proposed in this work will be release at https://github.com/XinJCheng/CSPN.
CSPN REF performed well with a simplified sequential network to compute disparity and a convolutional spatial propagation network to refine disparity.
Learning Depth with Convolutional Spatial Propagation Network
ArXiv
ArXiv
Computer Science Medicine