We propose a convolutional approach to correct multipath interference (MPI) errors, in real-time and on complex scenes captured with off-the-shelf, time-of-flight (ToF) cameras. We introduce a novel two-stage learning scheme: we first pre-initialize our network to learn real depth representations, and then re-train these to learn MPI corrections from synthetic data. The leftmost image shows one of our synthetic validation scenes: our method (green) provides a better fit to ground truth data than ToF depth from the camera. In the sequence to the right, we illustrate how our system allows for real-time MPI compensation (10 milliseconds per frame) in real scenes captured with ToF devices, preserving temporal consistency. Please refer to the supplemental video for more results. Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated * Part of this work was done during Julio Marco's internship at Microsoft Research Asia. by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input.
Marco et al. use an encoder to learn a mapping from captured ToF measurements to a representation of (MPI-corrupted) depth, and then combine this with a limited number of simulated, direct-only ToF measurements to train a decoder that produces MPI-corrected depth maps REF .
DeepToF: Off-the-Shelf Real-Time Correction of Multipath Interference in Time-of-Flight Imaging
ACM Transactions on Graphics 36, 6, Article 219 (November 2017)
Mathematics Computer Science