  Gradient-based algorithms are crucial to modern computer-vision and graphics
applications, enabling learning-based optimization and inverse problems. For
example, photorealistic differentiable rendering pipelines for color images
have been proven highly valuable to applications aiming to map 2D and 3D
domains. However, to the best of our knowledge, no effort has been made so far
towards extending these gradient-based methods to the generation of depth
(2.5D) images, as simulating structured-light depth sensors implies solving
complex light transport and stereo-matching problems. In this paper, we
introduce a novel end-to-end differentiable simulation pipeline for the
generation of realistic 2.5D scans, built on physics-based 3D rendering and
custom block-matching algorithms. Each module can be differentiated w.r.t
sensor and scene parameters; e.g., to automatically tune the simulation for new
devices over some provided scans or to leverage the pipeline as a 3D-to-2.5D
transformer within larger computer-vision applications. Applied to the training
of deep-learning methods for various depth-based recognition tasks
(classification, pose estimation, semantic segmentation), our simulation
greatly improves the performance of the resulting models on real scans, thereby
demonstrating the fidelity and value of its synthetic depth data compared to
previous static simulations and learning-based domain adaptation schemes.
