  Dense depth map capture is challenging in existing active sparse illumination
based depth acquisition techniques, such as LiDAR. Various techniques have been
proposed to estimate a dense depth map based on fusion of the sparse depth map
measurement with the RGB image. Recent advances in hardware enable adaptive
depth measurements resulting in further improvement of the dense depth map
estimation. In this paper, we study the topic of estimating dense depth from
depth sampling. The adaptive sparse depth sampling network is jointly trained
with a fusion network of an RGB image and sparse depth, to generate optimal
adaptive sampling masks. We show that such adaptive sampling masks can
generalize well to many RGB and sparse depth fusion algorithms under a variety
of sampling rates (as low as $0.0625\%$). The proposed adaptive sampling method
is fully differentiable and flexible to be trained end-to-end with upstream
perception algorithms.
