  In this work, we present a lightweight, tightly-coupled deep depth network
and visual-inertial odometry (VIO) system, which can provide accurate state
estimates and dense depth maps of the immediate surroundings. Leveraging the
proposed lightweight Conditional Variational Autoencoder (CVAE) for depth
inference and encoding, we provide the network with previously marginalized
sparse features from VIO to increase the accuracy of initial depth prediction
and generalization capability. The compact encoded depth maps are then updated
jointly with navigation states in a sliding window estimator in order to
provide the dense local scene geometry. We additionally propose a novel method
to obtain the CVAE's Jacobian which is shown to be more than an order of
magnitude faster than previous works, and we additionally leverage
First-Estimate Jacobian (FEJ) to avoid recalculation. As opposed to previous
works relying on completely dense residuals, we propose to only provide sparse
measurements to update the depth code and show through careful experimentation
that our choice of sparse measurements and FEJs can still significantly improve
the estimated depth maps. Our full system also exhibits state-of-the-art pose
estimation accuracy, and we show that it can run in real-time with
single-thread execution while utilizing GPU acceleration only for the network
and code Jacobian.
