  Existing color-guided depth super-resolution (DSR) approaches require paired
RGB-D data as training samples where the RGB image is used as structural
guidance to recover the degraded depth map due to their geometrical similarity.
However, the paired data may be limited or expensive to be collected in actual
testing environment. Therefore, we explore for the first time to learn the
cross-modality knowledge at training stage, where both RGB and depth modalities
are available, but test on the target dataset, where only single depth modality
exists. Our key idea is to distill the knowledge of scene structural guidance
from RGB modality to the single DSR task without changing its network
architecture. Specifically, we construct an auxiliary depth estimation (DE)
task that takes an RGB image as input to estimate a depth map, and train both
DSR task and DE task collaboratively to boost the performance of DSR. Upon
this, a cross-task interaction module is proposed to realize bilateral cross
task knowledge transfer. First, we design a cross-task distillation scheme that
encourages DSR and DE networks to learn from each other in a teacher-student
role-exchanging fashion. Then, we advance a structure prediction (SP) task that
provides extra structure regularization to help both DSR and DE networks learn
more informative structure representations for depth recovery. Extensive
experiments demonstrate that our scheme achieves superior performance in
comparison with other DSR methods.
