  Self-supervised representation learning is a critical problem in computer
vision, as it provides a way to pretrain feature extractors on large unlabeled
datasets that can be used as an initialization for more efficient and effective
training on downstream tasks. A promising approach is to use contrastive
learning to learn a latent space where features are close for similar data
samples and far apart for dissimilar ones. This approach has demonstrated
tremendous success for pretraining both image and point cloud feature
extractors, but it has been barely investigated for multi-modal RGB-D scans,
especially with the goal of facilitating high-level scene understanding. To
solve this problem, we propose contrasting "pairs of point-pixel pairs", where
positives include pairs of RGB-D points in correspondence, and negatives
include pairs where one of the two modalities has been disturbed and/or the two
RGB-D points are not in correspondence. This provides extra flexibility in
making hard negatives and helps networks to learn features from both
modalities, not just the more discriminating one of the two. Experiments show
that this proposed approach yields better performance on three large-scale
RGB-D scene understanding benchmarks (ScanNet, SUN RGB-D, and 3RScan) than
previous pretraining approaches.
