  This paper explores self-supervised learning of amodal 3D feature
representations from RGB and RGB-D posed images and videos, agnostic to object
and scene semantic content, and evaluates the resulting scene representations
in the downstream tasks of visual correspondence, object tracking, and object
detection. The model infers a latent3D representation of the scene in the form
of 3D feature points, where each continuous world 3D point is mapped to its
corresponding feature vector. The model is trained for contrastive view
prediction by rendering 3D feature clouds in queried viewpoints and matching
against the 3D feature point cloud predicted from the query view. Notably, the
representation can be queried for any 3D location, even if it is not visible
from the input view. Our model brings together three powerful ideas of recent
exciting research work: 3D feature grids as a neural bottleneck for view
prediction, implicit functions for handling resolution limitations of 3D grids,
and contrastive learning for unsupervised training of feature representations.
We show the resulting 3D visual feature representations effectively scale
across objects and scenes, imagine information occluded or missing from the
input viewpoints, track objects over time, align semantically related objects
in 3D, and improve 3D object detection. We outperform many existing
state-of-the-art methods for 3D feature learning and view prediction, which are
either limited by 3D grid spatial resolution, do not attempt to build amodal 3D
representations, or do not handle combinatorial scene variability due to their
non-convolutional bottlenecks.
