  We present a technique for estimating the relative 3D rotation of an RGB
image pair in an extreme setting, where the images have little or no overlap.
We observe that, even when images do not overlap, there may be rich hidden cues
as to their geometric relationship, such as light source directions, vanishing
points, and symmetries present in the scene. We propose a network design that
can automatically learn such implicit cues by comparing all pairs of points
between the two input images. Our method therefore constructs dense feature
correlation volumes and processes these to predict relative 3D rotations. Our
predictions are formed over a fine-grained discretization of rotations,
bypassing difficulties associated with regressing 3D rotations. We demonstrate
our approach on a large variety of extreme RGB image pairs, including indoor
and outdoor images captured under different lighting conditions and geographic
locations. Our evaluation shows that our model can successfully estimate
relative rotations among non-overlapping images without compromising
performance over overlapping image pairs.
