  We present a relocalization pipeline, which combines an absolute pose
regression (APR) network with a novel view synthesis based direct matching
module, offering superior accuracy while maintaining low inference time. Our
contribution is twofold: i) we design a direct matching module that supplies a
photometric supervision signal to refine the pose regression network via
differentiable rendering; ii) we modify the rotation representation from the
classical quaternion to SO(3) in pose regression, removing the need for
balancing rotation and translation loss terms. As a result, our network
Direct-PoseNet achieves state-of-the-art performance among all other
single-image APR methods on the 7-Scenes benchmark and the LLFF dataset.
