  This paper presents DeepI2P: a novel approach for cross-modality registration
between an image and a point cloud. Given an image (e.g. from a rgb-camera) and
a general point cloud (e.g. from a 3D Lidar scanner) captured at different
locations in the same scene, our method estimates the relative rigid
transformation between the coordinate frames of the camera and Lidar. Learning
common feature descriptors to establish correspondences for the registration is
inherently challenging due to the lack of appearance and geometric correlations
across the two modalities. We circumvent the difficulty by converting the
registration problem into a classification and inverse camera projection
optimization problem. A classification neural network is designed to label
whether the projection of each point in the point cloud is within or beyond the
camera frustum. These labeled points are subsequently passed into a novel
inverse camera projection solver to estimate the relative pose. Extensive
experimental results on Oxford Robotcar and KITTI datasets demonstrate the
feasibility of our approach. Our source code is available at
https://github.com/lijx10/DeepI2P
