Figure 1: Levenberg-Marquardt/Gauss-Newton optimization methods are widely-used for direct image alignment problems, where the Jacobian matrix is critical for fast and robust optimization. However, when initial alignment is not good enough, numerical Jacobian is unable to provide good gradient directions for the optimization procedure as only the neighboring pixels are employed in the computation of partial derivatives. To this end, we propose the learning of Jacobian based on the feature map of both images (which leverages a large receptive field as shown in green shadow). The joint learning of feature representation and Jacobian matrix enables us to have constraints on both the function and the partial derivatives in the optimization process, making it converges in fewer iterations and achieve more robust results. Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions. Deep learning approaches has been applied to address this problem by learning better feature representations using convolutional neural networks, yet still require a good initialization. In this paper, we demonstrate that the inaccurate numerical Jacobian limits the convergence range which could be improved greatly using learned approaches. Based on this observation, we propose a novel end-to-end network, RegNet, to learn the optimization of image-to-image pose registration. By jointly learning feature representation for each pixel and partial derivatives that replace handcrafted ones (e.g., numerical differentiation) in the optimization step, the neural network facilitates end-to-end optimization. The energy landscape is constrained on both the feature representation and the learned Jacobian, hence providing more flexibility for the optimization as a consequence leads to more robust and faster convergence. In a series of experiments, including a broad ablation study, we demonstrate that RegNet is able to converge for large-baseline image pairs with fewer iterations.
A deep network has been designed recently for computing the pose for direct image to image registration REF .
RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
ArXiv
ArXiv
Computer Science