We introduce a novel loss for learning local feature descriptors which is inspired by the Lowe's matching criterion for SIFT. We show that the proposed loss that maximizes the distance between the closest positive and closest negative patch in the batch is better than complex regularization methods; it works well for both shallow and deep convolution network architectures. Applying the novel loss to the L2Net CNN architecture results in a compact descriptor -it has the same dimensionality as SIFT (128) that shows state-of-art performance in wide baseline stereo, patch verification and instance retrieval benchmarks. It is fast, computing a descriptor takes about 1 millisecond on a low-end GPU. Many computer vision tasks rely on finding local correspondences, e.g. image retrieval [1, 2], panorama stitching [3], wide baseline stereo [4], 3D-reconstruction [5, 6] . Despite the growing number of attempts to replace complex classical pipelines with end-to-end learned models, e.g., for image matching [7], camera localization [8], the classical detectors and descriptors of local patches are still in use, due to their robustness, efficiency and their tight integration. Moreover, reformulating the task, which is solved by the complex pipeline as a differentiable end-to-end process is highly challenging. As a first step towards end-to-end learning, hand-crafted descriptors like SIFT [9, 10] or detectors [9, 11, 12] have been replace with learned ones, e.g., LIFT [13], MatchNet [14] and DeepCompare [15] . However, these descriptors have not gained popularity in practical applications despite good performance in the patch verification task. Recent studies have confirmed that SIFT and its variants , DSP-SIFT [17]) significantly outperform learned descriptors in image matching and small-scale retrieval [18] , as well as in 3D-reconstruction [19] . One of the conclusions made in [19] is that current local patches datasets are not large and diverse enough to allow the learning of a high-quality widely-applicable descriptor. In this paper, we focus on descriptor learning and, using a novel method, train a convolutional neural network (CNN), called HardNet. We additionally show that our learned descriptor significantly outperforms both hand-crafted and learned descriptors in real-world tasks like image retrieval and two view matching under extreme conditions. For the training, we use the standard patch correspondence data thus showing that the available datasets are sufficient for going beyond the state of the art.
Inspired by nearest neighbor matching strategy, Mishchuk et al. proposed HardNet REF , which was trained by a novel metric loss with the same architecture as L2Net, and it achieved state-of-the-art performance.
Working hard to know your neighbor's margins: Local descriptor learning loss
ArXiv
ArXiv
Computer Science