Finding correspondences between images via local descriptors is one of the most extensively studied problems in computer vision due to the wide range of applications. Recently, end-to-end learnt descriptors [1, 2, 3] based on Convolutional Neural Network (CNN) architectures and training on large datasets have demonstrated to significantly outperform state of the art features. These works are focused on exploiting pairs of positive and negative patches to learn discriminative representations. Recent work on deep learning for learning feature embeddings examines the use of triplets of samples instead of pairs. In this paper we investigate the use of triplets in learning local feature descriptors with CNNs and we propose a novel in-triplet hard negative mining step to achieve a more effective training and better descriptors. Our method reaches state of the art results without the computational overhead typically associated with mining of negatives and with lower complexity of the network architecture. This is a significant advantage over previous CNN-based descriptors since makes our proposal suitable for practical problems involving large datasets. Learning with triplets involves training from samples of the form {a a a, p p p, n n n}, where a a a is the anchor, p p p is a positive example, which is a different sample of the same class as a a a, and n n n is a negative example, belonging to a different class than a a a. In our case, a a a and p p p are different viewpoints of the same physical point, and n n n comes from a different keypoint. The goal is to learn the embedding f (x x x) s.t. δ + = || f (a a a) − f (p p p)|| 2 is low (i.e., the network brings a a a and p p p close in the feature space) and δ − = || f (a a a) − f (n n n)|| 2 is high (i.e., the network pushes the descriptors of a a a and n n n far apart). With this aim, we examine two different loss functions for triplet based-learning: the margin ranking loss and the ratio loss. The margin ranking loss is defined as where µ is an arbitrarily set margin. It measures the violation of the ranking order of the embedded features inside the triplet, which should be δ − > δ + + µ. If that is not the case, then the network adjusts its weights to achieve this result. For its part, the ratio loss optimises the ratio distances within triplets. It learns embeddings such that The goal of this loss function is to force ( e δ + e δ + +e δ − ) 2 to 0, and ( e δ − e δ + +e δ − ) 2 to 1. There is no margin associated with this loss, and by definition we have 0 ≤λ ≤ 1 for all values of δ − , δ + . Fig. 1 illustrates both approaches and their loss surface. In λ (δ + , δ − ) the loss remains 0 until the margin is violated, and after that, there is a linear increase not upper bounded. In contrast,λ (δ + , δ − ) has a clear slope between the two loss levels, and the loss reaches a 1-valued plateau quickly when δ − > δ + . All previous proposals based on triplet based learning use only two of the possible three distances within each triplet, ignoring the distance δ − = || f (p p p) − f (n n n)|| 2 . We take it into account to define the in-triplet hard negative as δ * = min(δ − , δ − ). If δ * = δ − , we swap {a a a, p p p}, and thus p p p becomes the anchor, and a a a becomes the positive sample. This ensures that the hardest negative inside the triplet is used for backpropagation. Subsequently, the margin ranking loss becomes λ (δ + , δ * ) = max(0, µ + δ + − δ * ). A similar expression can be devised for the ratio loss. To test our proposal we build our descriptor by training a shallow network architecture {Conv(7,7)-Tanh-Pool(2,2)-Conv(6,6)-Tanh-FC(128)} from 5M triplets sampled on-the-fly using patches from the Photo-tour dataset [4] . We evaluate its performance in patch pair classification, where we measure the ability of the descriptor to discriminate positive patch pairs from negative ones, and in nearest neighbour patch matching, where we measure the descriptor precision in matching feature points between different views of a same scene. Our networks outperform previously introduced single-scale convolutional feature decriptors, and in some cases with large margin. Moreover, they are 10 times faster than DeepCompare [3] , and 50 times faster than MatchNet [1] and DeepDesc [2] . In fact, when running on GPU we reach speeds of 10µs per patch, which is comparable with the CPU speeds of fast binary descriptors. We observe also that ratio-loss based descriptors are more suitable for patch pair classification and that margin-loss based ones work better in nearest neighbour patch matching. Details of our proposal are described more fully in the paper, along with extensive experimental work. We provide all the learned models and the training code for all descriptor variants at https://github.com/vbalnt/tfeat.
Triplets-net REF shows that learning with triplets outperforms learning with pairs when using the same network and utilizes the ratio loss to suit triplets.
Learning local feature descriptors with triplets and shallow convolutional neural networks.
BMVC
Computer Science