A Stricter Constraint Produces Outstanding Matching: Learning More Reliable Image Matching Using a Quadratic Hinge Triplet Loss Network *
A BSTRACT
Image matching is widely used in many applications, such as visual- based localization and 3D reconstruction.
Compared with traditional local features (e.g., SIFT) and outlier elimination methods (e.g., RANSAC), learning-based image matching methods (e.g., HardNet and OANet) show a promising performance under challenging environments and large-scale benchmarks.
However, the existing learning-based methods suffer from noise in the training data and the existing loss function, e.g., hinge loss, does not work well in image matching networks.
In this paper, we propose an end-to- end image matching method that with less training data to obtain a more accurate and robust performance.
First, a novel data cleaning strategy is proposed to remove the noise in the training dataset.
Second, we strengthen the matching constraints by proposing a novel quadratic hinge triplet (QHT) loss function to improve the network.
Finally, we apply a stricter OANet for sample judgement to pro- duce more outstanding matching.
The proposed method shows the state-of-the-art performance when applied to the large-scale and challenging Phototourism dataset that also reported the 1st place in the CVPR 2020 Image Matching Challenges
Workshop Track1 (un- limited keypoints and standard descriptors) using the reconstructed pose accuracy metric.
1 I NTRODUCTION
Image matching is the foundation task of some high-level 3D com- puter vision tasks, such as 3D reconstruction, structure-from-motion, camera pose estimation, etc.
The goal of the task is to solve the corre- spondences within the same physical region from two images which have a common vision area [9].
Currently, with the widespread de- velopment of deep learning technologies applied in many computer vision areas and the need for long-term large-scale environments tasks, the shortcomings of the traditional keypoint-based image matching method become apparent. The major difﬁculty lies in that usually local features are evaluated through the accuracy of descriptors on small datasets, which only illustrates the matching performance, and is not suitable for integrating and evaluating some downstream tasks.
The use of a deep learning-based method for image matching can overcome the disadvantages of traditional keypoint-based methods, which demonstrates the ability to integrate multi-stage tasks [11] [24]. This could be used to optimize and evaluate different evaluation metrics in large datasets.
Although without using complex hand- crafted accompanied by an appropriate pipeline, the method faces challenging conditions, such as illumination variations, viewpoint changes, repeated textures that are apparent in outdoor datasets where scenes scales and conditions are highly variable.
This harms performance for both accuracy and robustness.
To solve this problem, further end-to-end solutions and modi- ﬁed descriptors have been proposed with the premise being that the descriptor or network could learn more reliable features across image pairs.
A novel representation of features with a log-polar sampling method [12] is proposed to achieve scale invariance.
Some works [11] [24] propose a jointly learning feature detection and description method, which could separately improve the image match- ing performance.
We propose a three-stage pipeline, including feature extraction, feature matching, and outlier pre-ﬁltering steps, to compute the corresponding pixels from image pairs, as shown in Fig.
For each step, we add some constraints to enforce the algorithm to converge to enable better matching.
Unlike the previous methods, our proposed method only requires a lightweight model and it does not need to train on multi-branches separately.
We show that our method outperforms previous algorithms and achieves state-of-the-art performance using the Phototourism benchmark with large-scale environments and under challenging condi- tions.
We provide a detailed insight into how an improved data processing strategy, HardNet [18] loss function, modiﬁed OANet [29] combined with guided match algorithm [2] could help the pipeline [Figure] Pairs Feature
Extraction
Feature
Matching Outlier Pre-filtering Task1: RANSAC
Task2: SfM
Pose
Error
Detector:
SIFTSURF
End-to-end method: SuperPoint, D2-Net, R2D
Descriptor: SOSNet, GIFT
HardNet
Ratio-test,GMS,
Guidedmatch
CNe,OANet
Traditional methods
Learning based methods √ √ √ √ to achieve accurate and robust matching results and evaluate the method on a pose estimation task. Our main contributions include: • We construct a new patch dataset based on the given Photo- tourism images and sparse model, which is similar to the UBC Phototour dataset [7] and pre-train our model on that dataset; • We propose a novel quadratic hinge triplet (QHT) loss function for the feature descriptor network HardNet, and an improved OANet combined with a guided matching strategy to compute reliable feature matches; • Through abundant experiments and the ablation study for each module, we show that our algorithm achieves state-of-the-art performance in the reconstructed poses, which ranks ﬁrst on both tasks of stereo and multi-view matching on the public Phototourism Image Matching challenge 2020 [1].
2 R ELATED W ORK
Image matching plays an important role in many high-level tasks in computer vision area [19] [25], which is to ﬁnd the corresponding pixels from the image pairs in the same physical region in which the scene imaged with geometrical constraints [9].
Feature extraction, feature matching and outlier pre-ﬁltering are the most vital com- ponents in image matching pipeline, in which traditional methods and deep learning-based methods are implemented and completed in recent years.
Fig. 2 shows the main parts of image matching based pose estimation pipeline, including some common methods and our chosen target algorithm to further improve.
Feature Extraction End-to-end feature extraction and match- ing methods are classiﬁed into detect-then-describe (e.g. SuperPoint [10]), detect-and-describe (e.g. R2D2 [24], D2-Net [11]), and describe-then-detect (e.g. DELF [20]) strategies according to the sequences for executing the detection and description processes, which are utilized in different needs.
However, they either have poor performance on challenging conditions and lack of repeatability on keypoints or show low efﬁciency on matching and storage.
Traditionally, detectors and descriptors are separately applied in the pipeline, SIFT [16] (and RootSIFT [3]) and SURF [4] are most popular detectors while some descriptors are followed, in which LogPolar [12] shows better performance than ContextDesc [17] from the relative pose error, while SOSNet [26] and HardNet [18] surpass GIFT [15] in the public validation set.
Outlier pre-ﬁltering There are a large number of incorrect matches in the matching pair, which will bring the noise to the subsequent pose estimation.
Traditional outlier removal methods include ratio-test [16], GMS [5], guided-match [2] methods, etc.
A method based on deep learning is to judge whether the match is an outlier through the pose relationship obtained by the regression of the convolutional network, but it is usually difﬁcult for network training to converge.
Another way is to convert the pose to the label of whether the match is correct through the epipolar constraint, and then the regression problem can be converted into a classiﬁcation problem to determine whether the match is inlier or outlier.
The standard binary cross-entropy loss can be used to learn the model.
However, the input matching pairs are unordered, so the network needs to be permutation-invariant (insensitive) to match sequence transformations, so it is not applicable to convolutional networks or fully connected networks.
CNe [28]draws on the idea of PointNet [23] and proposes a multi-layer weight-sharing perceptron to makes the network irrelevant to the correspondences’ order transformation.
For each input matching pair, the same perceptron is used to process iterations, but this also leads that each matching pair is processed independently, lacks information circulation, and cannot integrate the predicted matching pair judgment results, and thus cannot learn the useful information in the entire image matching pair.
Context normalization [21] is an instance normalization method that is often used widely in image migration [14] and GAN [27].
It normalizes the processing results of image matching pairs to exchange and circulate information.
But this simple and violent use of mean variance ignores the complexity of global features, and at the same time cannot capture the correlation information between parts.
OANet [29] proposed the use of DiffPool and DiffUnPool layers to promote the information ﬂow and communication of internal neurons.
To estimate a more accurate pose of the given image pairs or 3D reconstruction through the matching process, we propose improving the pipeline as follows:
• Obtain stable and accurate keypoints.
The keypoints should appear invariably, e.g., a certain point in the ﬁrst image could also appear stably in the second image taken under a different viewpoint or illumination change.
• Provide reliable and distinguishable descriptors.
Under differ- ent environmental conditions, the same keypoint should have a similar descriptor, so that the same keypoint in different images can be paired using nearest neighbor matching.
• Hold powerful outlier pre-ﬁltering ability.
Filtering the wrong matches to get most of the correct matching pairs could help to get an accurate pose estimation result ﬁnally.
3 M ETHOD
In this paper, we aim to learn accurate and reliable image matching in large-scale challenging conditions.
In contrast with previous works which use end-to-end solutions of insufﬁcient accuracy at the feature extraction stage or traditional solvers through the image matching pipeline, we combined traditional detector solver with an improved HardNet [18] and OANet [29] to pretrain on a constructed dataset.
In particular, we propose a dataset construction method to
[Figure]
Outlier pre
[Figure] filtering [Figure] optimize hyperparameters before training on a large-scale dataset and propose a new loss function in the description stage based on HardNet network, during the outlier pre-ﬁltering stage.
We combined a guided match with a proposed stricter OANet to learn more accurate matches.
The whole architecture of the proposed method is presented in Fig.
Firstly, 128 dimensions of features are extracted from a basic HardNet with the input 32 × 32 patches.
Then, after the matching stage, with the proposed stricter OANet, the correspondences could be transferred into an N binary classiﬁcation (inlier/outlier), which is permutation-invariant and feasible with convolutional layers.
Finally, the pre-ﬁltered matches are used to compute pose estimation in stereo and multi-view tasks.
3.1 Dataset Construction 3.1.1 UBC Dataset Construction for Pretraining
In order to quickly train our lightweight method and search for some hyperparameters to further use into the pretrained model, thus we use the UBC Phototour dataset for the pretraining, which is patch data and suitable for HardNet training.
3.1.2 Phototourism Dataset Construction for Training
After pretraining on the constructed UBC Phototour dataset for the fast parameter selection during the training period, we construct a clean dataset with low noise and less redundancy from Phototourism training scenes.
Removing the redundant data could largely shorten the training period signiﬁcantly, while noise label reducing could enable the loss function gradient descent optimization to improve the network performance.
To reduce the noise label in the dataset, we set the threshold to discard the images with low conﬁdence, in which with the least 25% 3d points are tracked less than 5 times.
When removing redundant data, we sample the 3d points with tracks results that have been tracked more than 5 times.
The sampling is repeated 10 times, and the Normalized Cross-correction (NCC) value is calculated for each sample (the lower of the NCC indicates the lower similarity of the two images and the larger the difference of the matching), and the result with the lowest NCC is retained.
In addition, we also applied data augmentation with random ﬂip and random rotation strategies in both pretraining and training processes.
3.2 Feature Extraction with Improved HardNet
Feature extraction includes keypoint detection and feature descrip- tion processes.
SIFT detector [16] is used ﬁrstly to extract the position and scale information for the selected keypoint of the given input image.
We adopted the OpenCV implemented SIFT with a low detection threshold to generate up to 8000 points and a ﬁxed ori- entation 1 .
A single pixel on the image cannot describe the physical information of the current key point, so in order to describe the key point and the information around it, the key point is cropped with its scale size, called a patch, and then resized to 32 × 32 as the input to the HardNet network to extract the patch’s descriptor.
After a relatively simple 7-layer HardNet, the input 32×32 patch is described as a 128-dimensional feature vector, which represents the description generated for each patch.
We retain the network structure and made improvements to its loss function to train the network stably and efﬁciently on the constructed dataset.
HardNet uses a triplet loss embedded with difﬁcult sample mining, to optimize it so that the distance of the sample descriptor within the class is small, and the distance in the sample description vector between the classes is large.
In order to further improve the effectiveness and stability of model learning, we apply a quadratic hinge triplet (QHT) loss based on the triplet loss, which is inspired by the ﬁrst order similarity loss in SOSNet [26]. This is done by adding the square term of the triple, which shares the same mining strategy as used in HardNet to ﬁnd the “hardest-within-batch” negatives.
The description loss function L des is expressed as:
[Equation]
Where d () stands for the L2 distance, d posi represents the distance between anchor descriptor and positive descriptor, d negi represents the minimum distance in all the distances between the anchor de- scriptor and the negative descriptor.
The positive sample represents the patch on different images corresponding to the same 3d point with the given anchor in the real world. On the contrary, the negative sample represents the patch obtained from different 3d points.
Quadratic triplet loss weights the gradients with respect to the parameters of the network by the magnitude of the loss.
Compared with HardNet, the gradient of QHT loss is more sensitive to changes in loss.
A larger d posi − d negi results in a smaller loss gradient, thus the model learning is more effective and stable.
In addition, the improved model could be sensitive to the noise in the data.
Incorrectly set positive and negative sample labels will degrade the performance of the model, this risk could be avoided through the previously proposed data denoising technique.
3.3 Stricter OANet with Dynamic Guided Matching
OANet proposed to use Diffpool and DiffUnPool layers on top of the CNe network [28] to promote the information circulation and communication of the internal neurons.
The two networks can be abstracted as seen in Fig. N matching pairs with D-dimensional features are processed by the perceptron of the CNe network to ob- tain the output results for the same dimension.
The OANet network reduces the input dimension to an M×D matrix through the Diffpool layer, and then raises the dimension back to an N×D output through DiffUnpool, and merges the information in the middle.
Diffpool maps the input N matches to M by learning the weights of a soft distribution to aggregate the information, and then DiffUnpool reor- ganizes the information back to N dimensions.
The network is not sensitive to disordered disturbances of input matching pairs.
We convert the accuracy of the pose to the accuracy of the match- ing to learn, so the accuracy of the matching will affect the learning ability of the model.
We have made the following improvements to OANet to make it stricter to learn from positive samples and to make matching judgments the label be more accurate, which effectively ﬁlters the exception points and improves the matching and pose estimation performance.
We shortened the threshold condition of the geometric error constraint from 1e-4 to 1e-6.
In addition, we also added a point-to-point constraint based on the OANet point- to-line constraint.
A point is judged as a negative sample when the projection distance is greater than 10 pixels.
Generally, inadequate matches may lead to inaccurate pose esti- mation.
Apart from OANet, for the image pairs with less thanmatches, we also proposed dynamic guided matching (DGM) to increase matches.
In contrast to traditional guided matching [2], the dynamic threshold is applied using the Sampson distance constraint according to the number of matches of an image pair.
We argue that a smaller number of matches requires a greater threshold.
The dynamic threshold th is set by the formula as:
[Equation]
where th init is 6 and n is the number of original matches of an image pair.
For the image pairs with more than 100 matches, we directly applied DEGENSAC [4] to obtain the inliers for submission.
3.4 Implementation Details
We train the proposed model integrated with improved SIFT, Hard- Net, OANet on the Phototourism dataset [1], in which HardNet is pretrained on the constructed UBC Patch dataset [7].
The whole model is implemented with Pytorch [22] on a NVIDIA Titan X GPU.
All models were trained from scratch and no pre-trained models were used.
For the stereo task, the co-visibility threshold is restricted to 0.1 while for the multi-view task, the minimum 3D points are to set to 100, while no more than 25 images are evaluated at a time.
In the description stage, the input patch size is cropped to 32 × 32 with the random ﬂip and random rotation.
Optimization was done by an SGD solver [6] with a learning rate of 10 and weight decay of 0.0001.
The learning rate was linearly decayed to zero withinepochs.
Inthe outlier pre-ﬁltering stage, for training, we employed a dynamic learning rate schedule where the learning rate was increased from zero to the maximum linearly during the warm-up period (the ﬁrst 10,000 steps) and decayed step-by-step after that.
Typically, the maximum learning rate was 1e-5, and the decay rate was 1e-8.
Besides, we trained the fundamental estimation model by setting the value of parameter geo loss margin to 0.1.
The side information of ratio test [16] with a threshold of 0.8 and mutual check was also added into the model input.
During inference time, the output of the model will undergo a DEGENSAC to further eliminate unreliable matches.
The use of DEGENSAC in a dynamic guided match shares the same conﬁguration with the iteration number of 100000, error type of Sampson, inlier threshold of 0.5, conﬁdence of 0. and enabled degeneracy check.
4 E XPERIMENTS
In this section, we evaluate our proposed method on the public Pho- totourism benchmark with the features results. The ﬁnal pose estimation results are computed online.
We ﬁrstly conduct performance for several separate modules, then qualitatively and quantitatively analysis the effectiveness of our method.
4.1 Experimental Settings 4.1.1 Datasets
UBC Phototour dataset [7] is a dataset which involves correspond- ing patches from the 3D reconstruction of Liberty, Notre Dame and
Yosemite.
Every tour consists of several bitmap images at the res- olution of 1024 × 1024 pixels, each image contains image patches as a 16 × 16 array, in which the patch is sampled as a 64 × 64 grayscale.
In addition, the matching information (sample 3D point index of each patch) and keypoint information (reference image index, orientation, scale, and position) are supplied separately in the ﬁle.
Phototourism dataset [1] was collected from multi-sensors in 26 popular tourism landmarks.
The 3D reconstruction ground truth is obtained with Structure from Motion (SfM) using Colmap, which includes poses, co-visibility estimates and depth maps.
The dataset is divided into a training set (16 scenes), a validation set (3 scenes of the training set) and a test set (10 scenes).
As a benchmark large-scale dataset, the scenes contained in the dataset includes vari- ous challenging conditions, e.g., occlusions, changing viewpoints, different time, repeated textures, etc.
4.1.2 Tasks and Evaluation metrics
We implement and compare our method with different approaches with respect to two downstream tasks: stereo and multi-view reconstruction with SfM, they both evaluate the intermediate metrics for further comparisons.
The two tasks evaluate different formats of the dataset, which is subsampled into smaller random subsets from the Phototourism dataset. Stereo task evaluates image pairs while the Multi-view task evaluates 5 25 images with a SfM recon- struction.
Stereo task uses the random sampling consensus method (RANSAC) [13] to estimate the matches between the two images that meet the motion consistency, thereby decomposing the pose rotation R and translation t. Multi-view task recovers the pose R and t of each image through a 3D reconstruction.
By calculating the cosine distance of the vector between the estimated pose and the ground-truth pose, the performance of image matching is measured by the size of the computed angle.
Given an image pair with co-visibility constraints in the stereo task or the 3D reconstruction from multi-view images, we may compute the following metrics in different experiments. The ﬁnal results are compared with the mAA metric.
• Mean average accuracy (%mAA) is computed by integrating the area under curve up to a maximum threshold, which indicates the angular angle between the ground-truth and the estimated pose vector.
• Keypoint repeatability (%Rep.) measures the ratio of possible matches and the minimum number of key points in the covisible view.
• Descriptor matching score(%MS) is the average ratio of correct matches and the minimum number of key points in the covisible view.
Mean absolute trajectory error (%mATE) measures the average deviation from the ground truth trajectory per frame.
• False positive rate (%FPR) is the ratio between the number of negative events wrongly categorized as positive and the total number of actual negative events
4.2 Qualitative and Quantitative Results
The data construction information for the Phototourism dataset is given in Table 1, it could be seen that the proposed dataset construction method could largely reduce the amount of dataset size thus further speed up the training process and help improve the model performance.
The training loss and FRP95 metric trend are shown in Fig. 5 and Fig. 6, using the UBC Phototour dataset, which indicates the loss stability and helps to understand the data characteristics.
The comparison of matching visualization performance for the stereo task is shown in Fig. 7, in which it is seen that our proposed method outperforms the traditional RootSIFT based method, and with more accurate matches.
Fig. 8 shows the visualization on Phototourism dataset on stereo task and multi-view task.
To validate the performance of loss of different scenes
[Table]
Table 2 lists the evaluation of correspondence on UBC Phototour dataset,
[Table]
[Figure]
which indicates the effects and improvements of our proposed QHT loss compared to HT (hinge triplet) loss.
Table 3 and Table 4 demonstrate our submitted ﬁnal results on Phototourism challenge Track1, we rank 1st on both tasks.
4.3 Ablation Study
To fully understand each module in our proposed method, we eval- uate different modules, above, using the Phototourism validation dataset.
In Table 5, our method is compared with several versions to see how data construction and an improved HardNet could help to improve the descriptor and to further enhance the performance for both the multi-view and stereo tasks.
We evaluate the following four variants as the feature descriptor while the other parts remain the same: 1) RootSIFT; 2) the original pretrained HardNet; 3) our proposed improved HardNet; 4) our proposed improved HardNet with a data construction technology.
The comparison of different feature descriptors indicates that the model performance increases largely by adding the loss constraints on HardNet. It shows an 8% improvement in the average mAA of both tasks. With data construc- tion, the improved HardNet yields the best performance, obtaining an average mAA of 0.7894.
To evaluate whether or not the proposed modiﬁed OANet could help improve the performance for outlier pre-ﬁltering, Table 6 lists the performance comparison of four different outlier pre-ﬁltering
[Table]
[Table] [Table]
[Table]
[Table]
methods: 1) using RootSIFT as the feature descriptor while ratio test and cross check are used for the outlier ﬁltering; 2) use of the pretrained HardNet with a ratio test and cross check; 3) use of the proposed improved HardNet with ratio test and cross check; 4) use of proposed improved HardNet with a modiﬁed OANet. The com- parison of different outlier removals indicates that the performance of modiﬁed OANet surpasses the ratio test with the cross check method by a 4% in average mAA
[Table]
5 C ONCLUSION
This paper presents a novel image matching approach, which: in- tegrates a data construction method to remove noise and to ensure more efﬁcient training; uses an improved HardNet QHT loss function to make the network more sensitive to noise gradient descent; uses a stricter OANet combined with guided match algorithm to reduce the recall rate of outlier pre-ﬁltering.
By proposing the QHT loss, which strengthens the distance constraints for the improved OANet with a stricter judgment on positive samples, an improved outstanding matching is produced using these stricter constraints.
The proposed method is crucial to obtain high-quality correspondences when applied to a large-scale challenging dataset with illumination vibrances, viewpoint changes and repeated textures.
Our experiments show that this method achieves a state-of-the-art performance on the public Phototourism Image Matching challenge.
A CKNOWLEDGMENTS
This research was funded in part by a Ph.D. scholarship funded jointly by the China Scholarship Council and QMUL.
We thank our colleagues from Didi Chuxing who provided insight and expertise that greatly assisted the research, especially for Zhongkun Chen,
Wei Shao, Jingchao Zhou, Tengfei Xing and Bin Xu.