SeqMatchNet: Contrastive Learning with Sequence Matching for Place Recognition & Relocalization
Author s
AfﬁliationAddress
Abstract: Visual Place Recognition (VPR) for mobile robot global relocalizationis a well-studied problem, where contrastive learning based representation training methods have led to state-of-the-art performance.
However, these methodsare mainly designed for single image based VPR, where sequential information,which is ubiquitous in robotics, is only used as a post-processing step for ﬁlteringsingle image match scores, but is never used to guide the representation learningprocess itself.
In this work, for the ﬁrst time, we bridge the gap between single image representation learning and sequence matching through SeqMatchNet whichtransforms the single image descriptors such that they become more responsive tothe sequence matching metric.
First, we propose a novel triplet loss formulationwhere the distance metric is based on sequence matching , that is, the aggregationof temporal order-based single-image Euclidean distances.
We use the same metric for mining negatives online during the training which helps the optimizationprocess by selecting appropriate positives and harder negatives.
To overcome thecomputational overhead of sequence matching for negative mining, we propose a2D convolution based formulation of sequence matching for efﬁciently aggregating distances within a distance matrix computed using single images.
We showthat our proposed method achieves consistent gains in performance as demonstrated on four benchmark datasets.
Source code will be made publicly available.
Introduction
In the context of mobile robotics, Visual Place Recognition (VPR) is deﬁned as the task of recognizing previously seen places by matching currently observed images with those stored in the referencemap [1, 2, 3].
It is one of the key capabilities of a mobile robot and is required for the task of visuallocalization, either in the form of coarse location estimation [4, 5] given a prior map or for loop closures and mapping in visual SLAM [6, 7].
These VPR-driven tasks are relevant for many practicalrobotic applications, including driverless cars, last mile delivery and exploratory rovers.
One of the main challenges in VPR is scene appearance variations as a place can be revisited during a different time of day, season or weather conditions, for example, nighttime [8] orsnowy conditions [9].
Under such difﬁcult scenarios, classical feature encoding techniques oftenfail [10, 11, 12, 13].
Hence, learning-based techniques have been developed for both local [14, 15]and global feature representations [4, 16, 17].
In VPR, learning a single image descriptor has beenaccomplished by both classiﬁcation [17, 18] and contrastive representation learning [4, 19] typeapproaches.
While these single image based methods address the appearance challenges to someextent, dealing with signiﬁcant perceptual aliasing typically requires the use of additional information in the form of image sequences [20, 21, 22, 23, 24].
In the VPR literature, several sequence-based models have been explored including the use of
[25], linear diagonal search [21], graphs [23] and particle ﬁlters [26].
Most of these methods follow an underlying common principle of aggregating match scores obtained by comparing single image representations.
The community has thus focused on both the aspects, that is, learning more robust single image global descriptors through a variety of deep learning techniques [4, 16, 19, 17, 27, 28] and developing robust sequence matching techniques [29, 30, 31, 32] –but in isolation .
That is, single image representations are learnt without ever considering the subsequent task of sequence score aggregation (or sequence matching).
Thus, there remains a disconnectbetween the two, as the latter never guides the learning process of the former.
In this paper, we address this issue through triplet-loss based contrastive learning with the following key contributions:
• for the ﬁrst time, we integrate sequence matching in the VPR training pipeline, bridging
the gap between learning single image descriptors and sequence matching;
• we propose a sequence matching metric based on temporal order-preserved average ofEuclidean distances for both computing triplet loss and mining negatives; and• ﬁnally, to overcome the computational overhead of sequence matching for negative mining,we propose a novel, GPU-friendly 2D convolutions based distance matrix computationmethod.
We show that transforming an existing single image descriptor by directly optimizing it for sequencematching using the latter both as a loss function and for negative mining leads to superior performance, as compared to traditional formulations based on single image based distance metrics.
Related Work
A detailed literature review on VPR is provided in [1, 3, 33, 2].
Here, we cover the recent VPR literature involving learning-based and sequence-based techniques, and related literature that employstriplets-based contrastive learning techniques.
2.1 Learning for VPR
Recent advances in deep learning using CNNs have led to signiﬁcant progress in the VPR literature, where superior performance was ﬁrst achieved by simply using off-the-shelf deep-learnt imageencoders [34, 35] for image retrieval [36] and place recognition [37], followed by analyses of different CNN layers and training data suitability [38, 39].
Several works exist that employ off-the-shelf
CNNs in unique ways to improve VPR [40, 41, 42, 31, 43, 44, 45].
Learning directly for VPR, oneof the most notable works, NetVLAD [4], re-established the well-known local feature aggregationtechnique: VLAD (Vector of Locally Aggregated Descriptor) [13] by employing a deep-learnt imageencoder, a learnable VLAD layer and triplets-based contrastive learning.
The follow-up literaturehas explored learning methods both in the context of feature aggregation and learning method types.
Chen et al.
[17] trained AMOSNet and HybridNet on a place classiﬁcation task to later use thelearnt features as place representations.
DeLF [46] and its recent global descriptor adaptation with
DeLG [18] learnt features through a landmark recognition task.
Sarlin et al.
[27] proposed HF-Netemploying knowledge distillation for simultaneously learning local and global features.
Differentpooling techniques such as GeM [16, 19], NetBoW [47, 48], NetFV [47], pyramid pooling [49, 50],max pooling [7], and graph attention pooling [51] have also been explored to learn robust image representations.
Other learning based methods include descriptor standardization [52, 39], autoenconders [53], CapsuleNet [54] based feature segregation [55], and omnidirectional CNNs [56].
However, all these methods are speciﬁcally designed for single image based representation learning andare agnostic to any subsequent sequential processing.
2.2 Sequences for VPR
Use of sequential information in VPR is known to improve robustness against perceptual aliasing as more visual evidence becomes available to disambiguate false matches, which are otherwisehard to prevent through limited information or occasional transient noise within single images.
Hoand Newman [20] proposed dynamic programming based
sub-sequence searching to detect loopclosures given a matrix of single image based distances.
Milford and Wyeth [21] proposed SeqSLAM, a sequence searching VPR method based on local velocity search within a column-normalizedscore matrix.
SMART [57] adapted SeqSLAM with the use of odometry information to deal withthe velocity-sensitivity problem.
Along the same lines, Lynen et al.
[29] transformed the distancematrix to obtain “placial indices” and computed vote density using odometry information to form
[Figure]
a “placeless” place recognition system.
A number of follow-up works [22, 58, 23, 59, 60] haveproposed novel sequence searching techniques to robustly estimate place match hypotheses, for example, by using models based on Network Flow [22], HMM [61], Directed Acyclic Graph [23],kNN-DTW [25], and Particle Filter [26, 62, 32].
However, all of these methods are agnostic to theunderlying feature representation techniques and do not use any learning-based mechanisms.
2.3 Triplet Loss and Contrastive Learning
Contrastive learning is used to learn a representation of the given signal by pulling closer the associated signals in contrast to other signals in the dataset [63, 64].
It has been employed for faceveriﬁcation [65], dimension reduction [66], object detection [67], video alignment [68], actionrecognition [69, 70], and several other tasks in scene understanding [71, 72, 73, 63, 74, 75].
Lossformulations typically differ among these methods; examples include noise-contrastive loss [76],normalized temperature based cross entropy loss [63], and max margin-based triplet loss [77] whichis also used in this work.
In the context of sequential imagery or videos, self-supervised tripletgeneration is quite common, as the positives can be sampled using either temporal proximity [78],‘clips’ from within the same video [79, 80], another view of the same scene [81], or a differentvideo from the same known cluster [82].
All of these methods aggregate frame-wise informationexplicitly by channel-wise concatenation of individual frames with 2D CNNs [70] or implicitly byencoding clips through a 3D CNN [79, 69, 80].
The supervisory signal in most of these methods iseither based on strict timestamps [78] or nearby frames/clips [79, 82].
Contrary to that, frame-levelcorrespondences are available for place recognition datasets through GPS or other sensors.
Thisaids in deﬁning supervised loss functions for accurate sequence alignment where order informationcan be explicitly enforced, unlike unordered set-based matching methods [83, 84, 85].
2.4 Learning with sequences for VPR
Recent work in VPR has started to explore learning based techniques for sequence-based methodsbut have mainly focused on generating sequential descriptors as a single summary vector representation of a sequence of images.
This includes online discriminative learning of memory cells [86],end-to-end multi-view learning via feature grouping, fusion and recurrence [87, 88], learning descriptor grouping for sequence matching optimization [89], unsupervised coresets based summarization [90], topological map learning via recurrence [91], unsupervised change-based descriptor adaptation [92], and supervised 1D temporal convolution based hierarchical VPR [93].
Although [87, 93]use sequential information with triplet loss, the learnt representation is a weighted combination ofa sequence of images that results in a single summary vector, thus doing away with the need forsequence matching within the triplet loss.
In this work, single image representations are adaptedindependent of the sequential information, and sequential information is only introduced as thematching metric when deﬁning the loss function.
SeqMatchNet
Here, we present all the components of SeqMatchNet (see Figure 1).
We ﬁrst explain how the existing single image representations are transformed; then we present our training framework whichincludes the triplet loss, sequence matching metric and convolutions-based distance matrix computation for negative mining; and ﬁnally, we describe the test time sequential query retrieval process.
3.1 Image Representation
Given an image I , its corresponding descriptor X ∈ R D is assumed to be pre-computed using anexisting single image representation method.
This descriptor is linearly transformed to X (cid:
) ∈ R Dusing a learnable fully-connected layer with weight matrix W ∈ R D × D and bias B ∈ R D .
In thesupplementary material, we provide results for deeper network conﬁgurations.
[Equation]
X (cid:48) is l2-normalized before computing distances.
W and B are shared across all the descriptors andalso when computing triplets, as explained next.
3.2 Training
Here, we present the training pipeline including contrastive triplet loss and the use of sequencematching as a distance metric both for computing loss and mining negatives.
Contrastive Triplet Loss:
Our objective is to learn X (cid:48) such that for any given anchor imagedescriptor X (cid:48) a , its distance from a positive image descriptor X (cid:48) p
(belonging to a nearby physicallocation) is smaller than its distance from a negative image descriptor X (cid:48)
n (belonging to a far-offphysical location) by a certain margin ( m ).
That is, the contrastive max-margin triplet loss L isdeﬁned as:
[Equation]
Sequence-based Distance Metric:
In most existing works, the distance computation in the aforementioned triplet loss formulation is based on single summary vector metrics [4, 87].
In thiswork, we instead use a sequence of image descriptors of length L centered at any given singleimage descriptor to deﬁne a distance metric, inspired by well-established sequence matching methods [21, 57, 23, 94, 93].
Thus, for a temporally-ordered list of images within a sequence, ordercorresponding pairwise Euclidean distances are computed between any two sequences centered at
X (cid:48)
i and X (cid:48) j , and then averaged to obtain the sequence matching distance 1 (see Figure 1):
[Equation]
The above sequential order-preserving distance measure is practical for many robotic applications,for example, autonomous driving, where an agent navigates repeatedly through the same route,especially when a shorter metric span (short sequence) is considered.
Unlike several existing videobased learning methods [78, 79, 70, 69, 82], the availability of motion and position information(odometry/GPS) from robot’s sensor data allows sub-sampling of image sequences using a ﬁxedmetric separation while also providing frame-to-frame correspondence across reference and querydatabases [6, 21, 4].
This enables simpliﬁed supervision with the help of the proposed distancemetric and loss formulation for the task of visual place recognition, unlike unordered set-basedmatching metrics used in other domains [83, 84, 85].
Using sequence-based matching during training as proposed has a threefold effect.
1) It simultaneously optimizes for L consecutive descriptors – this prevents the system from over-focusing onany one single descriptor within a sequence which is too hard (a positive or a negative) to optimizestandalone (as in traditional approaches) but might only have a minimal impact on the aggregatedsequence distance.
This effect can be observed through the negative distance proﬁles based as visualized in Section 5.5. 2)
Since the individual descriptors within a sequence are matched in a1 d ij follows all the distance metric properties since it is an average of temporal-order preserving manner, the anchor-positive distance so achieved is ideally the lowest– this prevents the system from unnecessarily optimizing for all possible combinations of anchorsand positives within a certain localization radius, e.g. trying to minimize a full L × L distancematrix between an anchor and a positive sequence as opposed to ideally minimizing only the diagonal elements of that matrix.
As an aggregated distance is computed between an anchor and anegative through sequence matching, the negatives selected online during training are harder thanthose which would have been selected when using single image based training – this gives priority tothose regions of the environment where several consecutively-occurring negatives make it difﬁcultto avoid perceptual aliasing.
While all the aforementioned effects highlight the beneﬁts of using sequence matching over single matching, other alternative strategies can also be deﬁned for temporalorder-preserved contrastive learning.
Distance Matrix Computation for Negative Mining:
Matching sequences of image descriptorsnaturally comes at an additional cost of computation.
This becomes a bottleneck during trainingwhen searching for hard negatives in a large reference database.
Since most of the sequence-based
VPR datasets are long vehicular trajectories of the environment, it is typical to ﬁrst compute adistance matrix through single image descriptor comparisons.
This distance matrix is then furtherprocessed to obtain sequence matching scores [20, 21, 5].
Leveraging GPUs’ high computationalcompatibility with convolution operations, we propose an efﬁcient 2D convolution based approachto compute sequence-based distance matrix M seq by convolving an identity matrix I L with the singledescriptor-based distance matrix M single : [Equation]
[Equation]
The above formulation computes the mean of the diagonal elements of M single as I L strides alongits rows (reference database) and columns (query database), thus the outcome is the same as that of
Equation 3.
3.3 Testing
Sequential Query Retrieval: During the test time, query images are retrieved using sequences oflength L centered at each query image.
In the batch approach of sequence matching (Equation 4) formining negatives during training, all the queries of the train set were available beforehand.
For thetest time, similar to an online operation, we assume query images to be accrued incrementally.
Thus,we re-use Equation 4 for computing sequence matching distances but perform convolutions onlyalong the rows of the matrix M single .
Using the resultant distance matrix M seq , a reference imagewith the lowest sequence matching distance is retrieved as a match for any given query sequence.
Experimental Settings
4.1 Datasets
We use four datasets in this work: i) Nordland [9] - Summer vs Winter, ii) Oxford Robotcar [8] andiii) Brisbane City Loop [95] - Day vs Night, and iv) MSLS [88] - Multi-city/Multi-conditions.
Thedataset description and train/test split details are provided in the supplementary material.
4.2 Pre-processing and Evaluation
We used NetVLAD with PCA+Whitening [4] as our image descriptor of size D = 4096 , computedby downsampling images to a resolution of 640 × 320 .
The learnt descriptors are retained to be of thesame size D to enable fair comparison against vanilla NetVLAD baselines.
For sequence matching, L = 5 is used for both training and testing.
For testing, localization radius of 10 meters, 20 metersand 10 frames is used respectively for Oxford, Brisbane/MSLS and Nordland 2 dataset.
We use Recall@K [96] as the evaluation metric as also used by previous works [4, 93, 87, 97].
Itis deﬁned as the ratio of queries that are correctly matched within its top-K retrieved images to thetotal number of queries.
Please refer to the supplementary material for training parameters details.
Results and Discussion
In this section, we ﬁrst present the key results of improving VPR using sequence matching withinthe training pipeline both as a loss function and for mining negatives, which is compared against thevanilla method of using single image based distance metrics.
Then, we present results for cross-citytesting of models trained on different cities to observe generalization trends.
Finally, we presentresults for state-of-the-art VPR comparisons considering a variety of sequence-based approachesalong with an ablation study on the length of sequence used.
In order to better understand theperformance variations between single image and sequence-based methods, we present a study oncross-testing single/sequence matching with/without sequence-based training in Section 5.2 of thesupplementary material.
5.1 Sequence Matching based Loss and Negative Mining
Table 1 shows how the choice of matching metric affects the recall performance when deﬁning theloss and mining negatives.
For this, we consider Equation 3 for computing loss and negatives duringtraining using a sequence length of 1 or 5 , referred to as Single and SeqMatch respectively in Table 1.
Results reported on the train/test splits of each of the three datasets are based on the use of sequencematching for test-time query retrieval.
It can be observed that given the retrieval task of sequencematching, using SeqMatch as a metric during training for both loss and mining leads to superiorperformance for all the datasets (last row: SeqMatch-SeqMatch) as compared to when not using itat all (ﬁrst row: Single-Single).
Furthermore, when using a cross-combination (second and thirdrow), use of SeqMatch either retains or improves performance.
In particular, it can be observed thatthe Oxford dataset necessarily requires SeqMatch-based loss to improve performance, leading to anabsolute increase in Recall@1 by 3% .
On the other hand, the Nordland dataset beneﬁts more from
SeqMatch-based mining, leading to a 7% absolute performance improvement.
This study showsthe importance of optimizing single image descriptors for sequence matching-based VPR using
SeqMatch as the metric (Equation 3) during training.
[Table]
5.2 Cross-City VPR Performance
In Table
[Table]
we present sequence matching results for training on one city and testing on the other.
Even though the environmental conditions vary across different cities, cross-city generalizationtrends look promising.
For example, SeqMatchNet-based Recall@1 for Oxford ( 0 . 70 ) and Brisbane( 0 . 29 ) during cross-testing is signiﬁcantly better than the use of sequence matching on top of vanilla
NetVLAD which respectively achieves 0 .
67 and 0 .
21 (in the ﬁrst row, referred to as No Transformation ).
A similar trend in relative performance improvements can be observed for MSLS-Amman,where trained models from all three cities lead to higher recall.
[Table]
0.26 / 0.36 / 0.41 Brisbane 0.70/0.84/0.
0.26 / 0.35 / 0.41 MSLS-Melbourne 0.58/0.72/0.
[Table]
[Table]
5.3 Benchmarking against State-of-the-Art Sequence-based Methods
Sequence-based information can be leveraged in different ways to improve single image based VPRperformance.
In Table 3, we compare against a variety of methods as following.
1) Sequentialdescriptors refer to single summary vector representation constructed through a sequence of images,which includes Smoothing/Averaging [92], Delta Descriptors [92] and SeqNet [93].
Sequencematching refers to aggregation of single image based distances which can either be based on thediagonal of the matrix or graph-based path search within the matrix [58, 5].
For the former, weconsider SeqNet’s single frame version S 1 which is the same as using SeqMatchNet but with L = 1in Equation 3, that is, without guiding single image descriptors with sequence matching metric.
For the latter, we consider Graph-based Image Sequence Matcher [58], referred to as GISM, and
Graph-based Relocalization using LSH [5], referred to as GRH.
3) Hierarchical sequence matchingis considered as per [93] where sequential descriptors shortlist candidates for sequence matching.
The ﬁrst row in Table 3 does not use any sequential information and corresponds to single imagebased VPR baseline.
For all other methods, we use a sequence length of L = 5 .
For all methodsthat use training, results are presented using models trained on the other city.
The best results withinany block are boldfaced and the overall best are additionally italicized.
The underlying single imagedescriptor for all the methods is either NetVLAD [4] or an adaptation of NetVLAD as cited.
It can be observed that in most cases, use of sequential information improves performance oversingle image descriptors as expected.
SeqMatchNet establishes a new state-of-the-art on the Oxforddataset considering Recall@K(5,20) and achieves the best results on the Brisbane dataset within thesequence matching block.
Among the sequential descriptors, SeqNet [93] achieves state-of-the-artresults on the Brisbane dataset but these results deteriorate when using S 5 within HVPR due toinferior sequence matching of S 1 .
However, this is not the case with the HVPR combination of
S 5 and SeqMatchNet, where Recall@1 improves over the baseline HVPR while maintaining theoverall state-of-the-art Recall@K(5,20).
This shows the relevance of improved sequence matchingwith SeqMatchNet which beneﬁts from the inclusion of sequence matching in its training pipeline.
5.4 Effect of Sequence Length
Table 4 shows the effect of sequence length on the recall performance of the proposed method ascompared to SeqNet [93] and vanilla NetVLAD with sequence matching.
For both the datasets(tested using models trained on the other city), it can be observed that SeqNet’s performance deteriorates with an increase in the sequence length.
This can be attributed to its sequence average poolinglayer which can potentially reduce the discriminative ability of the descriptors during retrieval whenusing longer sequences for such orderless averaging.
On the other hand, in the proposed SeqMatch- Net, averaging of distances between corresponding pairs of images within a sequence enforces anexplicit order (as a linear identity warp between two short time-series), leading to state-of-the-art performance especially when longer sequence lengths are used. The proposed method also outperformsvanilla NetVLAD based sequence matching which shows that the learnt descriptor transformation through SeqMatchNet leads to robust representations suited to the task. 5.5 Visualizing Query-Negative Distance Proﬁles
In our proposed method, we use sequence matching based hard negative mining as a mean to obtain harder negatives than what is achievable through single descriptor matching.
In Figure 2, weshow the mean (left) and standard deviation (right) of distances between a query and its top 50 hardnegatives, where negatives are computed using either single descriptors (orange) or sequence-basedmatching (blue).
In the mean negative distance plot on the left, it can be observed that single descriptors based matching (orange) leads to negative selection solely based on the central elements ofthe sequence at query frame index t .
This is apparent from the dip in its distance proﬁle at t whiledisregarding the corresponding distance values in its vicinity.
On the other hand, sequence-basednegative mining (blue) simultaneously considers all the elements in the sequence and is also able toselect negatives when the central element is not necessarily the hardest one but the overall sequenceindeed is.
Conclusion
Recent advances in visual place recognition have been accelerated by learning based techniques,which are predominantly governed by single image based distance metrics.
The use of sequentialinformation, although well established in the literature, has never been explored in the context ofbiasing single image representations to suit the subsequent task of sequence matching.
In this paper,we bridge this gap for the ﬁrst time through SeqMatchNet and show that overall VPR performancecan be improved by including sequence matching in the training pipeline by adapting an existing single image descriptor.
For this, we proposed temporal order-preserved average of Euclidean distancesas the sequence matching metric which is used to both compute the loss and mine negatives online.
To reduce the overhead of sequence matching for the latter, we proposed a novel 2D convolutionbased processing of the distance matrix.
We established that the single image descriptors learntthrough SeqMatchNet are more robust and lead to superior performance with the help of sequencematching metric as opposed to the use of single image matching metrics.
In mobile robotics, localization pipelines are typically comprised of several steps of robustness inducing processes.
The same is true of VPR where single image global representation matching isfollowed by performance enhancement techniques like sequential processing [20, 21], local featurematching [98, 42], feature ﬁltering [99, 100], query expansion [101], and even dimensionality reduction [4].
One can expect that including these subsequent processes in the representation learningprocess can improve the overall robustness.
This work takes a step in that direction by the inclusionof sequence matching in the training itself which leads to consistent performance gains.
This studywill hopefully encourage the researchers to reconsider the existing VPR and localization pipelineswhich can beneﬁt from end-to-end task-relevant learning.