L EARNING S UPER - FEATURES FOR I MAGE R ETRIEVAL
Anonymous authors
Paper under double-blind review
A BSTRACT
Methods that combine local and global features have recently shown excellent performance on multiple challenging deep image retrieval benchmarks, but their use of local features raises at least two issues.
First, these local features simply boil down to the localized map activations of a neural network, and hence can be extremely redundant.
Second, they are typically trained with a global loss that only acts on top of an aggregation of local features; by contrast, testing is based on local feature matching, which creates a discrepancy between training and testing.
In this paper, we propose a novel architecture for deep image retrieval, based solely on mid-level features that we call Super-features .
These Super-features are constructed by an iterative attention module and constitute an ordered set in which each element focuses on a localized yet discriminant image pattern.
For training, only image labels sufﬁce.
A contrastive loss operates directly at the level of Superfeatures and focuses on Super-features that match across images.
It is combined with a second complementary loss which encourages diversity.
Experiments on common landmark retrieval benchmarks validate that Super-features substantially outperform state-of-the-art methods when using the same number of features, and only require a signiﬁcantly smaller memory footprint to match their performance.
I NTRODUCTION
Image retrieval is a task that models exemplar-based recognition, i.e .
a class-agnostic, ﬁne-grained understanding task which requires to retrieve all images matching a query image over an (often very large) image collection.
It requires learning features that are discriminative enough for a highly detailed visual understanding but also robust enough to extreme viewpoint/pose or illumination changes.
A popular image retrieval task is landmark retrieval, whose goal is to single out images of the exact same landmark out of millions of images, possibly containing a different landmark from the exact same ﬁne-grained class ( e.g . ‘gothic-era churches with twin bell towers’).
While early approaches relied on handcrafted local descriptors, recent methods use image-level (global) or local Convolutional Neural Networks (CNN) features, see Csurka & Humenberger (2018) for a review.
The current state of the art performs matching or re-ranking using CNN-based local features (Noh et al., 2017;
Cao et al., 2020; Tolias et al., 2020) and only learns with global ( i.e . image-level) annotations and losses.
This is done by aggregating all local features into a global representation on which the loss is applied, creating a discrepancy between training and inference.
It has been shown that attention maps from modules like the ones in (Vaswani et al., 2017) are able to capture intermediate scene-level information, which makes them fundamentally similar to mid-level features (Xiao et al., 2015; Chen et al., 2019).
Unlike individual neurons in CNNs which are highly localized, attention maps may span the full input tensor and focus on more global or semantic patterns.
Yet, the applicability of mid-level features for instance-level recognition and image retrieval is currently underwhelming; we argue that this is due to the following reasons: generic attention maps are not localized and may ﬁre on multiple unrelated locations; at the same time, object-centric attentions such as the one presented in Locatello et al.
(2020) produce too few attentional features and there is no mechanism to supervise them individually.
In both cases, methods apply supervision at the global level, and the produced attentional features are simply not discriminative enough.
In this paper, we present a novel image representation and training framework based solely on attentional mid-level features we call Super-features .
We introduce an iterative Local feature Integration
Transformer (LIT), which tailors existing attention modules to the task of image retrieval.
Compared to the slot attention (Locatello et al., 2020), for example, it is able to output an ordered and much larger set of features, as it is based on learned templates, and has a simpliﬁed recurrence mechanism.
For learning, we devise a loss that is applied directly to Super-features, yet only requires image-level annotations.
It pairs a contrastive loss on a set of matching Super-features across matching images, with a decorrelation loss on the attention maps of each image, to encourage Super-feature diversity.
In the end, for each image our network extracts a ﬁxed-size set of Super-features that are semantically ordered, i.e ., each answering to different types of patterns; see Figure 1 for some examples.
At test time, we follow the protocol of the best performing recent retrieval methods, except that we aggregate and match Super-features instead of local features.
Our experiments show that the proposed method signiﬁcantly outperforms the state of the art on common benchmarks like (cid:82) Oxford and (cid:82) Paris (Radenovi ´ c et al., 2018a), while requiring less memory.
We further show that performance gains persist in the larger scale, i.e . after adding 1M distractor images, while exhaustive ablations suggest that Super-features are less redundant and more discriminative than local features.
Contributions.
Our contribution is threefold: (a) an image representation based on mid-level features ( Super-features ) and an iterative module to extract them; (b) a framework to learn such representations, based on a loss applied directly on Super-features yet only requiring image-level labels; (c) extensive evaluations that show signiﬁcant performance gains over the state of the art for landmark image retrieval.
We call our method Feature Integration-based Retrieval or FIRe for short.
B ACKGROUND : L EARNING LOCAL FEATURES WITH A GLOBAL LOSS
Let function f : (cid:73) Ñ R W ˆ H
ˆ D denote a convolutional neural network (CNN) backbone that encodes an input image x P (cid:73) into a p W ˆ H ˆ D q -sized tensor of D -dimensional local activations over a p W ˆ H q spatial grid.
After ﬂattening the spatial dimensions, the output of f can also be seen as set of L “ W ¨ H feature vectors denoted by (cid:85) “ t u l P R D : l P 1 ..
L u ; note that the size of this set varies with the resolution of the input image.
These local features are then typically whitened and their dimension reduced, a process that we represent by function o p¨q in this paper.
Global representations, i.e . image-level feature vectors, are commonly produced by averaging all local features, e.g . via global average pooling (Babenko & Lempitsky, 2015).
A global contrastive loss for training.
Tolias et al.
(2020) argue that optimizing global representations is a good surrogate for learning local features to be used together with efﬁcient match kernels for image retrieval.
When building their global representation g p (cid:85) q , they weight the contribution of [Figure] each local feature to the aggregated vector using its l 2 norm:
[Equation]
where }¨} 2 denotes the l 2 norm.
Given a database where each image pair is annotated as matching with each other or not, they propose minimizing a contrastive loss over tuples of global representations.
Intuitively, this loss encourages the global representations of matching images to be similar and those of non-matching images to be dissimilar.
Let tuple p (cid:85) , (cid:85) ` , (cid:86) ´ 1 , . . .
, (cid:86) ´
n q represent the sets of local features of images p x , x ` , y ´ 1 , . .
, y ´ n q , where x , x ` are matching images ( i.e . a positive pair) and y ´ 1 , . .
y ´ n are not matching image x
( i.e . are negatives).
Let r¨s ` denote the positive part and µ a margin hyper-parameter, they deﬁne a contrastive loss over global representations as:
[Equation]
In HOW , Tolias et al.
(2020) employ the global contrastive loss of Eq.(2) to learn a model whose local features are then used with match kernels such as ASMK (Tolias et al., 2013) for image retrieval.
ASMK is a matching process deﬁned over selective matching kernels of local features; it is a much stricter and more precise matching function than comparing global representations, and is crucial for achieving good performance.
By learning solely using a loss deﬁned over global representations and directly using ASMK over local features (cid:85) , HOW achieves excellent image retrieval performance.
L EARNING WITH S UPER - FEATURES
Methods using a global loss and then use local features (cid:85) for matching have a number of disadvantages.
First, using local activations as local features leads to high redundancy, as they exhaustively cover highly overlapping patches of the input image.
Second, using ASMK on features from a model trained with a global loss introduces a mismatch between training and testing: the local features used for ASMK are only trained implicitly , but are expected to individually match in the matching kernel.
To obtain less redundant feature sets, we propose to learn mid-level features we call Super-features using a loss function that operates directly on those Super-features, and also use the latter during retrieval; the train/testing discrepancy of the pipeline presented in Section 2 is thus eliminated.
In this section, we ﬁrst introduce the Local feature Integration Transformer ( LIT ), an iterative attention module which produces an ordered set of Super-features (Section 3.1).
We then present a framework for effectively learning such features (Section 3.2) that consists of two complementary losses: A contrastive loss that matches individual Super-features across positive image pairs, and a decorrelation loss on the Super-feature attention maps that encourages them to be diverse.
An overview of the full pipeline is depicted in Figure 2.
We refer to our approach as Feature Integration-based Retrieval or FIRe for short, an homage to the feature integration theory of Treisman & Gelade (1980).
3. L OCAL FEATURE I NTEGRATION T RANSFORMER (LIT)
Inspired by the recent success of attention mechanisms for encoding semantics from global context in sequences (Vaswani et al., 2017) or images (Caron et al., 2021), we rely on attention to design our
Local feature Integration Transformer ( LIT ), a module that outputs an ordered set of Super-features .
Let LIT be represented by function Φ p (cid:85) q : R L ˆ D Ñ R N ˆ d that takes as input the set of local features (cid:85) and outputs N Super-features.
We deﬁne LIT as an iterative module:
[Equation]
where φ denotes the core function of the module applied T times, and (cid:81)
P R N ˆ d denotes a set of learnable templates , i.e . a matrix of learnable parameters.
Super-features are progressively formed by iterative reﬁnement of the templates, conditioned on the local features from the CNN.
The architecture of the core function φ is inspired by the Transformer architecture (Vaswani et al., 2017) and is composed of a dot-product attention function ψ , followed by a multi-layer perceptron (MLP).
The dot-product attention function ψ receives three inputs, the key , the value and the querywhich are passed through layer normalization and fed to linear projection functions K , V and Q that project them to dimensions d k , d v and d q , respectively. In practice, we set d k “ d v “ d q “ d “ 1024 .
The key and value inputs are set as the local features u l P (cid:85) across all iterations.
The query input is the set of templates (cid:81) t “ t q tn P R d , n “ 1 .. N u .
It is initialized as the learnable templates (cid:81) 0 for iteration 0 , and is set as the previous output of function φ for the following iterations.
After projecting with the corresponding linear projection functions, the key and the query are multiplied to construct a set of N attention maps over the local features, i.e ., the columns of matrix α P R L ˆ N given by:
[Equation]
The dot product between keys and templates can be interpreted as a tensor of compatibility scores between local features and templates.
These scores are normalized across templates via a softmax function, and are further turned into attention maps by l 1 normalization across all L spatial locations.
This is a common way of approximating a joint normalization function across rows and columns, also used by Locatello et al.
The input value is ﬁrst projected with V , re-weighted with the attention maps and then residually fed to a MLP 3 to produce the output of function φ :
[Equation]
Following standard image retrieval practice, we further whiten and l 2 -normalize the output of Φ p (cid:85) q to get the ﬁnal set of Super-features.
Speciﬁcally, let Φ p (cid:85) q “ r ˆ s 1 ; . .
; ˆ s N s P R N ˆ d be the raw output of our iterative attention module, we deﬁne the ordered set of Super-features as:
[Equation]
where, as in Section 2, o p¨q denotes dimensionality reduction and whitening.
Figure 2 (right) illustrates the architecture of LIT.
Note that all learnable parameters of φ are shared across all iterations.
What do Super-features attend to?
Each Super-feature in (cid:83) is a function of all local features in (cid:85) , invariant to permutation of its elements, and can thus attend arbitrary regions of the image.
To visualize the patterns captured by Super-features, Figure 1 shows the attention maps of the same ﬁve Super-features for three images, including two of the same landmark (ﬁrst two rows).
The type of patterns depends on the learned initialization templates (cid:81) 0 ; this explains why the Super-features form an ordered set and allows to directly compare Super-features with the same ID.
We observe the attention maps to be similar across images of the same landmark and to contain some mid-level patterns (such as a half-circle on the second column, or windows on the third one).
3. L EARNING WITH S UPER - FEATURES
We jointly ﬁne-tune the parameters of the CNN and of the LIT module using contrastive learning.
However, unlike recent approaches like HOW (Tolias et al., 2020) or DELG (Cao et al., 2020) that use a global contrastive loss similar to the one presented in Eq.
(2) for learning, we introduce a contrastive loss that operates directly on Super-features, the representations we use at test time , and yet only requires image-level labels.
Given a positive pair ( i.e . matching images of the same landmark) and a set of negatives, we select promising Super-feature pairs by exploiting their ordered nature and without requiring any extra supervision signal.
We then minimize their pairwise distance, while simultaneously reducing the spatial redundancy of Super-features within an image.
Selecting matching Super-features.
Since we are only provided with pairs of matching images, i.e . image-level labels, deﬁning correspondences at the Super-feature level is not trivial.
Instead of approximating sophisticated metrics as in (Liu et al.,
; Mialon et al., 2021)–see Section 5 for a discussion–, we leverage the fact that Super-features are ordered and we select promising matches and ﬁlter out erroneous ones only relying on simple, nearest neighbor-based constraints.
For any s P (cid:83) , let (cid:105) p s q be the function that returns the position/order or Super-feature ID , i.e .
(cid:105) p s i q “ i, @
s i P (cid:83) .
Further let function n p s , (cid:83) q “ arg min s i P (cid:83) } s ´ s i } 2 be the function that returns the nearest neighbor of s from set (cid:83) .
Now, given a positive pair of images x , x ` , let s P (cid:83) , s 1 P (cid:83) 1 be two Super-features from their Super-feature sets (cid:83) , (cid:83) 1 , respectively.
In order for
Super-feature pair p s , s 1 q to be eligible , all of the following criteria must be met: a) s , s 1 have to be reciprocal nearest neighbors, b) they need to pass Lowe’s ﬁrst-to-second neighbor ratio test (Lowe, 2004) and c) they need to have the same Super-feature ID.
Let (cid:80) be the set of eligible pairs and τ the hyper-parameter that controls the ratio test; the conditions above can formally be written as:
[Equation]
We set τ “ 0 . 9 .
Our ablations (Section 4.1) show that all four criteria are important.
Note that the pair selection step is non-differentiable and no gradients are computed for Super-features not in (cid:80) .
We further discuss this in Appendix B.4 and empirically show that all Super-features are trained.
A contrastive loss on Super-features.
Once the set (cid:80) of all eligible Super-feature pairs has been constructed, we deﬁne a contrastive margin loss on these matches.
Let pair p “ p s , s ` q P (cid:80) be a pair of Super-features, selected from a pair of matching images x , x ` , and let (cid:78) p j q “ t n kj : k “ 1 .. n u for j “ 1 .. N be the set of Super-features with Super-feature ID j extracted from negative images p y ´ 1 , . . .
, y ´ n q .
The contrastive Super-feature loss can be written as:
[Equation]
where µ 1 is a margin hyper-parameter and the negatives for each s are the Super-features from all n negative images of the training tuple with Super-feature ID equal to i p s q .
Reducing the spatial correlation between attention maps.
To obtain Super-features that are as complementary as possible, we encourage them to attend to different local features, i.e . different locations of the image.
To this end, we minimize the cosine similarly between the attention maps of all Super-features of every image.
Speciﬁcally, let matrix α “ r ˜ α 1 , . . . , ˜ α N s now be seen as column vectors denoting the N attention maps after the last iteration of LIT.
The attention decorrelation loss is given by:
[Equation]
In other words, this loss minimizes the off-diagonal elements of the N ˆ N self-correlation matrix of ˜ α .
We ablate the gains from this loss and all others presented in this section in Section 4.1.
Image retrieval with Super-features.
Our full pipeline, FIRe, is composed of a model that outputs
Super-features, trained with the contrastive Super-feature loss of Eq.(8) and the attention decorrelation loss of Eq.(9).
As our ablations show (see Section 4), the combination of these two losses is required for achieving state-of-the-art performance, while adding a third loss on the aggregated global features as in Eq.(2) does not bring any gain.
[Table]
E XPERIMENTS
This section validates our proposed FIRe approach on standard landmark retrieval tasks.
We use the SfM-120k dataset (Radenovi´c et al., 2018b) following the 551/162 3D model train/val split from Tolias et al.
For testing, we evaluate instance-level search on the (cid:82) Oxford (Philbin et al., 2007) and
the (cid:82) Paris (Philbin et al., 2008) datasets in their revisited version (Radenovi´c et al., 2018a), with and without the 1 million distractor set called (cid:82) 1M.
They both contain 70 queries, with 4993 and 6322 images respectively.
We report mAP on the Medium (med) and Hard (hard) setups, or the average over these two setups (avg).
Image search with Super-features.
At test time, we follow the exact procedure described in
Tolias et al.
and extract Super-features from each image at 7 resolutions/scales { 2.0, 1.414, 1.0, 0.707, 0.5, 0.353, 0.25 } .
We then keep the top Super-features (the top 1000 unless otherwise stated) according to their L2 norm and use the binary version of ASMK with a codebook ofclusters.
Note that one can measure the memory footprint of a database image x via the number of non-empty ASMK clusters, i.e . clusters with at least one assignment, denoted as | (cid:67) p x q| .
Implementation details.
We build our codebase on HOW (Tolias et al., 2020) 4 and sample tuples composed of one query image, one positive image and 5 hard negatives.
Each epoch is composed of 400 batches of 5 tuples each, while hard negatives are updated at each epoch using the global features of Eq.(1).
We train our model for 200 epochs on SfM-120k using an initial learning rate of 3 .
10 ´ 5 and random ﬂipping as data augmentation.
We multiply the learning rate by a factor of 0 .at each epoch, and use an Adam optimizer with a weight decay of 10 ´ 4 .
We use a ResNet50 (He et al., 2016) without the last convolutional block as backbone (R50 ´ ).
For LIT, we use D “ d k “ d q “ d v “ d “ 1024 .
Following HOW, we reduce the dimensionality of features to 128 and initialize o p¨q using PCA and whitening before training and keep it frozen.
We pretrain the LIT module with the backbone on ImageNet-1K for image classiﬁcation, see details in Appendix B.6.
We use µ 1 “ 1 . 1 in Eq.(8) and weight (cid:76) super and (cid:76) attn with 0 .
02 and 0 . 1 , respectively.
4. A NALYSIS AND ABLATIONS
In this section, we analyze the proposed FIRe framework and perform exhaustive ablations on the training losses and the matching constraints.
The impact of the number of iterations ( T ) and templates ( N ) in LIT are studied in Appendix A.1.
For the rest of the paper, we set T “ 6 and N “ 256 .
Matching constraints ablation.
Table 1 reports the impact of the different matching constraints: nearest neighbor, reciprocity, Lowe ratio test and constraint on Super-features ID.
Adding reciprocity and Lowe’s ratio test signiﬁcantly improves performance, which indicates that it reduces the number of incorrect matches.
Keeping all pairs with the same ID yields lower performance: the ID constraint alone is not enough.
One possible explanation is that two images from the same landmark may differ signiﬁcantly, e.g . when some parts are visible in only one of the two images.
In that case, the other constraints allow features attending non-overlapping regions of a positive image pair to be excluded from (cid:80) .
Finally, combining the selective ID constraint with the others yields the best performance.
In order to better understand the impact of the Super-feature ID constraint, a constraint only applicable for Super-features due to their ordered nature, we measure the quality of selected matches during training with and without it. Since there is no ground-truth for such localized matches on landmark
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
C o rr e l a t i o n high low
[Figure] retrieval datasets, we measure instead the ratio of matches coming from the positive pair, over all matches (from the positive and all negatives).
Ideally, a minimal number of matches should come from the negatives, hence this ratio should be close to 1.
In Figure 3 we plot this match ratio for all epochs; we observe that it is signiﬁcantly higher when using the Super-features ID constraint.
Training losses ablation.
We study the impact of the different training losses in Table 2.
We start from a global loss similar to HOW (ﬁrst row).
We then add the decorrelation loss (second row) and observe a clear gain.
It can be explained by the fact that without this loss, Super-features tend to be redundant (see the correlation matrices in Figure 4).
Adding the loss operating directly on the Super-features further improves performance (third row).
Next, we remove the global loss and keep only the loss on Super-features alone (fourth row) or with the decorrelation loss (last row).
The latter performs best (fourth vs last row).
Figure 4 displays the correlation matrix of Superfeatures attention maps with and without (cid:76) attn .
Without it, we observe that most Super-features have correlated attentions.
In contrast, training with (cid:76) attn leads to uncorrelated attention maps.
This is illustrated in Figure 1 which shows several attention maps focusing on different areas.
Varying the number of features at test time.
Figure 5 compares HOW (Tolias et al., 2020) with our approach, as we vary the number of local features / Super-features.
The x axis shows the average number of clusters used in ASMK for the database images, i.e ., which is proportional to the average memory footprint of an image.
We observe that our approach does not only signiﬁcantly improve accuracy compared to HOW, but also requires overall less memory.
Notably, FIRe matches the best performance of HOW with a memory footprint reduced by a factor of 4 .
For both methods, performance goes down once feature selection no longer discards background features.
The gain in terms of performance is even larger when considering a single scale at test time (see Appendix B.1).
Statistics on the number of selected features per scale.
The left plot of Figure 6 shows the percentage of the 1000 selected features that comes from each scale, for HOW and FIRe.
For HOW most selected features come from the higher resolution inputs; by contrast, selected Super-features are almost uniformly distributed across scales.
Interestingly, Figure 6 (middle) shows that HOW keeps a higher percentage of the features coming from coarser scales.
Yet, the ﬁnal feature selection [Figure] for HOW is still dominated by features from higher scales, due to the fact that the number of local features signiﬁcantly increases with the input resolution (see Figure 6 right).
4. C OMPARISON TO THE STATE OF THE ART
We compare our method to the state of the art in Table 3.
All reported methods are trained on SfM120k for a fair comparison 5 .
First, we observe that methods based on global descriptors (Gordo et al., 2016; Revaud et al., 2019a) compared with a L 2 distance tend to be less robust than methods based on local features (Noh et al., 2017;
Tolias et al., 2020).
DELG (Cao et al., 2020) shows better performance, owing to a re-ranking step based on local features, at the cost of re-extracting local features for the top-ranked images of a given query, given that local features would take too much memory to store.
Our FIRe method outperforms DELF (Noh et al., 2017) as well as HOW (Tolias et al., 2020) by a signiﬁcant margin when extracting 1000 features per image.
Importantly, our approach also requires less memory, as it uses fewer ASMK clusters per image, as shown in Figure 5: for the whole (cid:82) 1M set, HOW uses 469 clusters per image while FIRe uses only 383, on average.
R ELATED W ORK
Image descriptors for retrieval.
The ﬁrst approaches for image retrieval were based on handcrafted local descriptors and bag-of-words representations borrowed from text retrieval (Sivic & Zisserman, 2003; Csurka et al., 2004), or other aggregation techniques like Fisher Vectors (Perronnin et al., 2010), VLAD (J ´ egou et al., 2010) or ASMK (Tolias et al., 2013).
First deep learning techniques were extracting a global vector per image either directly or by aggregating local activations (Radenovic et al., 2018) and have shown to highly outperform handcrafted local features, see Csurka & Humenberger (2018) for a review.
Methods that perform matching or re-ranking using CNN-based local features are currently the state of the art in the area (Noh et al., 2017; Teichmann et al., 2019; Cao et al., 2020; Tolias et al., 2020).
They are able to learn with global ( i.e . image-level) annotations.
Most of them use simple variants of attention mechanisms (Noh et al., 2017; Ng et al., 2020) or simply the feature norm (Tolias et al., 2020) to weight local activations.
Low-level features aggregation.
Aggregating local features into richer regional features has a long research history; crucial to the success of traditional descriptor based approaches, popular methods include selecting discriminative patches in the input image (Singh et al., 2012; Gordo, 2015), regressing saliency map scores at different resolutions (Jiang et al., 2013), aggregating SIFT descriptors with coding and pooling schemes (Boureau et al., 2010), or mining frequent patterns in sets of SIFT features (Singh et al., 2012). More recently, (Hausler et al., 2021) introduced a multi-scale fusion of patch features.
Predicting mid-level representations of input images, such as depth maps or segmentations, has also been used as auxiliary objectives to improve training in reinforcement learning (Sax et al., 2019).
Biological studies have shown that similar phenomena are at play in the human brain (Long et al., 2018).
[Table]
Supervision for local features.
Several works focus on providing supervision at the level of local features in the context of contrastive learning.
Xie et al.
(2021) and Chen et al.
(2021) obtain several views of an input image using data augmentations with known pixel displacements, such as crops and offsets.
Similarly, Liu et al.
(2020) train a model to predict the probability for a pair of local features to match, evaluated using known displacements.
Wang et al.
(2020) and Zhou et al.
(2021) obtain local supervision by relying on epipolar coordinates and relative camera poses between shots.
Positive pairs in image retrieval depart from these setups, as pixel-level correspondences cannot be known.
To build matches without such signal, Wang et al.
(2021a) use a standard nearest neighbor algorithm to build pairs of features, similarly to our approach, but without the use of ﬁltering which is critical to our ﬁnal performance.
Using the current model predictions to deﬁne targets is reminiscent of modern self-supervised learning approaches which learn without any label (Caron et al., 2021;
Grill et al., 2020).
The additional ﬁltering step can be seen as a way to keep only the most reliable model predictions, similar to self-distillation as in e.g .
(Sohn et al., 2020).
Other recent approaches for supervising local features include exploring set-level optimization strategies to deﬁne set-level alignment (Wang et al., 2021b) or approximating optimal transport kernels (Mialon et al., 2021).
Attention modules.
Our LIT module is an iterative variant of standard attention (Bahdanau et al., 2015; Vaswani et al., 2017), adapted to map a variable number of input features to an ordered set of N output features, similar to Lee et al.
The Perceiver model (Jaegle et al., 2021) has demonstrated the ﬂexibility of such constructions by using it to scale attention-based deep networks to large inputs.
Our design was heavily inspired by slot-attention (Locatello et al., 2020), but has some key differences that enable us to achieve high performance in more complex visual environments: a) unlike the slot attention which initializes its slots with i.i.d sampling, we learn the initial templates and therefore deﬁne an ordering on the output set, a crucial property for selecting promising matches; b) we replace the recurrent network gates with a residual connection across iterations.
These modiﬁcations, together with the attention decorrelation loss enable our module to go from a handful of object-oriented slots to a much larger set of mid-level features.
For object detection, Carion et al.
(2020) rely on a set of learned object queries to initialize a stack of transformer layers.
Unlike ours, their module is not recurrent; Appendix A.1 experimentally shows substantial beneﬁts from applying LIT T times, with weight sharing, to increase model ﬂexibility without extra parameters.
C ONCLUSIONS
We present an approach for encoding mid-level features for image retrieval, a task that has up to now been dominated by approaches that work at the local feature level.
We design an attention mechanism that outputs an ordered set of such features that are more discriminative and expressive than local features.
Exploiting their ordered nature and without any extra supervision, we present a loss working directly on these mid-level features Our method not only signiﬁcantly improves performance, but also requires less memory, a crucial requirement for scalable image retrieval.
Appendix Table of Contents A
Additional
A.1 Impact of the iteration and the number of templates hyper-parameters . . . . . .
A.2 Impact of hard negatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Impact of the update function . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Extended ablation on matching constraints . .
A.5 Impact of the template initialization . . . . . . . . . . . . . . . . . . . . . . . .
B Analysis and discussions
B.1 Single-scale results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Consistency of attention maps across scales . . . . . . . . . . . . . . . . . . . .
B.3 Redundancy in Super-features . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Are all Super-features trained? . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5 Using the Super-feature loss with local features . . . . . . . . . . . . . . . . . .
B.6 Pretraining the backbone together with LIT . . . . . . . . . . . . . . . . . . . .
B.7 Computational cost of Super-features extraction . . . . . . . . . . . . . . . . .
B.8 Is Google Landmarks v2 clean an appropriate training dataset for testing on (cid:82) Oxford and (cid:82) Paris? . . . . . . . . . . . . . . . . . . . . . . . [Table]
C Application to visual localization
In this appendix, we present additional ablations (Appendix A), a deeper analysis of our framework (Appendix B) as well as results of the application of our FIRe framework for image retrieval to the task of visual localization (Appendix C).
We brieﬂy summarize the ﬁndings in the following paragraphs.
Ablations.
We study the impact of some hyper-parameters of our model, namely the size N of the set of Super-features, and the number of iterations T in the LIT module, in Appendix A.1.
We show that 256 Super-features and 6 iterations offer the best trade-off between performance and computational cost.
We also show in Appendix A.2 that we obtain further performance gains of over 1% by increasing the number of negatives to 10 .
We then study the impact of replacing the residual connection inside the LIT module by a recurrent network (Appendix A.3).
In Appendix A.5 we present an extended version of Table 1 with further matching constraints and we ﬁnally study the impact of the template initialization on performance in Appendix A.4.
Properties of Super-features.
We show single-scale results in Appendix B.1.
In Appendix B.2, we display the attention maps of Super-features, at different scales for a ﬁxed Super-feature ID.
We further study the amount of redundancy in Super-features, compared to local features, in Appendix B.3.
Next, we verify in Appendix B.4 that all Super-features receive training signal, as a sanity check.
We discuss the case of applying a loss directly on local features in Appendix B.5 and give details about the pretraining on ImageNet in Appendix B.6.
In Appendix B.7 we report the average extraction time for the proposed Super-features, while in Appendix B.8 we discuss the fact that there is an overlap between the queries from the common (cid:82) Oxford and (cid:82) Paris datasets and the
Google Landmarks-v2-clean dataset that is commonly used as training set for retrieval on (cid:82) Oxford and (cid:82) Paris.
Finally, in Appendix B.9 we further discuss the rationale behind our choice of referring to the Super-features as ‘mid-level’.
Application to visual localization.
We further evaluate our model using a visual localization setting on the Aachen Day-Night v1.1 dataset (Sattler et al., 2018) in Appendix C. To do so, we leverage a retrieval + local feature matching pipeline, and show that it is beneﬁcial to use our method, especially in hard settings.
A A DDITIONAL ABLATIONS A. I MPACT OF THE ITERATION AND THE NUMBER OF TEMPLATES HYPER - PARAMETERS
In this section, we perform ablations on the number of Super-features N and the number of iterations
T in the Local feature Integration Transformer.
We ﬁrst study the impact of the number N of Superfeatures extracted for each scale of each image in Figure 7a.
We observe that the best value is 256 on both the validation and test sets ( (cid:82) Oxford and (cid:82) Paris).
We then study the impact of the number of iterations T in Figure 7b.
While the performance decreases when doing 2 or 3 iterations compared to just 1, a better performance is reached for 6 iterations, after which the performance saturates while requiring more computations.
We thus use T “ 6 .
[Figure]
A. I MPACT OF HARD NEGATIVES
Each training tuple is composed of one image pair depicting the same landmark, and 5 negative images ( i.e . from different landmarks).
We plot the performance when varying this number in Figure 8.
We observe that adding more negatives improves overall the performance on all datasets, i.e . by more than 1 % when increasing it to 10 negatives, but at the cost of a longer training time as more images need to be processed at each iteration.
This is why we use 5 negatives in the rest of the paper as it offers a good compromise between performance and cost.
[Figure]
A. I MPACT OF THE UPDATE FUNCTION
The formula for ψ in Equation (5) sums the previous (cid:81) with the output of the attention component α ¨ V p (cid:85) q , i.e ., with a residual connection.
This is a different choice than the one made in the object-centric slot attention of Locatello et al.
(2020), which proposes to use a Gated Recurrent
Unit: ψ p (cid:85) ; (cid:81) q
“ GRU p α ¨ V p (cid:85) q , (cid:81) ).
We thus compare the residual connection we use to a GRU function and report results in Table 4 with T “ 3 .
We observe that the residual connection reaches a better performance in all datasets while having the interest of not adding extra parameters.
[Table]
In summary, we propose the LIT module to obtain a few hundred mid-level features for image retrieval while slot attention handles a handful of object attentions.
Technical differences to the slot attention include: a) we use a learned initialization of the templates instead of i.i.d.
sampling, which allows to obtain an ordered set of features, and thus to apply the constraint on the ID for the matching, leading to a clear gain, see Table 1 and Figure 3, b) to handle a larger number of templates, we also add a decorrelation loss on the attention maps, which has clear beneﬁt, see Table 2 and
Figure 4, c) we use a residual connection with 6 iterations instead of a GRU which performs better for our task (see Table 4).
A. E XTENDED ABLATION ON MATCHING CONSTRAINTS
[Table]
We show in Table 5 an extended version of Table 1 of the main paper, where we evaluate all possible combinations of constraints among reciprocity, Lowe’s ratio test and the Super-feature ID.
We observe that the reciprocity constraint and the Super-features ID constraint are the two most important ones, while the Lowe’s ratio only only brings a small improvement
A. I MPACT OF THE TEMPLATE INITIALIZATION
In the LIT module, initial templates (cid:81) 0 P R N ˆ d are learned together with the LIT module.
We can therefore assume that they are adapted to the task at hand.
To explore the sensitivity to the initial templates, we run a variant of FIRe where the initializations are not ﬁnetuned, but instead frozen to the values after pretraining on ImageNet.
We report performance in Table 6.
We observe that the two variants perform overall similarly.
This ablation suggests that the initial templates are up to some point transferable to other tasks.
[Table]
[Figure]
HOW
FIRe
[Figure]
[Table]
HOW
FIRe
B A NALYSIS AND DISCUSSIONS B. S INGLE - SCALE RESULTS
Similar to Figure 5 of the main paper, we perform the same ablation in Figure 9 when while extracting features at a single scale ( 1 . 0 ).
We observe that FIRe signiﬁcantly improves the mAP on the two datasets compared to HOW.
The average number of clusters, i.e . the memory footprint of images, remains similar for both methods at a same number of selected features.
This stands in contrast to the multi-scale case where our approach allows to save memory (about 20%), which we hypothesize is due to the correlation of our features across scales that we discuss below.
B. C ONSISTENCY OF ATTENTION MAPS ACROSS SCALES
At test time, we extract Super-features at different image resolutions.
We propose here to study if the attention maps of the Super-features across different image scales are correlated.
We show in Figure 10 the attention maps at the last iteration of LIT for the different image scales.
We observe that they ﬁre at the same image location at the different scales.
Note that the attention maps are larger/smoother at small scale (right columns), as for visualization, we resize lower resolution attention maps to the original image size.
B. R EDUNDANCY IN S UPER - FEATURES
To evaluate the redundancy of Super-features versus local features, Figure 11 displays the average cosine similarity between every local features / Super-feature and its K nearest local features /
Super-features from the same image, for different values of K .
We observe that Super-features are signiﬁcantly less correlated to the most similar other ones, compared to local features used in HOW.
[Figure]
[Figure]
B. A RE ALL S UPER - FEATURES TRAINED ?
The loss in Equation (8) only operates on a subset of the Super-feature pairs that pass the criteria of Equation (7); if a Super-feature ID is not matched, it receives no training signal.
To investigate if all of the IDs contribute to the loss, we monitored how many times each ID is matched at each epoch and report the percentage over all 2000 training tuples per epoch.
In Figure 12 we report mean, standard deviation and minimum number of matches for the N Super-feature ID.
We clearly observe that all Super-features receive training signals regularly and each Super-feature is matched about one quarter of the time on average.
B. U SING THE S UPER - FEATURE LOSS WITH LOCAL FEATURES
It is worth noting that the loss in Equation (8) could also theoretically be used over local feature activations.
It could be computed on pairs of local features that pass the conditions described in
Equation (7), after removing the constraint on having the same Super-feature ID.
Unfortunately, we were unable to get any gains over HOW (Tolias et al., 2020) when appending such a loss side-byside with the global loss over local features.
Empirically, our ablations (see Section 4.1) show that adding the constraint on the Super-features ID, which is only possible with ordered feature sets, is key to the success of our approach, and signiﬁcantly improves the quality of the matching. [Figure] % m a t c
h e s p e r
S up e r -f ea t u r e
[Figure] B. P RETRAINING THE BACKBONE TOGETHER WITH LIT
Before training on landmark retrieval, we pretrain the backbone network, with the proposed LIT module appended after the last layer, for image classiﬁcation on ImageNet-1K. Given that we remove the last convolutional block similar to HOW, to avoid overﬁtting we further append a classiﬁcation head composed of one fully-connected layer, batch norm, leaky ReLU, dropout and another fully-connected layer at the end of the network and train it for 80 epochs using a standard crossentropy loss and in addition the decorrelation loss (cid:76) attn weighted by 0 .
1 , the same weight as during ﬁnetuning.
We start with a learning rate of 0 .
01 and divide it by 10 every 20 epochs.
This new architecture reaches a top-1 (resp.
top-5) accuracy of 73.48% (resp.
91.54%) on the ImageNet validation set.
Note that this performance is a bit lower than a standard ResNet-50: this is because we have removed the last convolutional block which contains most parameters in the ResNet architecture.
B. C OMPUTATIONAL COST OF S UPER - FEATURES EXTRACTION
We report the time required for extracting multi-scale features for 5000 images, for HOW and FIRe.
On our server, it took 157 seconds for HOW and 172 for FIRe, i.e . extraction for Super-features only requires 10% more wall-clock time.
B. I S G OOGLE L ANDMARKS V 2 CLEAN AN APPROPRIATE TRAINING DATASET FOR TESTING ON (cid:82) O XFORD AND (cid:82) P ARIS ?
In the comparison to the state of the art (Table 3), all reported methods are trained on the SfM120k dataset.
Several recent works have also released a model trained on the Google Landmarks v dataset (Weyand et al., 2020) or its clean version (Yokoo et al., 2020) with excellent performance on (cid:82) Paris.
However, we ﬁnd out that several of the query landmarks from (cid:82) Paris and (cid:82) Oxford were present in the training set of the cleaned version of Google Landmarks v2, such as ‘La Defense’, ‘Eiffel Tower’, ‘Sacr´e Coeur’ or ‘Hotel les Invalides’ for (cid:82) Paris or such as ‘Mary Madgalen’, ‘Bodleian
Library’, ’All Soulds College’ or ‘Radcliffe’ for (cid:82) Oxford.
B. O N USING THE TERM “ MID - LEVEL ” FOR OUR S UPER - FEATURES
Formulated as a learned weighted aggregation over the whole input image, each one of the proposed Super-features leverages global context.
Yet, we empirically observe that each one of them learns to attend to multiple localized areas of the image, and that many of them seem to be ‘more strongly responding’ to ‘mid-level’ patterns (see for example Figure 1).
Although each Super-feature is a weighted sum over features from all locations, in practice this sum is dominated by the features from only a small subset of locations.
It is noteworthy that this is not the case for attention maps of (self-)attention modules in general; this is prominent in our case due to the existence of L attn in
Eq 9 that explicitly minimizes the spatial correlation between attention maps (see also Figure 4).
For simplicity and possible emerging connections to other related works, we use the term ‘mid-level’ to refer to the set of features learned by our method.
C A PPLICATION TO VISUAL LOCALIZATION
In this section, we evaluate FIRe for the task of visual localization, where retrieval is used as a ﬁrst-stage ﬁltering and before more precise, local feature-based geometric matching.
To this end, we follow the pipeline proposed by Kapture 6 (Humenberger et al., 2020) on the Aachen Day-Night v1.1 dataset (Sattler et al., 2018).
In this scenario, a global Structure-from-Motion map is built from the training images using R2D2 local descriptors (Revaud et al., 2019b).
At test time, given a query image to localize, image retrieval is used to retrieve the top-50 nearest images.
On these retrieved images, R2D2 local features are extracted and matched with the ones on the query image, and this is used to estimate the position of the camera.
The percentage of successfully localized images within three levels of thresholds is then reported on the day or night images, following the visual localization benchmark protocol 7 , the latter ones being more challenging as training images are taken during daytime.
We compare our retrieval method to AP-GeM (Revaud et al., 2019a) and HOW (Tolias et al., 2020) and report results in Table 7.
AP-GeM is the default method used in Kapture (Humenberger et al., 2020).
We observe that using FIRe leads to better visual localization, specially in the most challenging scenario of night image localization and the strictest localization threshold: Performance improves by 2% compared to AP-GeM and by 1.5% compared to HOW on night images at a threshold of 0.25m and 2°.
Overall, for either day or night matching, FIRe either improves or performs on par to both methods compared.
[Table]