Paper: Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition (ICLR-2024)

Reviewer comment (Experiments):
Pitts250k is also a common VPR dataset. The proposed approach has shown excellent results on multiple benchmark datasets. Providing the results on the Pitts250k dataset might make the experiment more complete.

Author response:
Thanks for your encouraging words and constructive comments.

The results of our SelaVPR on Pitts250k are shown in the table below. MixVPR achieves the previous best results on Pitts250k, and SelaVPR outperforms MixVPR.

|  Method  |   | Pitts250k  |  |  
| :---: | :---: | :---: | :---: | 
|   | R@1 | R@5 | R@10 |
| MixVPR | 94.1 | 97.9 | 98.7 |
| SelaVPR | 95.7 | 98.8 | 99.2 |

Reviewer comment (Experiments):
Re-ranking top-100 candidates seems a common setting for two-stage VPR methods. However some works also show the performance with different numbers of re-ranking candidates [1], which can help other researchers choose the optimal number of candidates when using this method. I think it is also necessary to show the performance of different numbers of candidates.

Author response:
The results of re-ranking different numbers of candidates are shown in the table below. Performance roughly reaches saturation when re-ranking top-100 candidates, which is our recommended setting for optimal recognition performance. Re-ranking top-50 (or even top-20) candidates is also a good option for faster two-stage retrieval.

|  Method  |   | Tokyo24/7  |  |  |   | Pitts30k  |  |  |   | MSLS-val  |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|   | R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 |
| re-ranking top-20  | 93.0 | 96.2 | 97.1 |  | 92.7 | 96.6 | 97.5 |  | 90.7 | 96.4 | 97.0 |
| re-ranking top-50  | 93.7 | 96.5 | 96.8 |  | 92.8 | 96.8 | 97.6 |  | 90.8 | 96.4 | 97.0 |
| re-ranking top-100  | 94.0 | 96.8 | 97.5 |  | 92.8 | 96.8 | 97.7 |  | 90.8 | 96.4 | 97.2 |
| re-ranking top-200  | 94.0 | 96.8 | 97.5 |  | 92.8 | 96.9 | 97.8 |  | 90.7 | 96.5 | 97.3 |

Reviewer comment (Writing):
The gap between the tasks of model pre-training and VPR is not very clear.

Author response:
The gap between the model pre-training and VPR tasks primarily refers to the difference in training objectives and data, which further results in the pre-trained models focusing on different objects from the specially trained VPR models. A robust VPR model should focus on the static discriminative backgrounds (e.g., buildings and trees) and ignore the dynamic foreground objects (e.g., pedestrians and vehicles), but a pre-trained model cannot do this, as vividly shown in the attention map visualizations in our paper.

Reviewer comment (Evaluation):
This paper does not show failure cases, which can inform future research in VPR.

Author response:
The failure cases mainly occur in some natural scenes that lack discriminative landmarks, which is difficult to address for existing VPR methods. It’s the focus of our future work.

Reviewer comment (Experiments):
This work finetunes the models on the MSLS dataset and further finetunes them on Pitts30k to test on Pitts30k and Tokyo24/7, which is the same as R2Former. However, the R2Former work provides the result on Pitts30k of the model that only trained on MSLS, which can prove the model's transferability to the domain gap. Can the proposed SelaVPR still outperform R2Former on Pitts30k using only MSLS for training?

Author response:
The results of R$^2$Former and our SelaVPR on Pitts30k using only MSLS for training are shown in the table below. Our SelaVPR also outperforms R$^2$Former. The results of SelaVPR trained only on MSLS are even better than those of SelaVPR only trained on Pitts30k (similar to the case with R$^2$Former), showing that our method has good transferability as well.

|  Method  |   | Pitts30k  |  |  
| :---: | :---: | :---: | :---: | 
|    | R@1 | R@5 | R@10 |
| R$^2$Former | 88.4 | 94.2 | 95.7 |
| SelaVPR | 92.2 | 96.5 | 97.6 |

Reviewer comment (Writing):
The clarity of the paper could be further improved.

Author response:
For the weaknesses, we will improve the clarity in the revised paper and try to use the hybrid adaptation method for other visual tasks in future work. Thanks again for your suggestions.

Reviewer comment (Evaluation):
Why is L2 distance used to measure global feature similarity, but dot product used to calculate local feature similarity?

Author response:
**[A1:]** To make the first stage (global retrieval) compatible with some one-stage VPR methods and the benchmark [1], the L2 distance is used to compute global feature similarity. For the convenience of finding mutual matches (i.e. fast computing), local feature similarity is based on the inner product. Since local features have been L2 normalized, the inner product is equal to cosine similarity.

Reviewer comment (Experiments):
Is it feasible to re-rank top-k candidate images directly using coarse patch tokens from ViT? and how is the performance?

Author response:
**[A2:]** It’s feasible in terms of program implementation. However, Re-ranking candidates with the coarse patch tokens from ViT does not make sense in two-stage VPR. Although it provides an improvement (but not as good as ours) on Pitts30k compared to global retrieval, it performs significantly worse than global retrieval on MSLS-val, where VPR methods are more susceptible to perceptual aliasing. That is, re-ranking with coarse patch features will bring negative effects and damage the results of global retrieval (i.e. the first stage retrieval). 

|  Method  |   | Pitts30k  |  |  |   | MSLS-val  |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|    | R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 |
| global adaptation for global retrieval  | 87.3 | 94.6 | 96.6 |  | 87.4 | 95.9 | 96.9 |
| coarse patch tokens for re-ranking  | 89.8 | 95.4 | 96.9 |  | 82.8 | 92.0 | 94.9 |
| dense local features for re-ranking  | 91.4 | 96.5 | 97.8 |  | 90.8 | 96.4 | 97.2 |

Reviewer comment (Novelty):
The paper is built solely around overcoming the results of existing methods, while insights and evidence-based contributions are not provided. After reading the paper there is still a big question: "why this method works and what do I learn that can perhaps be used to design methods in different applications?" It is also unclear whether the adaptation would work for CNNs pre-trained on ImageNet, as they share the same object-centric bias.

Author response:
Insights: The pre-trained foundation models (e.g. DINOv2) have achieved remarkable performance on many vision tasks given their ability to produce well-generalized representations. The recent AnyLoc work [1] also uses DINOv2 as the backbone, but without any adaptation or fine-tuning. Although AnyLoc uses a larger version backbone (based on ViT-giant), its performance is obviously worse than ours. Our work shows that there is a gap between the tasks of model pre-training and VPR (as we described in the third paragraph of the Introduction section). Model adaptation is a necessary thing to fully unleash the capability of pre-trained foundation models for VPR.

Contributions: The main contribution is the hybrid global-local adaptation method to bridge the gap and fully unleash the capability of pre-trained models for VPR. Our work provides an effective and efficient way to seamlessly adapt pre-trained foundation models to produce both global and (dense) local features for two-stage VPR. Moreover, we propose a mutual nearest neighbor local feature loss to train the local adaptation module, which is combined with global feature loss for fine-tuning. Different from other two-stage methods, the local features produced by our model are directly used in cross-matching for re-ranking, without time-consuming geometric verification.

For different applications: Our method is designed to seamlessly adapt the pre-trained foundation model for the VPR task. However, we think the hybrid adaptation method (to get both global features and local features) may also be useful for other applications. 

Would it work for CNN: For CNN models pre-trained on ImageNet, adaptation is not necessary. We can fully fine-tune such a model directly. Most of the adaptation methods are designed for the foundation models (especially these transformer-based models) pre-trained on a large amount of data (larger than ImageNet).

About the performance: The powerful representation ability of the foundation model is the basis for good performance. But more importantly, our method bridges the gap between the tasks of model pre-training and VPR through seamless adaptation, i.e., the implication of doing adaptation is bridging the gap. The adapted model can produce the global features and local features more suitable for VPR to address the challenges in VPR tasks and achieve excellent performance.

Reviewer comment (Evaluation):
At ICLR, a thorough analysis would be expected regarding the reasons why the performance is much higher, the implications of doing adaptation, and the real scientific contributions behind this work — not just the fact that the method achieves better results than state-of-the-art (sota) methods.

Author response:
About the performance: The powerful representation ability of the foundation model is the basis for good performance. But more importantly, our method bridges the gap between the tasks of model pre-training and VPR through seamless adaptation, i.e., the implication of doing adaptation is bridging the gap. The adapted model can produce the global features and local features more suitable for VPR to address the challenges in VPR tasks and achieve excellent performance.

Reviewer comment (Novelty):
There are no motivations or justifications provided for why the adapters are designed in a certain way, and what the differences are with respect to existing approaches for adaptation of foundation models. The hypothesis behind this design is not stated, and no explanations (with experimental evidence) are given about their working principle.

Author response:
**[A2:]**  Although visual foundation models can provide powerful feature representations, the objects that need to be focused on may be different for various downstream tasks. Freezing the foundation model can retain the powerful representation capabilities, while inserting the lightweight trainable adapters so that the entire model can be adapted to the VPR task after fine-tuning on VPR datasets, that is, the trained adapter has learned VPR-related knowledge. Our visualization results well illustrate that the adapted model can focus on objects that are helpful for place recognition and ignore dynamic foreground interference, which is the key reason for its good performance. Meanwhile, our hybrid adaptation method combines global adaptation and local adaptation, which not only yields a robust global representation, but also enables the model to output dense local features for spatial matching re-ranking, thereby further boosting performance.

Please see A2 for Q2. The design of our global adaptation is inspired by previous adapter-based work. However, we design a simple local adaptation method to adapt the ViT-based foundation models to produce dense local features for the local cross-matching in two-stage VPR. Our hybrid adaptation method combines the global adaptation and local adaptation to produce both global features and dense local features for two-stage VPR, which is different from previous adaptation works.

Reviewer comment (Experiments):
Although data-efficiency is a key argument for using foundation models, the authors only mention it without providing substantial experimental evidence about how it benefits their approach.

Author response:
The adaptation method can keep the powerful pre-trained image representations intact. When there is only insufficient training data in the downstream task, it mitigates the risk of over-fitting. This is a benefit shared by most adaptation methods. It is also an advantage of our approach over most other VPR methods. The experimental evidence is in the Appendix.A of our paper.

Reviewer comment (Experiments):
The proposed adaptation mechanisms still require fine-tuning more than 50M parameters, which is much more than other methods train. Together with the 300M+ parameters of the foundation backbones, these models have much more capacity than whatever methods were used previously. The authors do not discuss this point or present experiments with adaptation of other (smaller) models.

Author response:
**[A3:]** The number of parameters of the adapter in our global adaptation is adjustable. Appropriately increasing the parameters of the adapter can slightly improve performance. However, in most cases, we can actually use a very lightweight adapter to obtain good performance. In the Appendix.C of our paper, we have shown the results of using only one adapter in each transformer block or reducing the bottleneck ratio to 0.25, and the performance is still good. Here, we provide the results (on Pitts30k) of using only the parallel adapter and reducing the bottleneck ratio from 0.5 (in the paper) to 0.05, as shown in the table below (SelaVPR** uses the more lightweight adapter. SelaVPR represents the original setup in paper). The SelaVPR** has only 5.19M trainable parameters (less than most methods), and the results are still excellent. For smaller backbones, fewer transformer blocks and lower token dimensions will naturally lead to fewer parameters of the adapter. Overall, the original paper focused more on recognition performance without excessively limiting the number of parameters. In fact, our method can achieve good performance using a significantly smaller number of parameters.

| Method  | Tunable Param (M)  | R@1 | R@5 | R@10 |
| :--- | :---: | :---: | :---: | :---: |
| ResNet50-GeM |  7  |  82.3 |  91.9 |  94.6 | 
| SelaVPR  |  53 |  91.4 |  96.5 |  97.8 |
| SelaVPR** |  5 |  91.2 |  96.4 |  97.4 |

Reviewer comment (Novelty):
A relevant reference is missing: Leyva-Vallina et al., Data-Efficient Large Scale Place Recognition With Graded Similarity Supervision; CVPR 2023.

Author response:
Thanks for the reminder. We’ll mention this work in our revised paper.

Reviewer comment (Experiments):
How would the adapters work with other (smaller) models?

Author response:
**[A1:]** The adapters can work with smaller models in the way described in the paper and achieve good performance. But it only makes sense if this model is pre-trained on a large amount of data and produces well-generalized representations. Otherwise, we can directly fully fine-tune the model. Here we provide the results (only fine-tuning the model on Pitts30k and testing on Pitts30k) of our adaptation method using a distilled version of DINOv2 (based on ViT-Base) as backbone, denoted as DINOv2-B. The DINOv2-L (based on ViT-Large) is the backbone used in our paper. Their performances are close, demonstrating that our adaptation method can be used with smaller models.

| Backbone  | R@1 | R@5 | R@10 |
| :---: | :---: | :---: | :---: |
| DINOv2-L   | 91.4 | 96.5 | 97.8 |
| DINOv2-B   | 90.5 | 96.1 |97.1 |

Reviewer comment (Evaluation):
What are the reasons and interpretations (with evidence) for why the proposed adapters work?

Author response:
**[A2:]**  Although visual foundation models can provide powerful feature representations, the objects that need to be focused on may be different for various downstream tasks. Freezing the foundation model can retain the powerful representation capabilities, while inserting the lightweight trainable adapters so that the entire model can be adapted to the VPR task after fine-tuning on VPR datasets, that is, the trained adapter has learned VPR-related knowledge. Our visualization results well illustrate that the adapted model can focus on objects that are helpful for place recognition and ignore dynamic foreground interference, which is the key reason for its good performance. Meanwhile, our hybrid adaptation method combines global adaptation and local adaptation, which not only yields a robust global representation, but also enables the model to output dense local features for spatial matching re-ranking, thereby further boosting performance.