Paper: GIM: Learning Generalizable Image Matcher From Internet Videos (ICLR-2024)

Reviewer comment (Experiments):
The heavy reliance on internet videos for training might introduce biases or noise. The generalization capability of GIM, when trained on other diverse datasets, remains an unanswered question.

Author response:
**Weakness 1**: Thanks for your insightful comment. Though naively adding random internet videos can potentially introduce noise or bias, we have demonstrated in the experiments that by selecting tourism videos captured at diverse geo-locations and time, GIM can effectively generalize to various zero-shot inputs, e.g., images from different domains and BEV point clouds. As we discuss in the response to all reviewers, we have trained another model with 100h of data showing that GIM scales well with more video data.

In terms of other diverse datasets, we focus on internet videos since they are arguably the most accessible and scalable data source. This ensures that the performance improvement of GIM can be reproduced by almost anyone on any future architecture. However, the key to the generalization of GIM is the data diversity and scale, rather than the source of the videos. Hence, we expect that as long as the dataset is **diverse** and **large**, it will work with GIM. So for example, adding proprietary videos captured in various environments, will likely further boost performance.

We will clarify the importance of our video selection strategy and the key to the generalization of GIM in the camera-ready paper.

Reviewer comment (Experiments):
Internet videos can be noisy, and their quality can vary. How resistant is GIM to such noise, and how does it handle low-quality data?

Author response:
**Weakness 3**: For the selection of internet videos, our recommendation is to choose videos with high resolution, clear images, long duration and small number of scene transitions. To ensure these points, we choose tourism videos. They are generally 0.5-2 hours long, with a person holding the camera and recording his/her travel, mostly without transitions. GIM also has some robustness to transitions in the video since drastic changes from one frame to another make it very difficult for matches to survive after robust fitting. Even if a few matches survive, it is very difficult for the label propagation to continue. Finally, GIM includes strong data augmentations during training, which introduces noise to make the model more resistant to various sources of error in the input data.

We will include the above answer in camera-ready paper.

Reviewer comment (Experiments):
While the use of internet videos is innovative, how did you ensure that the videos used for training represent a diverse range of scenarios, especially given the potential for internet content to have biases?

Author response:
**Question 1**: As mentioned in the answer of weakness 1, we did not randomly download videos from the internet. Instead, as mentioned in Sec. 3 of the main paper, we selectively used tourism videos for training. The selected tourism videos are usually captured by a person traveling around various places. Even only on YouTube, we can find tourism videos spanning thousands of hours and covering various regions, years, times, weather conditions, and lighting. These videos are not limited to a single type of scene; they include diverse activities like walking, cycling, driving and more. This ensures the data diversity and minimizes the bias. 

Based on your suggestion, we will describe our video selection process and its effectiveness in details in our camera-ready paper. The code will also be released upon acceptance.

Reviewer comment (Evaluation):
How did you address concerns of overfitting, especially with the consistent improvement seen with increasing video data?

Author response:
**Question 3**: Please refer to the response of weakness 2.

Reviewer comment (Experiments):
Lacking real indoor datasets in the benchmark: This is a nitpick but it would be great to have more real indoor datasets in the benchmark. Right now, most of the real datasets are driving-related and the indoor dataset only covers basements and corridors.

Author response:
**Weakness 1.**: This is a very good suggestion. To facilitate better zero-shot evaluation, we will include more diverse real-world indoor scenes (e.g., SUN3D, ScanNet V2) during the code release of GIM and ZEB.

Reviewer comment (Experiments):
Will the performance continue to improve if the self-training is repeated multiple times or do the authors expect the models to start degrading due to noisy pseudo-labels?

Author response:
**Question 1.**: Thank you for this insightful question. Since the ``iterative self-training process'' could be interpreted in different ways, we provide answers based on two possible interpretations. We will also clarify this in the camera-ready paper.

**(Case 1) Using a fixed set of videos in multiple rounds of iterative self-training**: In this case, each round of self-training uses the current best GIM model (trained in the previous round) to re-generate the labels on the same set of videos, and then updates the GIM model with the re-generated labels. The expected improvement would come from better video labels and longer training time on the same set of videos. In terms of the video labels, Table 2 of the main paper shows that a better label generation method leads to better GIM performance. Hence, we would expect that this bootstrapping could in principle provide a further performance boost. However, the improvement of a single method may not be very significant in practice, due to the use of multiple complementary methods for label generation. In terms of the longer training time, image matching models are prone to overfitting, hence early stopping is applied during training. GIM does not change the baseline architecture or training curriculum and might also exhibit the overfitting problem for long training times, if the video dataset is too small and not expanded over time. Hence, if ``iterative self-training'' refers to case 1, it may slightly improve results but will probably not be as effective as expanding the video dataset at the same time. We discuss this case below and expect it to be the best way to boost the performance of GIM with iterative self-training.

**(Case 2) Expanding the number of videos in multiple rounds of iterative self-training**: In this case, each round of self-training uses the current best GIM model (trained in the previous round) to generate labels on a **new** set of videos, and then performs the next round of self-training with both the old and new videos. Table 2 of the main paper shows that the most effective way to improve GIM is to add more video data, rather than generating better labels on a fixed set of videos. Hence, given the limited time and training resources (GPUs) and the potentially infinite amount of available internet videos, using the current best GIM model to generate labels on a new set of videos can best improve the performance. In this case, albeit less likely, overfitting may still happen when training the model for a long time. Hence, we recommend keeping early stopping active and re-initializing GIM weights for multiple rounds of self-training.

Reviewer comment (Experiments):
Current work focuses on tourism videos. Would the approach work for other types of videos such as egocentric videos?

Author response:
**Question 2.**: We expect that GIM will also work with egocentric videos since there is nothing specific to the camera perspective in the algorithm. The more important aspect is that a similar level of diversity as in the case of the tourism videos is ensured (e.g., different geo-locations, lightning conditions, etc.). Training GIM on a mixture of diverse tourism videos and ego-centric videos may also be a good choice and boost generalization performance further. It is an interesting direction for further investigation.

Reviewer comment (Reproducibility):
This paper is very impressive, I think the only thing left is just some implementation details because the self-training part is very short and only about the ground-truth instead of the training itself.

Author response:
Thank you for the encouraging review, your appreciation of our simple yet effective method, and your valuable suggestions. 

**Weaknesses (Implementation details and code release)**: The training part mostly follows the baseline code as mentioned in the ``implementation details'' part of the paper. 

To ensure reproducibility, we will provide more details such as training time on different GPUs and for different base architectures, as discussed in the response to Reviewer-sN7h. We will also release the pre-trained GIM models, the training and ZEB evaluation data, and the code for label generation and model training.

Reviewer comment (Reproducibility):
The only thing left is just open-sourcing the proposed code of the label propagation and training data to verify its accuracy.

Author response:
Thank you for the encouraging review, your appreciation of our simple yet effective method, and your valuable suggestions. 

**Weaknesses (Implementation details and code release)**: The training part mostly follows the baseline code as mentioned in the ``implementation details'' part of the paper. 

To ensure reproducibility, we will provide more details such as training time on different GPUs and for different base architectures, as discussed in the response to Reviewer-sN7h. We will also release the pre-trained GIM models, the training and ZEB evaluation data, and the code for label generation and model training.

Reviewer comment (Reproducibility):
Please share more about the training details. By domain specific training is it just swapping the GT or there are more details that's not being covered in the paper?

Author response:
**Questions (Meaning of domain specific training)**: In the paper, we refer to training on standard image matching datasets (such as e.g., MegaDepth, ScanNet and unlike in-the-wild data from YouTube) as *domain-specific training*. In the experiments, we use the officially released models, i.e., SuperGlue (OUT), LoFTR (OUT) and DKM (OUT), as *domain-specific* models for GIM label generation. We will clarify this point in the camera-ready paper to avoid misunderstandings.

Reviewer comment (Experiments):
The introduction says "multiple" complementary image matching methods are used. While in the experiments section, it seems that only one complementary image matching method is used for each baseline method (SuperGlue, LoFTR & DKM). Will the number of complementary methods affect the performance?

Author response:
**Weakness (1)**: You are right, the complementary methods are introduced to increase the number of correspondences. Specifically multi-method matching is conducted only on nearby video frames ($<$80 frame interval), in which setting complementary methods are still able to provide reliable correspondences, albeit with inferior performance. The reason that complementary methods can increase the number of correspondences is that different methods often generate correspondences with different distributions. For example, RootSIFT generates correspondences on key point pixels. DKM outputs are mostly located on non-key point pixels. This strategy enhances the correspondence density between nearby video frames, which allows label propagation to produce training images with a larger pose difference.

We do not just use 1 complementary method for each baseline, but rather all available methods that are inferior to the baseline, i.e., for training GIM$\_{SuperGlue}$, we use RootSIFT+SuperGlue to generate video labels. For training GIM$\_{LoFTR}$, we use RootSIFT+SuperGlue+LoFTR and for training GIM$\_{DKM}$, we use RootSIFT+SuperGlue+LoFTR+DKM to generate video labels. More complementary methods should in principle benefit the performance of GIM. For fair comparison, we use only inferior complementary methods so that we do not use labels generated by better methods to improve the baseline.

We will clarify both points in camera-ready manuscript to avoid misunderstandings, thanks for your question.

Reviewer comment (Reproducibility):
Does batch size affect the self-supervision performance? From the implementation details, the GIM label generation on 50 hours of YouTube videos takes 4 days on 16 A100 GPUs. How long will it take to re-train a baseline network using the generated labels (and on which type and how many GPUs)? Will the proposed method still work on an RTX 3090, which is commonly used in universities?

Author response:
**Weakness (2)**: Yes, batch size affects the self-supervision performance; generally enlarging the batch size improves the performance. We follow the official implementation of the baselines to set the batch size.

In terms of the training time of GIM, if we use 8 A100 GPUs, it takes about 4, 5 and 7.5 days to train GIM$\_{SuperGlue}$, GIM$\_{LoFTR}$, and GIM$\_{DKM}$ respectively.

Yes, GIM works on GPUs with smaller GPU memory such as the RTX-3090, which is the GPU we used for initial experiments. For label generation, it takes 15 days to process 50 hours of videos on 4 RTX-3090 GPUs. For training, it takes 9.5, 12, and 19 days to train GIM$\_{SuperGlue}$, GIM$\_{LoFTR}$, and GIM$\_{DKM}$ respectively. We apply gradient accumulation when using the RTX-3090 GPU to ensure the same training batch size as on the A100 GPU.

We will clarify these points in the appendix of the camera-ready manuscript to ensure the reproducibility of GIM across various setups.