Paper: Segment Anything Model is a Good Teacher for Local Feature Learning (ICLR-2024)

Reviewer comment (Experiments):
The authors should consider providing insights into the rationale behind choosing SAM as the source for extracting semantic information. Discussing the unique characteristics of SAM that make it suitable for distilling the semantic relationships within the given context would strengthen the paper's argument. For example, SEEM can also serve as a suitable baseline for addressing the task at hand.

Author response:
Thanks for your suggestion.

**SAM's attractive properties:**
As shown in the main paper Figure 1 (b), SAM has three good properties conducive to guided local feature learning.

1. Compared to traditional semantic segmentation, SAM is trained on a large amount of data and can **segment any object** without being restricted by the class of the training set. Both segmentation and local feature learning can be viewed as pixel-level representation learning, where segmentation is pixel-level classification while local features are used for matching. Thus we can improve the matching discriminativeness of local features by distilling the semantic discriminativeness of SAM pixel representations. However, traditional semantic segmentation models can only deal with the limited number of categories contained in the dataset and cannot be generalized to arbitrary scenarios.

2. **SAM yields semantic groupings at the part level**, and we can take advantage of this fine-grained semantic relationship to provide additional weakly supervised descriptor optimization supervision. In contrast, semantic segmentation tends to group all pixels of an object together, e.g., the pixels of the wheel and the window in Fig. 1 (a) are identified with the label of “car”, which is not conducive for use in guided learning of discriminative local features.

3. **SAM can detect more detailed edges**, whereas edge regions tend to be more prone to keypoints and contain more distinguishing information. We can achieve better local feature detection and description by making the network more focused on these edge regions.

**Discussion:**
Some of SAM's subsequent works have enhanced interactivity (SEEM), understanding (Semantic-SAM, LISA, SAM-CLIP), and efficiency (Fast SAM, Mobile SAM), but they do not additionally extend the three properties mentioned above that are relevant to our approach. Notice that our contribution is to improve local feature learning using a visual fundamental model without adding additional computational consumption. Our approach opens up the possibility of introducing visual fundamental models into the realm of geometric vision and inspires explorations using fundamental models other than SAM.

[1] An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021

Reviewer comment (Novelty):
Although I believe in the soundness of the good results that the authors have demonstrated, a major issue that makes me skeptical is whether the contribution and novelty are substantial enough to warrant a full paper.

Author response:
**A1:** Contribution and Novelty

**Contribution:**

Our core contribution is an exploration of how visual fundamental models can be applied to geometric vision tasks. We think this is illuminating and can be borrowed for other tasks, allowing non-semantic related tasks to benefit from the fundamental model wave as well.

**Motivation and Novelty:**

We systematically explore how the attractive properties of SAM can be utilized to guide local feature learning from three perspectives, intermediate representation, high-level semantic grouping, and low-level edge structure.

1. **Semantic relationship distillation (PSRD):**
   Both segmentation (pixel-level classification) and local feature learning (pixel-level matching) can be viewed as pixel-level representation learning, and thus **the use of semantic discriminativeness to improve matching discriminativeness** is promising. However common semantic segmentation can only handle a limited number of categories, so we resort to the visual fundamental model SAM and category-independent semantic relationship distillation to break the category and scenario constraints.

2. **Semantic grouping (WSC):**
   Instead of recognizing an entire object as a category (e.g., the pixels of the wheel and the window in Fig. 1 (a) are all identified as car), SAM can produce fine-grained part-level semantic groupings. These **semantic groupings can provide valuable weakly supervised signal supervision** (samples in same group are just weakly supervised positive samples, since they are not ground truth correspondences) for local feature learning, i.e. pixels belonging to the same semantic grouping should be closer in the description space, and on the contrary pixels of different groupings should be kept at a distance in the description space. In particular, to preserve local feature discriminability within the same semantic grouping, we add a fixed margin for positive sample optimization.

3. **Edge attention guidance (EAG):**
   Edge regions are more worthy of the network’s attention than mundane regions. On one hand, corner and edge points in the edge region are more likely to be detected as keypoints. On the other hand, the edge region contains rich information about the geometric structure thus contributing more to the discriminative nature of the local descriptor. Therefore, we use the fine-grained boundaries distilled from the SAM as explicit attention map to direct the network to pay more attention to the boundary regions during local feature detection and description.

Moreover, since the above improvements work only in the training phase, they do not introduce additional computational consumption in the inference phase.

**Discussion:**

We can understand your concerns; however, similarities in implementation details should not detract from the novelty of the methods themselves.

1. Our PSRD and [1] are just similar in implementation, but the motivation and physical meaning are different.
   i) The goal of [1] is for teacher and student predictions to maintain consistent relationships between samples (mini batch-wise) whereas our PSDR maintains consistent relative relationships between pixel-level representations (pixel-wise).
   ii) While most distillation methods focus on migrating knowledge from large teacher models to small models performing the same task, our PSRD introduces semantic discriminative properties of segmentation tasks (pixel-level classification) to pixel-level matching tasks.
   iii) Similarly, CLIP [2] only utilized the common contrast learning loss implementation, but not to the detriment of his success and far-reaching impact.

2. WSC differs from comparative learning loss in that we set fixed boundaries between positive samples prompting it to optimize to a certain distance and no longer provide gradients. This has the advantage of ensuring pixel discriminability within the same semantic grouping.

3. To the best of our knowledge, explicit steering of networks via edge information has rarely been explored before.

Reviewer comment (Novelty):
Many of the techniques used in the paper are borrowed from others' implementation. For example, the Pixel Semantic Relational Distillation (PSRD) is to compare two similarity matrix which is a widely used knowledge distillation loss [1].

Author response:
**Discussion:**

We can understand your concerns; however, similarities in implementation details should not detract from the novelty of the methods themselves.

1. Our PSRD and [1] are just similar in implementation, but the motivation and physical meaning are different.
   i) The goal of [1] is for teacher and student predictions to maintain consistent relationships between samples (mini batch-wise) whereas our PSDR maintains consistent relative relationships between pixel-level representations (pixel-wise).
   ii) While most distillation methods focus on migrating knowledge from large teacher models to small models performing the same task, our PSRD introduces semantic discriminative properties of segmentation tasks (pixel-level classification) to pixel-level matching tasks.
   iii) Similarly, CLIP [2] only utilized the common contrast learning loss implementation, but not to the detriment of his success and far-reaching impact.

Reviewer comment (Novelty):
The semantic grouping is from the original SAM implementation.

Author response:
**A1:** Contribution and Novelty

**Motivation and Novelty:**

2. **Semantic grouping (WSC):**
   Instead of recognizing an entire object as a category (e.g., the pixels of the wheel and the window in Fig. 1 (a) are all identified as car), SAM can produce fine-grained part-level semantic groupings. These **semantic groupings can provide valuable weakly supervised signal supervision** (samples in same group are just weakly supervised positive samples, since they are not ground truth correspondences) for local feature learning, i.e. pixels belonging to the same semantic grouping should be closer in the description space, and on the contrary pixels of different groupings should be kept at a distance in the description space. In particular, to preserve local feature discriminability within the same semantic grouping, we add a fixed margin for positive sample optimization.

**Discussion:**

2. WSC differs from comparative learning loss in that we set fixed boundaries between positive samples prompting it to optimize to a certain distance and no longer provide gradients. This has the advantage of ensuring pixel discriminability within the same semantic grouping.

Reviewer comment (Writing):
The presentation of the paper lacks clarity in conveying the motivation behind techniques used, making the paper seem more like an engineering solution for a specific task rather than a systematic method.

Author response:
**A1:** Contribution and Novelty

**Contribution:**

Our core contribution is an exploration of how visual fundamental models can be applied to geometric vision tasks. We think this is illuminating and can be borrowed for other tasks, allowing non-semantic related tasks to benefit from the fundamental model wave as well.

**Motivation and Novelty:**

We systematically explore how the attractive properties of SAM can be utilized to guide local feature learning from three perspectives, intermediate representation, high-level semantic grouping, and low-level edge structure.

1. **Semantic relationship distillation (PSRD):**
   Both segmentation (pixel-level classification) and local feature learning (pixel-level matching) can be viewed as pixel-level representation learning, and thus **the use of semantic discriminativeness to improve matching discriminativeness** is promising. However common semantic segmentation can only handle a limited number of categories, so we resort to the visual fundamental model SAM and category-independent semantic relationship distillation to break the category and scenario constraints.

2. **Semantic grouping (WSC):**
   Instead of recognizing an entire object as a category (e.g., the pixels of the wheel and the window in Fig. 1 (a) are all identified as car), SAM can produce fine-grained part-level semantic groupings. These **semantic groupings can provide valuable weakly supervised signal supervision** (samples in same group are just weakly supervised positive samples, since they are not ground truth correspondences) for local feature learning, i.e. pixels belonging to the same semantic grouping should be closer in the description space, and on the contrary pixels of different groupings should be kept at a distance in the description space. In particular, to preserve local feature discriminability within the same semantic grouping, we add a fixed margin for positive sample optimization.

3. **Edge attention guidance (EAG):**
   Edge regions are more worthy of the network’s attention than mundane regions. On one hand, corner and edge points in the edge region are more likely to be detected as keypoints. On the other hand, the edge region contains rich information about the geometric structure thus contributing more to the discriminative nature of the local descriptor. Therefore, we use the fine-grained boundaries distilled from the SAM as explicit attention map to direct the network to pay more attention to the boundary regions during local feature detection and description.

Moreover, since the above improvements work only in the training phase, they do not introduce additional computational consumption in the inference phase.

**Q1:** Intuition of each technique

**A2:**

1. The motivation for each technique is described in answer 1 and in the "Introduction" section.

2. For the examples mentioned:
   Unlike semantic segmentation, SAM does not project pixels to a specified semantic category, so the features extracted by the SAM encoder contain only category-independent semantic discriminative properties. This results in pixel features of the same class of objects (e.g., people) extracted from different images are not aligned. Direct distillation does not apply in this case (as reported in appendix section 1.4), so we resort to distilling the semantics contained in the encoder by exploiting the relative relationship between pixels (i.e., pixel representations of the same object are closer together).

Reviewer comment (Novelty):
My first concern is about the novelty of the paper. It is commendable to leverage SAM to enhance model performance in corresponding tasks. However, acquiring structured information through SAM (PSRD), and using semantic grouping to construct positive and negative samples, thereby introducing contrastive learning, have already been briefly discussed in previous works (SFD2, TPR). From this, the paper is more like an integration of some schemes combined with the SAM model. Hence, its technical novelty is somewhat weak from my perspective.

Author response:
Thank you for the reminder. We have further emphasized the differences from existing approaches in related work.

**Differences from existing methods:**

**1) Motivation.** Although some schemes (SFD2, SeLF) have used semantic segmentation information to improve the performance of local features on visual localization tasks, they perform poorly on tasks other than visual localization, as shown in the main paper Table 1 and the "Limitations" section reported in the SFD2. Semantic segmentation can only segment certain **specific categories** (e.g., visual localization-related street scenes), preventing such approaches from generalizing to other scenarios. In contrast, our approach aims to make semantic information accessible to any scenario by subtly leveraging SAM **category-independent** semantic information (representation relations, semantic grouping at the part level, boundaries).

**2) Implementation.** Our approach differs significantly from existing methods in its concrete implementation. **i) semantic distillation:** Unlike semantic segmentation, SAM does not project pixels to a specified semantic category, so the features extracted by the SAM encoder contain only category-independent semantic discriminative properties. This results in pixel features of the same class of objects (e.g., people) extracted from different images are not aligned. Direct distillation (like SFD2, SeLF) does not apply in this case (as reported in appendix section 1.4), so we resort to distilling the semantics contained in the encoder by exploiting the relative relationship between pixels (i.e., pixel representations of the same object are closer together). **ii) semantic grouping:** SFD2 constructs the metric learning loss according to each specific category (e.g., cars, people). It has two problems, on the one hand, it can only work on a limited number of categories, and on the other hand, category-level semantic grouping is not reliable for local feature learning. For example, the pixels of wheels and windows in Fig. 1 (a) are recognized as cars, which is not conducive to discriminative learning of local features. In contrast, our proposed WSC is category-independent and uses part-level semantic grouping. **iii) edges:** To the best of our knowledge, existing schemes rarely explore the improvement of boundary information for local feature learning.

Reviewer comment (Experiments):
In practical applications, is it really necessary to adopt all the components described in the paper? The actual contributions of different losses to model optimization still need further verification. For example, in Table 3, how about the performance without using EAG? In addition, are WCS and EAG really irreplaceable by each other? For instance, could eliminating WCS and adjusting the weight of EAG not affect the final results? This raises concerns about the actual importance of WCS (the semantic embedding) in the current framework.

Author response:
Thanks to your suggestion, we performed a more thorough ablation study in Appendix Table 7. Specifically, we report the results of removing each component individually in addition to applying the components sequentially.  Since the three components exert influence from different perspectives (intermediate representations, high-level semantic segmentation, and low-level edge structure), working together produces optimal performance.

Reviewer comment (Experiments):
Regarding Table 4 in the Appendix, the performance seems quite sensitive to the choice of hyperparameters. Would adjusting the loss weight of EAG also further affect the choice of these hyperparameters? After all, as mentioned in the main article, the features enhanced by EAG are further used for subsequent WCS calculations.

Author response:
Thanks to your suggestion, we have added an exploration of EAG loss weights in the Appendix Table 8. There are only subtle changes of about 0.2 on the MMA@3 as we tune the EAG loss weights. The minor changes in the MMA@3 show that it is difficult to achieve the effect of imposing WCS by only adjusting the loss weights. It is not so difficult to allow the network to extract boundaries, as shown in Figure 1 of the appendix. Therefore the effect of adjusting the EAG loss weights on the final performance is slight.

Reviewer comment (Reproducibility):
Some technical implementation details are still unclear. For example, for Local Feature Detection, is the enhanced C3 computed first before proceeding to calculate C4? The structural diagram in Figure 2 and the description in the subsection on Local Feature Detection have caused confusion regarding more specific technical details. I hope the authors can polish this part.

Author response:
Thank you for the reminder, and we provided a more precise description of this process. We enhance $C_{3}$ only when the $\rm concat$ feature maps are fed into the detection head (because only the shape of $C_{3}$ is consistent with the edge map), so $C_{4}$ is not related to the enhanced $C_{3}$. In addition, we will release the code that precisely specifies the implementation details to facilitate further understanding by the reader.

Reviewer comment (Experiments):
As mentioned in the introduction, how to obtain a large number of effective annotated samples for local descriptor learning is still very challenging. Could it be possible to demonstrate through experiments that the proposed method can effectively reduce the requirement of the training samples?

Author response:
Thanks for your suggestion. **(1)** We add a comparison of training data sizes with advanced methods in the Appendix Table 5. The results show that our method uses less data and is uncorrelated with the test benchmarks, achieving the best performance due to the introduction of the additional supervision provided by SAM. **(2)** Due to time constraints, we will leave more exploration of SAMFeat's role in mitigating data dependencies for future work. Since it is cheap for SAM to provide supervision, we will collect a large amount of web data and mix it with the original dataset to train SAMFeat V2. We believe that the SAMFeat V2 enhancement should be more in terms of generalization; however, the current benchmark lacks the relevant evaluation. We will collect more image pairs of different types and build a large open-world image-matching benchmark to evaluate current methods.

Reviewer comment (Experiments):
It is not clear how much overhead for the training after adding the extra loss functions. For example, if I understand correctly, the relationship distillation seems to be very computational heavy in that the affinity matrix contains quadratic number of entries to the image size. Likewise, the second loss term need to compute the with-group and cross-group similarities for all image regions. Overall, I am afraid that the proposed method is very time consuming during training.

Author response:
1. Even though the computation cost for the relationship matrix distillation is quadratic, we downsampled the feature map to 1/8 of the original size. This also quadratically reduced the computation cost.

2. In terms of quantitative analysis, our method only requires training for 6 hours using two Nvidia RTX 3090 GPUs. Compared to other work like ASLFeat (42 hours on a single NVIDIA RTX 2080Ti) and TRR (30 hours for training with two NVIDIA-A100 GPUs), this demonstrates a totally reproducible cost for individual researchers. As for the overhead for the training after adding the extra loss functions, we provide a detailed table in Appendix Table 6. Each loss function will inevitably cause extra training costs; however, the tradeoff between a minimal incremental in time and the improvement in accuracy is reasonable, and the final training cost is acceptable. This further demonstrates the lightweight nature of our approach in the field of feature learning and description, making it easily implementable and resource-efficient.

Reviewer comment (Experiments):
The experiments in the paper is not satisfactory. There is not much detailed and deep analysis on the proposed method as I mentioned earlier. Given the two tables Table 1 and 2, it is hard to tell whether the proposed method is fairly compared with previous works. For example, the authors mentioned that they used MTLDesc as the baseline method. In Table 1, the reported MMA@3 is 78.7 for it. However, when it goes to Table 3, the first row shows 75.7 MMA@3. This makes me a bit confused and doubted whether the authors are conducting solid and fair comparisons across the board. More importantly, it is also not clear about the main difference between the proposed method and others, regarding the learning techniques, vision encoder, training regimes, etc.

Author response:
Our experiments are conducted fairly.

1. For the confusion about Table 1 and Table 3, we sincerely apologize for not elaborating clearly and causing confusion to you, and we have corrected this in the main paper. In short, our baseline, the first line of the main paper table 3, is not the exact MTLDesc. To enhance clarity and simplicity, our baseline adopts SuperPoint's VGG-styled backbone with MTLDesc's attention-weighted descriptor loss. MTLDesc used complex modules such as Transformer to achieve high matching accuracy. Thus, with no doubt, our baseline (75.7 MMA@3 in main paper table 3) is lower in performance compared to MTLDesc (78.7 MMA@3 in main paper table 1) Compared to the MTLDesc which has 10.1 Million trainable parameters, SAMFeat is much more lightweight and only has 2.1 Million trainable parameters.

2. Regarding the fairness of our experiment, we ensure fairness in our evaluations; each method undergoes the same assessment criteria in HPatches and Aachen V1.1. Our Evaluation code is attached in the supplementary materials and is open-sourced. We ensure full transparency and fairness in evaluation and comparison. While each method may have unique training settings, the parameters of the visual encoders and the sizes of the training datasets are comparable. From the perspective of the visual encoder, compared to the MTLDesc which has 10.1 Million trainable parameters, SAMFeat is much more lightweight and only has 2.1 Million trainable parameters. Regarding the training dataset aspect, please refer to Appendix Table 5 for a comprehensive report on our dataset specifics. Our SAMFeat used fewer training samples and is more lightweight, but achieves the best performance. It's noteworthy that our training methods align with SuperPoint and MTLDesc, providing consistency and comparability in the training process.

Reviewer comment (Experiments):
Following the last point, the incorporation of SAM for local feature learning is interesting and valuable. However, it is not clear how the settings for the distillation affect the final performance. For example, the density of grid in SAM for automatical segmentation, the image resolutions, the number of sampling points for the proposed training losses, etc, all of them are not studied, which make the contribution of the work and effectiveness of the proposed method hard to assess.

Author response:
Thank you for your suggestion. We acknowledge the importance of studying the impact of various settings on distillation. In our experiments, we adhered to the default settings of SAM, including the grid density. The training settings, such as image resolution and the number of sampling points, were kept consistent with MTLDesc to ensure a fair and direct comparison since the training data were identical. Notably, using these uniform and default settings resulted in a significant performance boost. Thus, the results obtained without meticulously fine-tuning each parameter underscore the effectiveness of our approach. In the appendix, we provide a detailed ablation study (Appendix Table 2 - Table 8) regarding the effect of different distillation loss settings and the effect of each distillation and learning module on the final outcome. We hope this could address your concerns. As you rightly pointed out, due to time constraints, we couldn't explore every parameter in-depth, and in many cases, we followed established practices from previous works.

Reviewer comment (Experiments):
Like I mentioned above, I do want to see a deeper analysis on the proposed techniques to distill SAM knowledge for local feature learning tasks.

Author response:
Thank you for your suggestion. We acknowledge the importance of studying the impact of various settings on distillation. In our experiments, we adhered to the default settings of SAM, including the grid density. The training settings, such as image resolution and the number of sampling points, were kept consistent with MTLDesc to ensure a fair and direct comparison since the training data were identical. Notably, using these uniform and default settings resulted in a significant performance boost. Thus, the results obtained without meticulously fine-tuning each parameter underscore the effectiveness of our approach. In the appendix, we provide a detailed ablation study (Appendix Table 2 - Table 8) regarding the effect of different distillation loss settings and the effect of each distillation and learning module on the final outcome. We hope this could address your concerns. As you rightly pointed out, due to time constraints, we couldn't explore every parameter in-depth, and in many cases, we followed established practices from previous works.