Paper: Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model (ICLR-2024)

Reviewer comment (Novelty):
The Mixture of Experts in introduced in the paper seems to be multi-scale strategies that are commonly used in image segmentation. It is unclear whether the added complexity of this method is warranted. A more straightforward explanation would likely render the technique more impactful and accecible.

Author response:
**Q2: MoE vs. common multi-scale strategies.**

A straightforward multi-scale strategy involves using multiple convolution branches and subsequently aggregating their features. In comparison, MoE proves to be more efficient due to its utilization of conditional computation, selectively activating only one branch during each forward pass. This conclusion is supported by the ablation study presented in **Table 3**, where MoE demonstrates faster training speed and reduced memory consumption. We also clarify the motivation of introducing MoE in **Section 3.1**.

An alternative multi-scale strategy entails integrating additional convolution networks alongside SAM to extract multi-scale features, akin to the ViT-adapter design [4]. However, this approach is no longer parameter-efficient, as it requires full fine-tuning the pre-trained backbone, which introduces significantly more parameters than LoRA, leading to increased training complexity. In contrast to these explicit multi-scale strategies, **Conv-LoRA operates more implicitly within ViT, which inherently processes features predominantly at a single scale.** This implicit integration distinguishes Conv-LoRA as a parameter-efficient alternative in addressing multi-scale considerations.

Reviewer comment (Theory):
Purely empirical work. No reasoning is given as to how the model improves performance.

Author response:
**Q1: The reasons of performance improvements.**

Our enhancements are rooted in two pivotal aspects. First, SAM relies on the plain ViT architecture, which is known for its absence of image-related local priors. **Conv-LoRA tackles this limitation by introducing convolution operations that incorporate hard-coded local priors.** The integration of local priors proves beneficial for segmentation tasks demanding pixel-level predictions.

Second, SAM's segmentation pretraining causes the ViT to emphasize low-level features while neglecting significant high-level semantic information essential for semantic segmentation. In contrast, **Conv-LoRA, through finetuning, facilitates the ViT in reclaiming the capacity to capture nuanced high-level semantic information.**

Reviewer comment (Theory):
MoE looks a bit opaque. What priors is it learning, and how do we grasp what it is doing?

Author response:
**Q4: What priors is MoE learning and how we grasp what it is doing?**

**MoE learns to dynamically select an appropriate scale for injecting local priors based on input features.** To illuminate its functionality, we conducted an analysis by tracking the frequency of each expert's selection across different datasets during inference. The detailed results are presented in **Figure 14 in Appendix H**. Notably, **distinct datasets exhibit preferences for different experts.** For instance, in Leaf Segmentation, MoE tends to favor experts with upsampling ratios of 3 and 4, while on the ISIC 2017 dataset, it tends to select the expert with an upsampling ratio of 2. Connecting these insights with the optimal scale ablation results in **Table 4**, it become evident that MoE selects the expert adaptively for each sample, and consequently, the distributions of MoE selection across datasets reflect their different data distributions. This observation supports that MoE behaves adaptively and effectively with respect to each sample’s property. **This adaptability reinforces the significance of MoE in tailoring its selection to the unique characteristics of diverse datasets,** **enhancing its effectiveness in local prior injection.**

Reviewer comment (Experiments):
Can the authors give more reasoning for freezing the prompt encoder? I quite liked the prompt encoder in the original model.

Author response:
**Q5: Reason of freezing the prompt encoder.** 

While the prompt encoder in SAM provides opportunities for seamless integration with other modules, such as object detectors, within larger systems, **certain applications may necessitate a standalone end-to-end model for deployment**. In this work, our focus is specifically on these application scenarios, **aiming for a model that operates independently without reliance on external prompts**. For simplicity, we have opted to remove the prompt encoder during both training and inference stages. The exploration of joint finetuning with the prompt encoder is left for future studies.

Reviewer comment (Theory):
The motivation for introducing the combination of MOE with convolutional parameters as a couple is not clear to me. It is not clear how each of them will benefit the performance as an add-on to the vanilla LoRA.

Author response:
**Q1: Motivation of MoE + convolution as a couple.**

Convolution is to inject the local prior into the plain ViT, and MoE is to help to do so efficiently. **Table 4** in our paper reveals that different datasets exhibit varying optimal scales. While a multi-scale strategy is a straightforward approach, our results indicate that employing MoE proves to be a more efficient solution. As demonstrated in **Table 3**, the utilization of MoE demonstrates notable advantages in terms of training speed (0.71 vs. 0.46 iterations per second) and lower training memory consumption (21.7GB vs. 23.4GB) compared to the multi-scale strategy. We also clarify the motivation of introducing MoE in **Section 3.1**.

Reviewer comment (Reproducibility):
It will be helpful to clarify how different the training procedure with the add-on module will be in comparison to the original SAM training process.

Author response:
**Q3: Compare the training procedure with original SAM.**

SAM’s image encoder was initialized from MAE pretrained ViT, and SAM was pretrained on billion masks and 11 million images. The training is a promptable segmentation task with a set of geometric prompts, including mask, point, and box. Given one image and a prompt, SAM predicts three nested masks (for the whole, part, and subpart), computes a linear combination of focal loss and dice loss for each mask, and backpropagates solely the minimal loss among the three predictions.

We aim to do parameter efficient finetuning (PEFT) of pretrained SAM on downstream tasks. During training, all the parameters in SAM’s image encoder are frozen except for the small amount of Conv-LoRA parameters. Additionally, we exclude the prompt encoder to transform SAM into an end-to-end model, eliminating the reliance on external prompts. In this finetuning process, the model directly predicts semantic segmentation masks from a given image, supervised by the ground truth masks specific to the downstream tasks.

Reviewer comment (Evaluation):
It will also be helpful to have a comparison in computational cost for those parameter-efficient tuning methods.

Author response:
**Q4: Computational cost comparison.**

The comparison below evaluates the training and inference speed, along with the per-epoch training time, utilizing the ISIC2017 dataset on a single V100 GPU (**Table 15 of Appendix F**). Despite Conv-LoRA exhibiting a slower processing speed compared to other methods, **it demonstrates consistent performance improvements across diverse semantic segmentation tasks**, as illustrated in **Table 1** of our paper. Our analysis indicates that the primary computational cost arises from the upscale and downscale operations. A potential future direction is investigating methods to inject local priors without the explicit scaling up and down of features. Additionally, we notice the presence of orthogonal works, such as token merging [2] [3], which may offer ways for accelerating Conv-LoRA.

*Training*:

| Method    | iter/s | min/epoch |
| --------- | ------ | :-------: |
| BitFit    | 1.82   |   18.3    |
| Adapter   | 1.73   |   19.3    |
| VPT       | 1.41   |   23.6    |
| LoRA      | 1.64   |   20.3    |
| Conv-LoRA | 1.22   |   27.3    |

*Inference*:

| Method    | iter/s |
| --------- | ------ |
| BitFit    | 4.62   |
| Adapter   | 4.38   |
| VPT       | 4.39   |
| LoRA      | 4.62   |
| Conv-LoRA | 3.64   |

Reviewer comment (Experiments):
As shown in Table 4, the scales vary amongst different datasets. Will this require extra tuning efforts for picking a suitable scale (experts)? Again, will the way how different sizes are combined in the inception structure be a better option?

Author response:
**Q5: Extra tuning efforts for picking suitable experts? Again, will the way how different sizes are combined in the inception structure be a better option?**

We employ MoE to **automate the selection of appropriate scales**, obviating the need for additional tuning efforts. In our experiments, we utilize 8 experts that cover a range of 8 scales, a configuration we have found to be effective for many datasets. As previously discussed, **our rationale for favoring MoE over the inception structure stems from its ability to handle scales larger than the default**, a capability that the inception structure lacks.

Reviewer comment (Theory):
The paper convincingly demonstrates the effectiveness of the PERF method it introduces, but its motivation and explanation for why it works is unintuitive and the supporting evidence is insufficient. The authors claim that adding convolutional layers reintroduces inductive biases that are helpful in the image domain and hard-coded into convolutional layers. However, in Conv-LoRA, the convolutional layers are not applied to images but on features that do not necessarily adhere to the locality prior by construction. There is a question of whether the locality prior can truly be reintroduced if it might have already been lost, or whether this should rather be regarded as a data-efficient method for finetuning that utilizes the learned locality of the early features in the ViT. Unless this understanding is wrong, the explanation for the good performance of Conv-LoRA should be reformulated as a hypothesis.

Author response:
**Q1(1): What do the authors mean by "SAM's local prior assumption" mentioned in the abstract?**

SAM’s local prior is grounded in its extensive segmentation pretraining. Through supervised training on a vast dataset encompassing 1 billion high-quality masks and 11 million images, **SAM has honed a robust capability to discern and capture local features within images**. Notably, SAM's encoder retains the ViT architecture, which inherently lacks a dedicated local prior. **However, this deficiency is effectively compensated for by the significant local prior acquired through segmentation pretraining**.

To substantiate SAM's local prior, we conduct an analysis using the mean attention distance as a metric [1] [2]. In **Appendix G (Figure 12)**, our findings reveal that SAM exhibits many heads in the deep blocks with short mean attention distances. This observation indicates SAM's heightened focus on local information during the later stages of the encoder. In contrast, the MAE pretrained ViT[3], representing SAM's initialization, displays consistently long mean attention distances among attention heads in the later stages. Consequently, SAM's segmentation pretraining induces a transformative shift in ViT's attention heads, steering them from a global-oriented to a local-oriented configuration. **This transformation underscores the efficacy of SAM's approach in imbuing the model with a distinctive local prior, enhancing its ability to capture fine-grained details within images.**

-------------------------------------------------------------------------------------------------------------------------------------------------------------

**Q1(2): How can "image-related inductive biases" be reinjected on top of features that do not have those biases?**

**It is crucial to clarify that the features do not entirely lose their "image-related inductive biases"**; rather, they retain certain biases even after processing through the model. Extensive pre-training on large-scale datasets has been demonstrated to equip ViTs with these inductive biases [1] [2]. SAM's large-scale segmentation pretraining further amplifies these inductive biases within the features.

To evaluate the inductive biases of the features, we conduct comprehensive comparisons among randomly initialized ViT, MAE pretrained ViT, and SAM ViT. Two key metrics, mean attention distance, and relative log amplitudes of Fourier transformed features[4], are presented in **Figure 13 in Appendix G**. Notably, as the training progresses from randomly initialized ViT to MAE ViT and ultimately to SAM ViT, we observe a consistent trend: the mean attention distance decreases, while the high-frequency signals in the features increase, indicating a focus on local information.

**It is important to underscore that Conv-LoRA is designed not to reintroduce inductive biases but rather to reinforce existing implicit biases within the features**. Throughout the paper, we deliberately use the term "reinforce" to accurately convey the model's objective, avoiding any implication of reintroduction. This distinction reflects our nuanced approach in ensuring that the model builds upon and strengthens the inherent image-related inductive biases within the features.

-------------------------------------------------------------------------------------------------------------------------------------------------------------

**Q1(3): "data-efficient method for finetuning"**

Regarding your speculation about our method being a data-efficient approach, we consider this to be a complementary explanation for the observed performance improvement. Because convolutional operations naturally contain inductive biases.

We also conduct experiments (**Table 16 in Appendix I**) in the few-shot setting, wherein acquiring data for downstream tasks is challenging, and only a limited number of training samples are available. Our experiments are performed on the Trans10K-v2 dataset, which is used for twelve-class transparent object segmentation. We randomly select the training samples, to meet the settings of 1, 2, 4, 8, and 16 shots (i.e., the number of labeled training examples per class), and the experiments are run for 100 epochs:

| Method    |    Shot    |               mIoU                |
| --------- | :--------: | :-------------------------------: |
| LoRA      | 1/2/4/8/16 |   13.32/18.01/22.70/25.89/38.11   |
| Conv-LoRA | 1/2/4/8/16 | **14.53/19.81/23.05/26.02/38.63** |

The performance improvement of Conv-LoRA compared to LoRA, is particularly pronounced in an extremely low-data setting (e.g., 1-shot). The result demonstrates that **the introduction of inductive bias in Conv-LoRA contributes to improved data efficiency.**

Reviewer comment (Evaluation):
The authors identify "SAM's foreground-background segmentation pretraining" as a weakness, but SAM actually outputs three prompt-dependent masks designed to allow the network to identify an object hierarchy (whole, part, sub-part) for ambiguous prompts. This seems closely related to multi-class segmentation and requires understanding of the image semantics. The paper could be strengthened by elaborating on and providing evidence for this alleged deficiency of SAM.

Author response:
**Q2: Why SAM's foreground-background pre-training is insufficient?**

SAM's foreground-background segmentation pretraining is instrumental in discerning object hierarchies. However, **it is important to recognize that the segmentation within an object, though beneficial for hierarchy identification, differs significantly from the demands of multi-class segmentation**. The latter necessitates a holistic understanding of global semantics and the ability to distinguish objects belonging to distinct classes.

SAM was trained on a dataset that exclusively comprises segmentation masks without explicit semantic information. In theory, to minimize loss, the fundamental objective for its encoder is to project pixels into a metric space where pixels from the same object are in close proximity, while those from distinct objects are distantly positioned. This projection requires an implicit understanding of "objectness", focusing on proximity within an image rather than preserving consistent representations of the same-type object across different images. **This introduces a potential challenge in aligning representations with semantics across diverse images.**

To substantiate this assumption, we conducted a linear probing experiment on the ImageNet-1K dataset, utilizing ViT base models from SAM and MAE. The results presented in **Section 4.2** of our paper reveal a notable discrepancy, with MAE exhibiting significantly higher accuracy compared to SAM (67.7% vs. 54.2%). Further evidence is presented in **Figure 12 of Appendix G**, illustrating that many attention heads near the tail of the ViT exhibit relatively small attention distances after SAM pretraining.

*linear probing experiment*:

| Model |   Acc.   |
| :---: | :------: |
|  MAE  | **67.7** |
|  SAM  |   54.2   |

**This tendency of SAM's pretraining to emphasize information retention for low-level segmentation comes at the expense of losing crucial information essential for high-level classification,** a key requirement for multi-class semantic segmentation. These findings collectively underscore the possible limitations of SAM's foreground-background pretraining in adequately addressing the complexities of multi-class semantic segmentation tasks.

Reviewer comment (Novelty):
While the paper includes an overview of other efforts for parameter-efficient finetuning (PERF), it should clarify what sets it apart from other work on PERF of SAM. Specifically, the authors mention the work of Zhang & Liu 2023 (SAMed), which also uses LoRA to adapt SAM to a new domain (medical images) and, from this reviewer's understanding, also repurposes SAM to work for multi-class segmentation in an end-to-end fashion. Given that Zhang & Liu 2023 is a recent work not published in a peer-reviewed venue, a direct comparison is not expected. However, the authors should revisit their claim in the introduction that Zhang & Liu (2023) fail to address SAM's limitation of not capturing "high-level image semantic information" and functioning as a "standalone, end-to-end solution." Similarly, the authors cite Shaharabany et al., 2023, who also adapt SAM to work fully automatically.

Author response:
**Q3: Misrepresented works.** 

Thank you for bringing attention to this aspect. We acknowledge your observation that both Zhang&Liu and Shaharabany et al. have indeed adapted SAM to function as an end-to-end model. In response to your feedback, we have revised our claims, removing the assertion of SAM being an "end-to-end solution" from our list of major contributions.

Regarding the notion of "capturing high-level semantic information," we wish to emphasize that Zhang&Liu did not explicitly delve into a detailed analysis of this particular limitation of SAM. We want to convey that, irrespective of whether these adaptations are made with an awareness of SAM's this limitation, **our emphasis lies in conducting in-depth analyses (linear probing and attention distance statistics) and offering insights into SAM's constraints**. We believe that such analyses are essential in shedding light on potential areas for improvement in SAM in the future.

Reviewer comment (Theory):
The explanation for Conv-LoRA's improved performance over LoRA is unconvincing. While ViTs learn about locality in images, this is not a prior built into the architecture. Clarification is needed on the meaning of "SAM's local prior assumption" and how "image-related inductive biases" can be reinjected on top of features that do not have those biases.

Author response:
**Q1(1): What do the authors mean by "SAM's local prior assumption" mentioned in the abstract?**

SAM’s local prior is grounded in its extensive segmentation pretraining. Through supervised training on a vast dataset encompassing 1 billion high-quality masks and 11 million images, **SAM has honed a robust capability to discern and capture local features within images**. Notably, SAM's encoder retains the ViT architecture, which inherently lacks a dedicated local prior. **However, this deficiency is effectively compensated for by the significant local prior acquired through segmentation pretraining**.

To substantiate SAM's local prior, we conduct an analysis using the mean attention distance as a metric [1] [2]. In **Appendix G (Figure 12)**, our findings reveal that SAM exhibits many heads in the deep blocks with short mean attention distances. This observation indicates SAM's heightened focus on local information during the later stages of the encoder. In contrast, the MAE pretrained ViT[3], representing SAM's initialization, displays consistently long mean attention distances among attention heads in the later stages. Consequently, SAM's segmentation pretraining induces a transformative shift in ViT's attention heads, steering them from a global-oriented to a local-oriented configuration. **This transformation underscores the efficacy of SAM's approach in imbuing the model with a distinctive local prior, enhancing its ability to capture fine-grained details within images.**

-------------------------------------------------------------------------------------------------------------------------------------------------------------

**Q1(2): How can "image-related inductive biases" be reinjected on top of features that do not have those biases?**

**It is crucial to clarify that the features do not entirely lose their "image-related inductive biases"**; rather, they retain certain biases even after processing through the model. Extensive pre-training on large-scale datasets has been demonstrated to equip ViTs with these inductive biases [1] [2]. SAM's large-scale segmentation pretraining further amplifies these inductive biases within the features.

To evaluate the inductive biases of the features, we conduct comprehensive comparisons among randomly initialized ViT, MAE pretrained ViT, and SAM ViT. Two key metrics, mean attention distance, and relative log amplitudes of Fourier transformed features[4], are presented in **Figure 13 in Appendix G**. Notably, as the training progresses from randomly initialized ViT to MAE ViT and ultimately to SAM ViT, we observe a consistent trend: the mean attention distance decreases, while the high-frequency signals in the features increase, indicating a focus on local information.

**It is important to underscore that Conv-LoRA is designed not to reintroduce inductive biases but rather to reinforce existing implicit biases within the features**. Throughout the paper, we deliberately use the term "reinforce" to accurately convey the model's objective, avoiding any implication of reintroduction. This distinction reflects our nuanced approach in ensuring that the model builds upon and strengthens the inherent image-related inductive biases within the features.

Reviewer comment (Writing):
The reasoning for why SAM's foreground-background pretraining is insufficient is unclear. The authors should elaborate on this point.

Author response:
**Q2: Why SAM's foreground-background pre-training is insufficient?**

SAM's foreground-background segmentation pretraining is instrumental in discerning object hierarchies. However, **it is important to recognize that the segmentation within an object, though beneficial for hierarchy identification, differs significantly from the demands of multi-class segmentation**. The latter necessitates a holistic understanding of global semantics and the ability to distinguish objects belonging to distinct classes.

SAM was trained on a dataset that exclusively comprises segmentation masks without explicit semantic information. In theory, to minimize loss, the fundamental objective for its encoder is to project pixels into a metric space where pixels from the same object are in close proximity, while those from distinct objects are distantly positioned. This projection requires an implicit understanding of "objectness", focusing on proximity within an image rather than preserving consistent representations of the same-type object across different images. **This introduces a potential challenge in aligning representations with semantics across diverse images.**

To substantiate this assumption, we conducted a linear probing experiment on the ImageNet-1K dataset, utilizing ViT base models from SAM and MAE. The results presented in **Section 4.2** of our paper reveal a notable discrepancy, with MAE exhibiting significantly higher accuracy compared to SAM (67.7% vs. 54.2%). Further evidence is presented in **Figure 12 of Appendix G**, illustrating that many attention heads near the tail of the ViT exhibit relatively small attention distances after SAM pretraining.

*linear probing experiment*:

| Model |   Acc.   |
| :---: | :------: |
|  MAE  | **67.7** |
|  SAM  |   54.2   |

**This tendency of SAM's pretraining to emphasize information retention for low-level segmentation comes at the expense of losing crucial information essential for high-level classification,** a key requirement for multi-class semantic segmentation. These findings collectively underscore the possible limitations of SAM's foreground-background pretraining in adequately addressing the complexities of multi-class semantic segmentation tasks.

Reviewer comment (Novelty):
The cited work Zhang & Liu 2023 appears to also adapt SAM to work without an image-dependent prior and to extract semantics from the segmentation head. Similarly, Shaharabany et al. adapt SAM to work fully automatically. The authors should ensure that these works are not misrepresented in the introduction.

Author response:
**Q3: Misrepresented works.** 

Thank you for bringing attention to this aspect. We acknowledge your observation that both Zhang&Liu and Shaharabany et al. have indeed adapted SAM to function as an end-to-end model. In response to your feedback, we have revised our claims, removing the assertion of SAM being an "end-to-end solution" from our list of major contributions.

Regarding the notion of "capturing high-level semantic information," we wish to emphasize that Zhang&Liu did not explicitly delve into a detailed analysis of this particular limitation of SAM. We want to convey that, irrespective of whether these adaptations are made with an awareness of SAM's this limitation, **our emphasis lies in conducting in-depth analyses (linear probing and attention distance statistics) and offering insights into SAM's constraints**. We believe that such analyses are essential in shedding light on potential areas for improvement in SAM in the future.

Reviewer comment (Experiments):
There is confusion about the authors’ choices regarding the scaling for the convolutional layers. The interpolation method used is not specified, and it is unclear if it is learned. If not learned, questions arise about whether larger upsampling factors can be effective with the chosen kernel size of 3x3. Adding a row to table 4 with a scaling factor of 8 would provide valuable insights.

Author response:
**Q4: Choice of Interpolation Method and Larger Upsampling Factors.** 

We utilize bilinear interpolation and have added a row with a scaling ratio of 8 in **Table 4** of our paper. The findings indicate that **a larger scaling factor does not consistently lead to improved performance, as the object scale exhibits variability across different datasets**. Hence, we propose to let the model itself learn to select scale dynamically through MoE (Mixture of Experts).

Reviewer comment (Reproducibility):
Clarify where exactly in the image encoder ViT the Conv-LoRA bypass is added.

Author response:
**Q5: Where is Conv-LoRA added in ViT?**

We apply Conv-LoRA to the query, key and value matrices in self-attention layers, the same locations of LoRA.

Reviewer comment (Writing):
In the introduction, the authors claim that SAM is underperforming on certain domains. The supporting references for this claim are located in the Related Work section. These references should also be added to the first mention of the claim.

Author response:
**Q6: Add more references.**
We have added the suggested references in the first paragraph of introduction.

Reviewer comment (Novelty):
The authors identify the adaptation of SAM for end-to-end multi-class segmentation as one of the major contributions of their work. A paragraph should be added to the related work section discussing other efforts to do so, if any exist.

Author response:
**Q7: Add related works.**
We have added one new paragraph in the related work section introducing other concurrent efforts in finetuning SAM. Following your suggestion, we have removed "end-to-end segmentation" from our major contributions.

Reviewer comment (Presentation):
Figures 8, 9, and 10 should have their own labels.

Author response:
**Q8: Missing labels in Figures 8,9, and 10**

Good catch. We have added labels in those figures.

Reviewer comment (Writing):
Ensure that all variable names are defined and used consistently (e.g., in Figure 4 and Appendix A). Some parts of Appendix A are unclear, such as the origin of variable B in equation 2.

Author response:
**Q9: Variable name consistency**

We add the definition of the variable names in the caption of **Figure 4.** 

In **Equation 2**, the dimensionality of $x$ is denoted as $B \times C_{in} \times H \times W$, where $B$ represents the batch size, $C_{in}$ denotes the number of input channels, and $H$ and $W$ correspond to the height and width, respectively. As encoder handles the channel dimension, the size of $W_ex$ being $B \times r \times H \times W$, where $W_e \in R^{r \times C_{in}}$. Previously, the batch size dimension was omitted; however, for clarity, we will now explicitly state the size of $x$ in Equation 2.

Reviewer comment (Presentation):
Add a figure with example images for the medical domain since it is prominently featured in the abstract.

Author response:
**Q10: Add example images from the medical domain**

We have provided some medical image examples in **Figure 6** **in Appendix E**.

Reviewer comment (Reproducibility):
Consider releasing the code to facilitate further research, collaboration, and practical application of the work in adapting SAM to new domains.

Author response:
**Q12: Code Release**

Good call. We are cleaning up our codebase and will release it later. We definitely want to contribute to the open-source community and facilitate future research.