Paper: LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning (ICLR-2024)

Reviewer comment (Theory):
The working reason of CSPC is not entirely clear. The author should provide a more detailed explanation of the fundamental distinctions among the three types of sources used to construct the input prompts mentioned in Sec 3.2 and elaborate on why "adding the average embedding of patch tokens to the output prompts" leads to performance improvements.

Author response:
The proposed CSPC (Class-aware Spatial Prompt Coding) is strategically designed to perpetually amass class conscious features, thereby fortifying the model’s prowess in distinguishing and identifying visual categories. The role of the mean of patch tokens behind CSPC is used for enhancing classification performance, as  the mean of patch tokens is commonly used for linear probing in self-supervised ViTs, such as DINO and MAE. Adding them to the prompt tokens will help improve the model’s prowess in distinguishing and identifying visual categories. 

More importantly, computation overhead is minimal to keep all patch semantics. Furthermore, we conduct additional ablation studies exploring more sophisticated methods for accumulating spatial and positional information. Specifically, we investigate alternative techniques like k-means clustering on spatial patch tokens to enrich the spatial coding process and provide a comparative analysis of these methods. The overall performance get an increase on all the compared datasets when we use k-means to incorporate spatial information, while the additional computational cost leads to a trade-off. To be noticed, more exploration space on this spatial prompt coding will leave for future work.

| Spatial Prompt | CUB       | Flowers   | Cars      | Dogs      | NABirds   |
|----------------|-----------|-----------|-----------|-----------|-----------|
| average        | 73.86     | 82.32     | 74.75     | 82.05     | 71.73     |
| k-means        | **74.32** | **82.56** | **74.87** | **82.23** | **71.86** |

Reviewer comment (Experiments):
About generalizability. The authors have exclusively conducted experiments on the standard ViT architecture under self-supervised pretraining strategies. This raises questions about how LSPT performs when applied to a supervised pretrained model and more advanced ViT architectures. Additionally, comparing LSPT with more advanced parameter-efficient fine-tuning methods [a] [b] could further substantiate its effectiveness.

Author response:
[a] Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. ICCV, 2023. 
[b] Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. CVPR, 2023.


To address concerns about generalizability, we extend our experiments to include supervised pre-trained models and more advanced ViT architectures on ViT-B/16 pre-trained on supervised ImageNet-21k in the Table below. This provide a broader perspective on LSPT's applicability and effectiveness. 

| Method                   | Nature | Specialized | Structured | AVG   |
|--------------------------|--------|-------------|------------|-------|
| VPT (ECCV’22)            | 78.48  | 82.43       | 54.98      | 69.42 |
| EXPRES (CVPR’23) [c]        | 79.70  | 84.00       | 55.00      | 70.21 |
| SNF (CVPR’23)[b]            | 83.79  | 86.13       | 59.61      | 74.10 |
| Bi-AdaptFormer (ICCV’23) [a] | 82.11  | 86.40       | 62.43      | 74.73 |
| LSPT (ours)              | **85.26**  | **88.57**       | **66.25**      | **77.95** |


[a] Jie et al. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. ICCV, 2023. 

[b] Wang et al. Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. CVPR, 2023.

[c] Das et al. Learning Expressive Prompting With Residuals for Vision Transformers. CVPR, 2023.

Reviewer comment (Evaluation):
The training and inference processes may be more complex compared to VPT or other parameter-efficient methods. It would be beneficial to include metrics such as FLOPs or inference times to assess the efficiency of the proposed method.

Author response:
We understand the importance of assessing the computational efficiency of LSPT. To this end, we include metrics like training and inference time per batch in table below, where we fine-tune ViT-B/16 mae pre-trained weights on CUB dataset. This offers a comprehensive comparison of LSPT's efficiency against GaPT, providing insights into its practicality.

| Method           | Training   time per batch (s) | Inference   time per batch (s) |
|------------------|-------------------------------|--------------------------------|
| GaPT   (ICML'23) | 0.2406                        | 0.0871                         |
| LSPT   (ours)    | **0.2428**                        | **0.0872**                         |

Reviewer comment (Theory):
If I understand correctly, the authors propose that VPT may be susceptible to the so-called long-term forgetting problem. In response, LSPT solves this problem by integrating information from prompt tokens of different layers using an LSTM. This raises the question: what is the connection between long-term forgetting problem and performance improvement? If this question can be addressed, would it be more effective to employ an attention-based architecture instead of an LSTM?

Author response:
The reviewer's question about the connection between long-term forgetting and performance improvement is crucial. To address this, we introduce deeper quantitative analysis on the forgetting gate in learned LSTM to further justify our design choices. Specifically, we calculate the forget gate to decide how much of the previous prompt will be forgotten and how much of the previous prompt will be used in next steps, as shown in Table below. The number in the *Forgetting* colomn means how much information are discarded on average before feeding to the next layer. These quantitative results further elucidate how the forgetting gate aids in reducing forgetting in LSPT compared to VPT and GaPT. This analysis include detailed examinations of the forgetting metrics across different prompt tuning methods, demonstrating the effectiveness of LSPT's approach.

To be noticed, compared to VPT and GaPT whose ratios of forgetting are fixed and decides merely by the architecture or hyperparameter, the ratio of forgetting for our LSPT is an average over all the layers and data, controlled dynamically by the learned LSTM. This flexibility and selectivity of rememberring and forgetting also contributes to the final improvement. 


| Method | Forgetting | Peformance |
| -------- | -------- | -------- |
| VPT (ECCV'22)     | 1.00     | 68.33     |
| GaPT (ICML'23)     | 0.85     | 70.56     |
| LSPT (ours)     | **0.68**     | **73.86**     |

Additionally, we explore the effectiveness of an attention-based architecture compared to LSTM, and the choice of LSTM for long-term modeling helps introduce forget gate to enlarge the variance for prompt tokens. The auto-aggressive Transformer layer is applied among tokens, which could increase their similarity by weighted sum of token values. So, we believe there exists a risk of losing discrepancy among latent toke embeddings during training. Specifically, we include experimental results comparing the performance of LSPT in the Table below when using LSTM versus auto-aggressive Transformer layers, providing insights into the effectiveness of our approach.


| LPC         | CUB       | Flowers   | Cars      | Dogs      | NABirds   | AVG       |
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|
| Transformer | 72.52     | 81.26     | 73.58     | 81.23     | 70.86     | 75.89     |
| LSTM        | **73.86** | **82.32** | **74.75** | **82.05** | **71.73** | **76.94** |

Reviewer comment (Novelty):
[c] seems quite closely related to LSPT. This concurrent work is worthy to cite and discuss.

Author response:
We acknowledge the importance of discussing concurrent works closely related to LSPT. In our revised paper, we added a citation on the suggested work, which tried to add residual learnable tokens to the output of various computations. We also compare it with LSPT on ViT-B/16 pre-trained on supervised ImageNet-21k in the Table below. This helps in situating our work within the current research landscape and highlighting the effectiveness of our LSPT.


| Method                   | Nature | Specialized | Structured | AVG   |
|--------------------------|--------|-------------|------------|-------|
| EXPRES (CVPR’23) [c]        | 79.70  | 84.00       | 55.00      | 70.21 |
| LSPT (ours)              | **85.26**  | **88.57**       | **66.25**      | **77.95** |

Reviewer comment (Novelty):
My major concern is the motivation. Why preserving the spatial information is crucial? For classification, proper abstraction can be more beneficial, which discards unnecessary spatial details or provides not interpretable attention maps.

Author response:
We appreciate the reviewer's query about the importance of preserving spatial information in classification tasks. In our revised response, we aim to align more closely with the underlying intuition of self-supervised Vision Transformers (ViTs) and the role of Visual Prompt Tuning (VPT) in leveraging spatial information.

The core intuition behind our approach is grounded in the understanding that self-supervised ViTs inherently encode semantics in attention weights, which highlight the relevance of different patch tokens and thus, the underlying spatial information. In such architectures, the attention mechanism is adept at discerning which parts of the input are more relevant for a given task, making the spatial context of these patch tokens particularly significant. VPT, in this context, emphasizes and harnesses this spatial information. The visual prompts in VPT are designed to encode information through an aggregation process driven by learnable attention weights. This mechanism allows for a more nuanced and detailed understanding of spatial relationships within the data, which is particularly beneficial for tasks that require fine-grained differentiation between classes.

To empirically validate this concept, we have conducted experiments where the Class-Aware Spatial Prompt Coding (CSPC) module was applied in isolation. The results, as detailed in our ablation study, show a marked improvement in performance when CSPC is used. This demonstrates that the explicit encoding of spatial information via CSPC significantly enhances the model's ability to discern and classify visual data with a higher degree of accuracy.

| LPC | CSPC | CUB   | Flowers | Cars  | Dogs  | NABirds | Nature | Specialized | Structured |
|-----|------|-------|---------|-------|-------|---------|--------|-------------|------------|
|  &cross;  | &cross;    | 68.33 | 80.05   | 67.67 | 78.83 | 65.22   | 67.34  | 82.26       | 37.55      |
| &cross;    | &check;    | **76.28** | **85.23** | **72.16** | **80.51** | **70.63** | **72.36** | **83.56**   | **45.15**  |

Reviewer comment (Theory):
In addition, I think the long-term modeling of prompts is not well-motivated. Why the long-term dependency is necessary? Why the long-term modeling is helpful for the classification? The authors have tried to explain this with the human brain, but it seems too ambiguous. No technical evidence was given.

Author response:
We appreciate the reviewer's request for a more technical justification for the necessity of long-term modeling in prompt tuning. In our revised response, we focus on building upon the foundations laid by GaPT and further elucidating the specific advancements introduced with our Long-term Prompt Coding (LPC).

The concept of layer-to-layer prompt tuning, as demonstrated by GaPT, involves using new trainable prompts at each layer (akin to VPT-Deep) while also retaining semantics from previous layers. GaPT has shown that combining immediate layer prompts with long-term dependencies from previous layers results in more effective learning. This insight forms the foundation of our approach in LSPT. Building on this intuition, LSPT advances the concept by implementing a mechanism for selective memory retention and forgetting. Unlike the hard fusion of prompts in GaPT, LSPT employs an LSTM-based approach that strategically manages the balance between retaining useful information and discarding irrelevant details from earlier layers. This method effectively counters the issue of exponential forgetting observed in GaPT, enhancing the model's capacity to utilize long-term dependencies without the burden of cumulative noise.

To empirically validate our approach, we have conducted experiments focusing on the impact of LPC in isolation. The results, which we present in our revised manuscript, indicate a significant improvement in performance when LPC is employed. This is further evidenced by the attention maps generated in later stages of our model, which are notably clearer and more focused than those in baseline models. Such visualizations, along with empirical numeric results, demonstrate the efficacy of our approach in harnessing long-term dependencies for improved classification performance.


| LPC | CSPC | CUB       | Flowers   | Cars      | Dogs      | NABirds   | Nature    | Specialized | Structured |
|-----|------|-----------|-----------|-----------|-----------|-----------|-----------|-------------|------------|
| &cross;   | &cross;    | 68.33     | 80.05     | 67.67     | 78.83     | 65.22     | 67.34     | 82.26       | 37.55      |
| &check;   |  &cross;    | **78.12** | **89.17** | **75.38** | **82.75** | **72.58** | **75.18** | **84.28**   | **48.39**  |

Reviewer comment (Theory):
The class-aware spatial prompt coding is not motivated, too. What is the role of the mean of patch tokens? Why they are added to the prompt tokens?

Author response:
The proposed CSPC (Class-aware Spatial Prompt Coding) is strategically designed to perpetually amass class conscious features, thereby fortifying the model’s prowess in distinguishing and identifying visual categories. The role of the mean of patch tokens behind CSPC is used for enhancing classification performance, as  the mean of patch tokens is commonly used for linear probing in self-supervised ViTs, such as DINO and MAE. Adding them to the prompt tokens will help improve the model’s prowess in distinguishing and identifying visual categories. 

More importantly, computation overhead is minimal to keep all patch semantics. Furthermore, we conduct additional ablation studies exploring more sophisticated methods for accumulating spatial and positional information. Specifically, we investigate alternative techniques like k-means clustering on spatial patch tokens to enrich the spatial coding process and provide a comparative analysis of these methods. The overall performance get an increase on all the compared datasets when we use k-means to incorporate spatial information, while the additional computational cost leads to a trade-off. To be noticed, more exploration space on this spatial prompt coding will leave for future work.

| Spatial Prompt | CUB       | Flowers   | Cars      | Dogs      | NABirds   |
|----------------|-----------|-----------|-----------|-----------|-----------|
| average        | 73.86     | 82.32     | 74.75     | 82.05     | 71.73     |
| k-means        | **74.32** | **82.56** | **74.87** | **82.23** | **71.86** |

Reviewer comment (Evaluation):
The long-term modeling ability is not supported by evidence. The visualization of attention maps do not represent the long-term modeling ability, and in addition, they can be cherry-picked. The authors should find more direct evaluation ways to show the long-term modeling ability, especially with quantitative measures.

Author response:
To address the reviewer's concerns about the lack of direct evidence for the long-term modeling ability, we introduce deeper quantitative analysis on the forgetting gate in learned LSTM to further justify our design choices. Specifically, we calculate the forget gate to decide how much of the previous prompt will be forgotten and how much of the previous prompt will be used in next steps, as shown in Table below. The number in the *Forgetting* colomn means how much information are discarded on average before feeding to the next layer. These quantitative results further elucidate how the forgetting gate aids in reducing forgetting in LSPT compared to VPT and GaPT. This analysis include detailed examinations of the forgetting metrics across different prompt tuning methods, demonstrating the effectiveness of LSPT's approach.

To be noticed, compared to VPT and GaPT whose ratios of forgetting are fixed and decides merely by the architecture or hyperparameter, the ratio of forgetting for our LSPT is an average over all the layers and data, controlled dynamically by the learned LSTM. This flexibility and selectivity of rememberring and forgetting also contributes to the final improvement. 


| Method | Forgetting | Peformance |
| -------- | -------- | -------- |
| VPT (ECCV'22)     | 1.00     | 68.33     |
| GaPT (ICML'23)     | 0.85     | 70.56     |
| LSPT (ours)     | **0.68**     | **73.86**     |

Reviewer comment (Experiments):
The proposed method achieves better performance than previous SOTAs, but it uses more learnable parameters. For FGVC, 4 times more parameters were used, and for VTAB-1K, 5 times more parameters were used. For a fair comparison, the proposed method should decrease the number of introduced prompts.

Author response:
We acknowledge the concern regarding the increased number of learnable parameters in LSPT. To address this, we conduct experiments with a reduced number of prompts, ensuring a fairer comparison with previous SOTAs. Comparison results in Table below demonstrate the efficiency of LSPT in achieving high performance with the same tunable parameters.

| Method         | Params | CUB   | Flowers | Cars  | Dogs  | NABirds | AVG   |
|----------------|--------|-------|---------|-------|-------|---------|-------|
| VPT (ECCV'22)  | 1.02x  | 68.33 | 80.05   | 67.67 | 78.83 | 65.22   | 72.02 |
| GaPT (ICML'23) | 1.02x  | 70.56 | 78.55   | 71.7  | 78.9  | 67.26   | 73.39 |
| LSPT (ours)    | 1.02x  | **72.18** | **81.35**   | **73.82** | **81.02** | **70.15**   | **75.70** |

Reviewer comment (Experiments):
In addition, the proposed method may require more inference time due to LSTM module. Please compare the inference time with other VPT-based methods.

Author response:
Thanks for the suggestion! The impact of the LSTM module on inference time is an important consideration. We include a detailed analysis of metrics like training and inference time per batch in the table below, where we fine-tune MAE pre-trained ViT-B/16 weights on CUB dataset. Our LSPT achieves comparable time computational overhead to GaPT, the state-of-the-art VPT baseline on self-supervised ViTs.

| Method           | Training   time per batch (s) | Inference   time per batch (s) |
|------------------|-------------------------------|--------------------------------|
| GaPT   (ICML'23) | 0.2406                        | 0.0871                         |
| LSPT   (ours)    | **0.2428**                        | **0.0872**                         |

Reviewer comment (Experiments):
For long-term modeling, why LSTM is used? The auto-aggressive Transformer layer can also be applied?

Author response:
The choice of LSTM for long-term modeling helps introduce forget gate to enlarge the variance for prompt tokens. The auto-aggressive Transformer layer is applied among tokens, which could increase their similarity by weighted sum of token values. So, we believe there exists a risk of losing discrepancy among latent toke embeddings during training.  Additionally, we include experimental results comparing the performance of LSPT in the Table below when using LSTM versus auto-aggressive Transformer layers, providing insights into the effectiveness of our approach.

| LPC         | CUB       | Flowers   | Cars      | Dogs      | NABirds   | AVG       |
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|
| Transformer | 72.52     | 81.26     | 73.58     | 81.23     | 70.86     | 75.89     |
| LSTM        | **73.86** | **82.32** | **74.75** | **82.05** | **71.73** | **76.94** |

Reviewer comment (Novelty):
This paper appears to exhibit a substantial degree of similarity to the methodologies presented in the GaPT paper (Yoo et al., 2023). For instance, section 3.1.2 seems to be a rephrased version of Section 2, 'Visual Prompt Tuning,' from GaPT. Equations (1, 2, 3) in this paper are identical to equations (4, 5) in GaPT. Furthermore, Table (1, 2) and the corresponding experimental results are exactly the same as those in GaPT. The overall structure of this paper closely mirrors that of GaPT. It is highly irregular and unacceptable to replicate aspects of a research paper to such an extent.

Author response:
We acknowledge the reviewer's observation regarding the similarities between our work and the GaPT paper. It is important to note that our work initially builds upon the foundational concepts of GaPT, intending to improve upon them. We revised Section 3.1.2 to clarify our advancements beyond GaPT's scope and ensure that the text reflects our original contributions more distinctly. Furthermore, we updated our equations and tables to better differentiate our work from GaPT, emphasizing the unique aspects of LSPT.

Reviewer comment (Reproducibility):
Including a delineated algorithmic framework would significantly improve the replicability of the proposed LSPT method.

Author response:
To enhance the replicability of our method, we agree with the reviewer's suggestion to include a detailed algorithmic framework. We add a pseudo-algorithm in our revised manuscript in Appendix Section B, providing clear steps and methodologies for implementing LSPT.

Reviewer comment (Presentation):
The connection between the two proposed modules, CSPC and LPC, is weak. There is no strong motivation presented to justify the inclusion of CSPC, especially since the title of the paper pertains solely to LPC. Nonetheless, the authors introduce CSPC first, which suggests a greater emphasis on the importance of CSPC.

Author response:
We appreciate the feedback on the connection between CSPC and LPC. Our intention is to demonstrate that both spatial and long-term aspects are crucial for addressing forgetting in visual prompt tuning. We provide a more detailed explanation in the revised manuscript to justify the inclusion and parallel importance of both CSPC and LPC within LSPT. 

Additionally, our title also includes spatial prompt tuning, which refers to the proposed CSPC (Class-aware Spatial Prompt Coding) strategically designed to perpetually amass class conscious features, thereby fortifying the model’s prowess in distinguishing and identifying visual categories. Our method named Long-term Spatial Prompt Tuning in the title include both spatial and long-term aspects, reflecting the comprehensive scope of our work.

Reviewer comment (Novelty):
More visual prompt tuning papers should be included in the related works. There are only two papers introduced currently.

Author response:
Our current related works focus on vpt for self-supervised ViTs, but we recognize the necessity of including a broader range of related works in visual prompt tuning. In the revised manuscript, we incorporated additional references [1,2,3] on VPT to both self-supervised and supervised ViTs to provide a more comprehensive background and context for our research.

[1] Das et al. Learning Expressive Prompting With Residuals for Vision Transformers. CVPR, 2023.

[2] Wang et al. Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition. CVPR, 2023.

[3] Jie et al. Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy. ICCV, 2023.

Reviewer comment (Experiments):
The scope of the proposed LSPT within the realm of visual prompt tuning is confined to SSL-ViTs. This limitation constrains the real-world applicability of the proposed LSPT method. Additionally, are there any other baselines within this specific experimental context?

Author response:
We understand the reviewer's concern about the confined scope of LSPT to SSL-ViTs. To address this, we expand our experimental setup to compare baselines on ViT-B/16 pre-trained on supervised ImageNet-21k in the Table below, thereby demonstrating the generality and applicability of LSPT across different types of vision transformers.


| Method                   | Nature | Specialized | Structured | AVG   |
|--------------------------|--------|-------------|------------|-------|
| VPT (ECCV’22)            | 78.48  | 82.43       | 54.98      | 69.42 |
| EXPRES (CVPR’23) [1]        | 79.70  | 84.00       | 55.00      | 70.21 |
| SNF (CVPR’23) [2]           | 83.79  | 86.13       | 59.61      | 74.10 |
| Bi-AdaptFormer (ICCV’23) [3] | 82.11  | 86.40       | 62.43      | 74.73 |
| LSPT (ours)              | **85.26**  | **88.57**       | **66.25**      | **77.95** |

Reviewer comment (Presentation):
The visual representation of the pipeline in Figure 2 could be aesthetically improved. The rationale for using a non-standard rectangle to depict the LSTM is unclear. Moreover, the relationship between CSPC and LPC is not adequately illustrated. Implicitly, Figure 2 suggests that LPC serves as an incremental enhancement to CSPC, solely to boost performance.

Author response:
Thanks for the suggestion! We updated Figure 2 in the revision to more clearly illustrate the relationship and parallel importance of CSPC and LPC. The aim is to visually communicate that both modules are integral to LSPT and operate in tandem rather than in a hierarchical manner.

Reviewer comment (Reproducibility):
Have the authors replicated the baseline studies themselves, or have they merely cited the results from GaPT? It would be advantageous to detail the variations encountered during the reproduction of baseline and proposed methods.

Author response:
Yes. We confirm that we have replicated all the baseline studies ourselves under the same code framework to ensure fair and valid comparison, not solely relying on the results from GaPT.

Reviewer comment (Theory):
While the proposed modules make intuitive sense, the explanations lack quantitative analysis or theorems to rigorously justify the designs. Some ablation studies analyze contributions but more analysis connecting the methods to mitigating forgetting could strengthen the approach.

Author response:
We appreciate the reviewer's suggestion for a more rigorous quantitative analysis. We introduce deeper quantitative analysis on the forgetting gate in learned LSTM to further justify our design choices. Specifically, we calculate the forget gate to decide how much of the previous prompt will be forgotten and how much of the previous prompt will be used in next steps, as shown in Table below. The number in the *Forgetting* colomn means how much information are discarded on average before feeding to the next layer. These quantitative results further elucidate how the forgetting gate aids in reducing forgetting in LSPT compared to VPT and GaPT. This analysis include detailed examinations of the forgetting metrics across different prompt tuning methods, demonstrating the effectiveness of LSPT's approach.

To be noticed, compared to VPT and GaPT whose ratios of forgetting are fixed and decides merely by the architecture or hyperparameter, the ratio of forgetting for our LSPT is an average over all the layers and data, controlled dynamically by the learned LSTM. This flexibility and selectivity of rememberring and forgetting also contributes to the final improvement. 


| Method | Forgetting | Peformance |
| -------- | -------- | -------- |
| VPT-deep (ECCV'22)     | 1.00     | 68.33     |
| GaPT (ICML'23)     | 0.85     | 70.56     |
| LSPT (ours)     | **0.68**     | **73.86**     |

Reviewer comment (Experiments):
The spatial prompt coding uses a simple patch token averaging, which seems like a heuristic. More sophisticated ways to accumulate spatial/positional information may exist. This component could likely be improved.

Author response:
We acknowledge the reviewer's concern regarding the simplicity of our spatial prompt coding method. To address this, we conduct additional ablation studies exploring more sophisticated methods for accumulating spatial and positional information. Specifically, we investigate alternative techniques like k-means clustering on spatial patch tokens to enrich the spatial coding process and provide a comparative analysis of these methods. The overall performance get an increase on all the compared datasets when we use k-means to incorporate spatial information, while the additional computational cost leads to a trade-off. To be noticed, more exploration space on this spatial prompt coding will leave for future work.

| Spatial Prompt | CUB       | Flowers   | Cars      | Dogs      | NABirds   |
|----------------|-----------|-----------|-----------|-----------|-----------|
| average        | 73.86     | 82.32     | 74.75     | 82.05     | 71.73     |
| k-means        | **74.32** | **82.56** | **74.87** | **82.23** | **71.86** |

Reviewer comment (Evaluation):
The computational overhead and memory requirements of LSPT are not analyzed. This could be important for deployments.

Author response:
We recognize the importance of understanding the computational and memory requirements of LSPT, especially for practical deployments. We include a detailed analysis of metrics like training and inference time per batch in the table below, where we fine-tune MAE pre-trained ViT-B/16 weights on CUB dataset. Our LSPT achieves comparable time computational overhead and memory compared to GaPT, the state-of-the-art VPT baseline on self-supervised ViTs. This analysis cover the computational complexity and memory footprint of LSPT, providing a clear picture of its deployment feasibility.

|     Method     | Max Memory Usage (GB) | Training Time per Batch (s) | Inference Time per Batch (s) |
|----------------|------------------|-----------------------------|------------------------------|
| GaPT (ICML'23) | 23.78         | 0.2406                      | 0.0871                       |
| LSPT (ours)    | 24.02         | 0.2428                      | 0.0872                       |