Paper: Hierarchical Side-Tuning for Vision Transformers (ICLR-2024)

Reviewer comment (Experiments):
I’d love to see a scaling analysis. E.g. what happens as you scale up the size of the HST network? I.e. compare the performance as you scale up the training data, network size — and compare the performance to analogous settings of LoRA.

Author response:
| Mask R-CNN 1x schedule (ViT-L) | Params(M) | APb  | APb(50) | APb(75) | APm  | APm(50) | APm(75) |
|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| Full-tuning| 337.3| 45.7 | 68.9| **49.4** | 41.5 | 65.6| **44.6**|
| Head-tuning| 33.6| 31.6 | 56.4 | 32.0| 31.3 | 53.3 | 32.5|
| LoRA-64| 39.84| 44.8 | 68.8 | 48.9| 41.2 | 65.3 | 44.0|
| HST| 39.62| **45.7** | **69.1**| 49.2 | **41.6** | **65.6** | 44.4|

We can observe that HST performs more satisfactorily when using larger models like ViT-L. There is a performance gap of 2.6 AP_b between HST and full finetune on the base model, while achieving the same or even better performance on the large model. Additionally, the LoRA method also demonstrates further improvement in performance with larger models, although it still remains slightly lower than full finetune by approximately 1.0 AP_b.

Reviewer comment (Experiments):
The ablation study on VTAB is well-done, but since the paper positions HST as aimed mainly at dense prediction it would be helpful to see a similar analysis on COCO — or at least the current ablation study broken down over classification/structured/specialized (as in Table 1).

Author response:
In the ablation study, we provided both the average precision on VTAB and the values of AP_b and AP_m using Mask R-CNN on COCO. Here, we present more detailed experimental results on VTAB.

|Ablation MetaT (N) | Natural | Specialized | Structured |
|:-:|:-:|:-:|:-:|
| 1 | 82.83 | 87.13 | 64.45 |
| 4 | 82.88 | 86.08 | 63.66 |
| 8 | 82.78 | 87.05 | 64.53 |
| 16 | 82.97 | 87.75 | 64.58 |
| 32 | 83.02 | 87.70 | 64.71 |

|Method|LN-Tuning|Weight-Sharing|GlobalT|FG Injection|Natural|Specialized|Structured|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|ViT-B w/. HSN| |  |  | | 81.44 | 85.43 | 57.16 |
|HST.a| +|  |  | | 82.84 | 86.17 | 61.90 |
|HST.b| + | + | |  | 82.74 | 86.67 | 62.26 |
|HST.c| + | + |+ |  | 82.74 | 86.67 | 62.26 |
|HST| +|+ |+|+| 82.83 | 87.13 | 64.45 |

Reviewer comment (Evaluation):
The grad-cam visualizations were nice, but were these cherry picked? The results were unsubstantiated with numbers (e.g. object localization via grad-cam) and weren’t really discussed in the paper. I’d either add a discussion or remove the visualizations.

Author response:
We can confidently state that these results have not been cherry-picked. In fact, due to limitations in the length of the manuscript, we were only able to temporarily provide a set of visualizations in the main text. Nevertheless, more visualizations are available in Figure 11 of the appendix. It can be observed that the visualization outcomes of Grad-CAM are as expected, owing to the hierarchical architecture of HSN employing convolutional embeddings and modeling fine-grained features in the shallow stages. Consequently, the model naturally has the ability to focus on detailed foreground representations.

Reviewer comment (Experiments):
I am curious how well-tuned were the baselines for LoRA, etc. Were these copied from existing papers, or did these have a similar HP search, compared to HST?

Author response:
While numerous PETL methods primarily concentrate on image classification tasks, the experimental results for these methods in image classification are widely available in multiple papers. Our paper's VTAB results are consistent with those reported. However, when it comes to dense prediction tasks, it becomes challenging to find relevant and dependable results in existing literature. Therefore, we took the initiative to independently replicate the results of all relevant methods in dense prediction tasks. We ensured that all experiments were conducted with the same set of training hyperparameters to maintain consistency across the evaluations.

Reviewer comment (Evaluation):
Why do you think the method outperforms full FT for VTAB, but not as strong for ObjDet/SemSeg? Is this simply a result of COCO being a large dataset? The scaling experiment above might help answer this.

Author response:
Under the experimental setup of the VTAB benchmark with only 1000 training images, full fine-tuning can easily lead to overfitting. However, when we transition to general image classification datasets such as CIFAR-100 or ImageNet-1k (as shown in Table 6 in our paper's appendix or Table 5 in SSF[1]), we observe that full fine-tuning still maintains optimal performance. Therefore, the size of the training dataset plays a significant role.

Additionally, it is important to note that detection and segmentation tasks differ fundamentally from classification tasks. Dense prediction tasks require models to discern features at various granularities, and simply adding a few trainable parameters to the backbone does not enhance the model's ability to model multi-scale features. As a result, the performance may be inferior.

Finally, as mentioned in the scaling analysis, based on the growth trend, we believe that PETL methods will exhibit more powerful performance as the pre-trained model becomes larger. Hence, the size of the model is also one of the influencing factors.

[1] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.

Reviewer comment (Experiments):
Figure 2: Why is the conv stem necessary? What happens when it is excluded? How is the architecture adapted for classification tasks in VTAB?

Author response:
Firstly, the conv stem serves a similar purpose to the patch embedding in popular pyramid-style vision Transformer models. It enables downsampling to reduce the number of input tokens by fourfold. The conv stem consists of two 3x3 convolutional layers with a stride of 2, which introduces spatial prior knowledge related to the image. In our experiments, we observed that the conv stem performs better in terms of both performance and efficiency compared to directly using a 4x4 patch embedding with a stride of 4.

If the conv stem layer is removed, we would directly model an H x W feature map. In this scenario, whether conducting self-attention or cross-attention, the model would face training instability due to the excessively large number of input tokens.

Lastly, when transitioning to image classification tasks, there is no need to utilize intermediate [1/4, 1/8, 1/16] features. Only the 1/32 output feature requires a global average pooling operation, followed by a linear layer for classification.

Reviewer comment (Experiments):
The paper emphasizes the importance of a side network which leverages intermediate activations extracted from the backbone and generates multi-scale features to make predictions. However, the proposed algorithm also incorporates several other designs that share similarity with previous work, for example, adding meta token is similar to VPT [2], and LN-tuning is reminiscent of BitFit [3] which tunes the bias in the network. Although the final results are very promising, it is not clear the improvement comes from the side network or other designs. An apple-to-apple comparison to baselines to show that the side network really matters (no LN tuning, not meta token, etc.) is probably important to support the claim that side network is important for dense prediction.

Author response:
>**w2: The proposed algorithm incorporates several other designs that share similarity with previous work, VPT and BitFit, and it is not clear the improvement comes from the side network or other designs.**

MetaT and prompt tokens in VPT serve the purpose of introducing trainable tokens into ViT. However, they have distinct capabilities within their respective fine-tuning frameworks. MetaT is designed to facilitate efficient computation, using linear complexity for cross-attention within HSN while integrating acquired knowledge into HSN. In contrast to VPT, our approach eliminates the need to manually determine the optimal number of tokens for each task, which often requires a large quantity. Additionally, LN-tuning is employed to reduce the distribution gap between MetaT and the output features of image patch tokens, enabling efficient meta-global/fine-grained injection.

Furthermore, to better illustrate the effects of the side network, we conducted two specific experiments. HST.v0 is without MetaT and LN-Tuning, with cross-attention relying on globalT to serve as the key and value. The second experiment is to add LN-Tuning to LoRA, verifying the effect of integrating different PETL methods. The results are as follows.

| Method | Mean(%) | APb| APm|
|-|:-:|:-:|:-:|
|HST.v0|74.1|33.5|32.9|
|LoRA|72.3|36.2|35.0|
|LoRA with LN_Tuning|72.9|37.0|36.0|

We can observe that despite the removal of MetaT and LN-Tuning, this method still achieves state-of-the-art performance in classification tasks and surpasses VPT and AdaptFormer in dense prediction tasks. However, the absence of MetaT and LN-Tuning results in the loss of their crucial ability to inject meta-global/fine-grained information, which is the most critical aspect of our framework. On the other hand, even with the integration of LN-Tuning, LoRA does not exhibit a significant improvement in accuracy. Therefore, although MetaT and LN-Tuning may share similarities with other methods, they play distinct roles in our framework, and each component complements the others to achieve optimal performance.

Reviewer comment (Reproducibility):
How to implement the hierarchical dense prediction in Fig. 2?

Author response:
In simple terms, by equipping ViT with a trainable side network to enable its multi-scale output capability, it can be viewed as an overarching pyramid-based architectural model. Subsequently, by incorporating neck modules (FPN, ChannelMapper, etc) and any desired detection or segmentation heads, it can effectively accomplish various dense prediction tasks.

Reviewer comment (Experiments):
It's not convincing of the analysis of "constraining the number of trainable prompts to a few number". Besides, the ablation study shows continuous growth when the number N increases.

Author response:
The quantity of MetaT in HST plays a crucial role in determining both the transfer performance and the computational complexity of the Side blocks. Increasing the number of MetaT has an intuitive link with performance enhancement, as observed in VPT as well. However, our findings indicate that using just one MetaT is sufficient to yield satisfactory results in image classification transfer tasks. Although a slight performance boost is observed with 32 MetaT, it significantly increases training and inference costs, which is similar in dense prediction tasks. Therefore, to strike a balance between speed and accuracy, it is favorable to choose varying MetaT quantities (not exceeding 8) based on specific tasks as a hyperparameter choice.

Reviewer comment (Evaluation):
14.4% (76.0% vs. 65.6%)  --> 10.4%

Author response:
Thanks for pointing out this mistake in our paper.  We will carefully review the paper to ensure that such mistakes are corrected in the final version.

Reviewer comment (Experiments):
Effect of globalT. In the ablation study, globalT shows improvements without GF injection. I would like to know if globalT is still necessary even if GF injection exists.

Author response:
As depicted in the table below, we performed an additional ablation experiment without GlobalT, focusing solely on FG Injection. We observed a significant improvement in performance solely due to the implementation of FG Injection. However, when combining both FG Injection and GlobalT, we achieved even better results. Importantly, the introduction of GlobalT resulted in negligible overhead.

|Method|LN-Tuning|Weight-Sharing|GlobalT|FG Injection|Mean(%)|AP_b| AP_m|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|HST.c| +| + | + | | 75.2|34.5|33.0|
|HST.d| + | + | | + | 75.7|39.5|37.1|
|HST| +|+ |+|+| 76.0|40.5|38.0|

Reviewer comment (Experiments):
In ablation study, weight-sharing does not drop the performance. Does this mean that injecting the features of each block is unnecessary? Or, if we reduce the number of side block in each stage and increase their middle channels to keep the similar parameters. How about this design?

Author response:
Thank you for your informative comment. We would like to say that the designs you mentioned were indeed part of our HSN early version. In that version, we experimented with setting only one Side block in each stage, resulting in a total of four Side blocks in the HSN. Furthermore, we fused the output features of different ViT blocks within the same stage, along with their respective meta tokens, and input them into the Side Block. However, this design approach did not yield satisfactory results. We speculate that the inadequate performance may be attributed to the insufficient number of Side blocks in effectively modeling the fusion of multi-scale image features and ViT feature information within the HSN. Despite having a larger number of channels, the network was too shallow. This discovery ultimately inspired us to the design of alignment of ViT blocks and Side blocks.

Reviewer comment (Experiments):
The effect of metaT is not proved.

Author response:
|Method| Mean(%)|AP_b|AP_m|
|-|:-:|:-:|:-:|
|only GlobalT| 75.3|38.0|36.1|
|only MetaT| 75.7| 39.5|37.1|
|MetaT + GlobalT|76.0|40.5|38.0|

As indicated in the table, the performance achieved solely using GlobalT does not surpass that obtained by solely using MetaT. This discrepancy is primarily due to MetaT's ability to extract more enriched feature from each ViT block compared to GlobalT. Additionally, similar to the prompts in VPT, MetaT demonstrates the capability to transfer information to downstream tasks. We apologize the oversight in not providing an analysis of MetaT's effectiveness in the paper. We will provide more details to elaborate on the effectiveness of MetaT.

Reviewer comment (Evaluation):
Can the authors provide more discussion about the comparison on Table 1, since the performance is not always the best.

Author response:
Firstly, let's delve into the VTAB-1k benchmark in detail. The VTAB-1k benchmark consists of 19 tasks spanning various domains, including Natural (images from standard cameras), Specialized (context-specific domains like medical and satellite imaging), and Structured (synthesized images with controlled conditions). Each task-specific dataset contains 1000 training samples, with varying sample counts per class, and is evaluated using the original test set.

It is important to note that with only 1,000 training images, different PEFT methods demonstrate their own strengths and weaknesses across diverse image domains. However, the crucial factor is whether a method can deliver impressive performance across all 19 tasks, showcasing robust and powerful transfer capabilities. Our HST method accomplishes exactly this, exhibiting outstanding performance across various domains: an average accuracy of 82.83% in Natural, 87.13% in Specialized, 64.45% in Structured, and an overall average accuracy of 76.0%, all of which currently stand as state-of-the-art performance benchmarks.

Furthermore, we conducted experiments in general image classification using CIFAR-100 and four other FGVC datasets. Our HST method outperforms other PETL methods when using both ImageNet-22k and MAE pre-trained weights, further reinforcing the robust transfer capabilities inherent in HST. For detailed results, please refer to Table 6 in Appendix B.

Reviewer comment (Experiments):
The performance on much large models, like ViT-L or ViT-H.

Author response:
| Mask R-CNN 1x schedule (ViT-L) | Params(M) | APb  | APb(50) | APb(75) | APm  | APm(50) | APm(75) |
|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| Full-tuning| 337.3| 45.7 | 68.9| **49.4** | 41.5 | 65.6| **44.6**|
| Head-tuning| 33.6| 31.6 | 56.4 | 32.0| 31.3 | 53.3 | 32.5|
| LoRA-64| 39.84| 44.8 | 68.8 | 48.9| 41.2 | 65.3 | 44.0|
| HST| 39.62| **45.7** | **69.1**| 49.2 | **41.6** | **65.6** | 44.4|

We can observe that HST performs more satisfactorily when using larger models like ViT-L. There is a performance gap of 2.6 AP_b between HST and full finetune on the base model, while achieving comparative or even better performance on the large model. Additionally, the LoRA method also demonstrates further improvement in performance with larger models, although it still remains slightly lower than full finetune by approximately 1.0 AP_b.