Paper: TaCA: Hot-Plugging Upgrades for Foundation Model with Task-agnostic Compatible Adapter (ICLR-2024)

Reviewer comment (Experiments):
The authors should validate the flexibility of the proposed TaCA module. How about the performance when the TaCA is aligned with the other LLMs?

Author response:
**A1:** The proposed TaCA demonstrates exceptional flexibility by seamlessly integrating with a range of downstream tasks, eliminating the need for additional fine-tuning costs. Its versatility has been validated across tasks such as video-text retrieval, video classification, visual question answering, zero-shot image classification, and zero-shot image-text retrieval. Table 3 further illustrates TaCA's compatibility with the BLIP-2 LLM, confirming its robust performance when paired with different language models.

Reviewer comment (Evaluation):
The qualitative analysis and visualization in experiments are missing.

Author response:
**A2:** We recognize the importance of qualitative analysis and visualization in our experimental assessment. The manuscript currently discusses qualitative aspects, and visualizations are provided through the inference framework and image generation samples in Appendix C. We will enhance our manuscript with a more comprehensive qualitative analysis and additional visualizations in line with your feedback.

Reviewer comment (Experiments):
The authors should validate the flexibility of the proposed TaCA module. How about the performance when the TaCA is aligned with the other LLMs?

Author response:
**A1:** The proposed TaCA demonstrates exceptional flexibility by seamlessly integrating with a range of downstream tasks, eliminating the need for additional fine-tuning costs. Its versatility has been validated across tasks such as video-text retrieval, video classification, visual question answering, zero-shot image classification, and zero-shot image-text retrieval. Table 3 further illustrates TaCA's compatibility with the BLIP-2 LLM, confirming its robust performance when paired with different language models.

Reviewer comment (Evaluation):
The qualitative analysis and visualization in experiments are missing.

Author response:
**A2:** We recognize the importance of qualitative analysis and visualization in our experimental assessment. The manuscript currently discusses qualitative aspects, and visualizations are provided through the inference framework and image generation samples in Appendix C. We will enhance our manuscript with a more comprehensive qualitative analysis and additional visualizations in line with your feedback.

Reviewer comment (Novelty):
The approach is incremental, and the techniques employed are all verified strategies. Specifically, it utilizes a combination of distillation methods and contrastive learning, forming a hybrid approach.

Author response:
**Q1: The approach is incremental.**

**A1:** We reclaim that our key contributions are: 

(1) For the **first time**, we introduce "hot-plugging upgrades", which directly deploys the new foundation models **without retraining cost**. Specifically, we introduce the **task-agnostic** compatible adapter (TaCA), which enhances performance across a variety of downstream tasks. 

(2) Our method not only demonstrates compelling practical value but also boasts remarkable industrial applications. Unlike the traditional approach of fine-tuning specific downstream heads, which often incurs prohibitively high training costs and exhibits suboptimal transferability, our approach shines. To illustrate, the process of retraining a Q-Former in BLIP-2 demands approximately 9 days utilizing 16 A100 GPUs. TaCA, in contrast, offers a versatile task-agnostic capability, enabling seamless adaptation to various downstream tasks without incurring any finetuning costs.

(3) The significance and novelty of our contributions have been duly acknowledged by Reviewer #TD34 and #N62r, further attesting to the groundbreaking nature of our work.

Reviewer comment (Experiments):
Why not conduct experiments on more basic image classification and retrieval datasets (e.g., MSCOCO and imagenet)? If the effectiveness of this method can be verified on a more basic dataset, I am willing to increase my score.

Author response:
**Q2: Add experiments on more basic image classification and retrieval datasets (e.g., MSCOCO and imagenet)**

**A2:** We have broadened the scope of our experiments to include fundamental image classification and retrieval tasks. The results, detailed in Tables 1 and 2, demonstrate TaCA's consistent performance improvements across these basic image tasks.



| **Model**         | **ImageNet** | **CIFAR100** |
| ----------------- | ------------ | ------------ |
| **ViT-B/16**      | 68.6         | 68.7         |
| **ViT-L/14**      | 75.3         | 77.9         |
| **TaCA-ViT-L/14** | 74.3(+5.7)   | 76.5(+7.8)   |

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

| **Model**         | **MSCOCO**   | **Flickr**  |
| ----------------- | ------------ | ----------- |
| **ViT-B/16**      | 42.75        | 72.18       |
| **ViT-L/14**      | 46.42        | 75.08       |
| **TaCA-ViT-L/14** | 44.71(+3.67) | 73.97(+2.9) |

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Reviewer comment (Novelty):
In my opinion, TaCA, which utilizes an adapter to align a large-scale visual model with a smaller-scale visual model, undermines the purpose of changing the visual model. This approach hampers the transferability of the large-scale visual model, potentially limiting its advantages. Moreover, based on the results presented in Table 2, TaCA only shows marginal enhancements in downstream video classification tasks compared to directly employing a large-scale visual model.

Author response:
**Q3: TaCA utilizes an adapter to align a large-scale visual model with a smaller-scale visual model. In Table 2, TaCA only shows marginal enhancements in downstream video classification tasks compared to directly employing a large-scale visual model**

A3: We acknowledge the confusion in our initial explanation. **The purpose of TaCA is to efficiently align the old model with a large-scale or powerful visual model, thereby enhancing downstream task performance.** In Table 2, the de-emphasized lines represent the results achieved by fine-tuning all downstream heads, which are task-specific and inefficient. Conversely, the emphasized lines depict the performance obtained by directly deploying the TaCA model without making any modifications to the downstream modules.

Reviewer comment (Experiments):
While this paper assesses the effectiveness of TaCA in video-related tasks, it overlooks numerous studies that apply CLIP to image-based tasks such as image-text retrieval, image segmentation, and few-shot image classification. The absence of experiments on image-related tasks in this paper creates a gap in evaluating TaCA.

Author response:
**Q4: Add experiments on image-related tasks like image-text retrieval, image segmentation, and few-shot image classification.**

**A4:** For further details on the experiments encompassing image-text retrieval, image segmentation, and few-shot image classification, please refer to the response to Question 2.

Reviewer comment (Experiments):
What would happen if Old VFM and New VFM were different (e.g., VIT-B to ResNet-50)? Can we distill and transfer knowledge between VFMs of different architectures to each other? For example, distilling knowledge from miniGPT or LLAMA (decoder only architecture) to CLIP?

Author response:
**Q5: What would happen if Old VFM and New VFM were different? Can knowledge of different architectures be distilled? For example, distilling knowledge from miniGPT or LLAMA (decoder only architecture) to CLIP?**

**A5:** (1) Our research in Table 9 establishes that TaCA can generalize across various foundational models, allowing for knowledge distillation between different architectures. However, the performance may be less optimal if the new model is not as capable as the previous one (e.g., replacing ViT-B with ResNet-50). (2) While this paper confirms TaCA's efficiency with visual models, its applicability to language models, such as distilling knowledge from decoder-only architectures like miniGPT or LLAMA to CLIP, remains an area for future exploration.

Reviewer comment (Novelty):
The method is incremental, and the methods used are all validated schemes, which is actually distillation method combined with contrastive learning.

Author response:
**Q1: The approach is incremental.**

**A1:** We reclaim that our key contributions are: 

(1) For the **first time**, we introduce "hot-plugging upgrades", which directly deploys the new foundation models **without retraining cost**. Specifically, we introduce the **task-agnostic** compatible adapter (TaCA), which enhances performance across a variety of downstream tasks. 

(2) Our method not only demonstrates compelling practical value but also boasts remarkable industrial applications. Unlike the traditional approach of fine-tuning specific downstream heads, which often incurs prohibitively high training costs and exhibits suboptimal transferability, our approach shines. To illustrate, the process of retraining a Q-Former in BLIP-2 demands approximately 9 days utilizing 16 A100 GPUs. TaCA, in contrast, offers a versatile task-agnostic capability, enabling seamless adaptation to various downstream tasks without incurring any finetuning costs.

(3) The significance and novelty of our contributions have been duly acknowledged by Reviewer #TD34 and #N62r, further attesting to the groundbreaking nature of our work.

Reviewer comment (Experiments):
Why not conduct experiments on more basic image classification and retrieval datasets.

Author response:
**Q2: Add experiments on more basic image classification and retrieval datasets (e.g., MSCOCO and imagenet)**

**A2:** We have broadened the scope of our experiments to include fundamental image classification and retrieval tasks. The results, detailed in Tables 1 and 2, demonstrate TaCA's consistent performance improvements across these basic image tasks.



| **Model**         | **ImageNet** | **CIFAR100** |
| ----------------- | ------------ | ------------ |
| **ViT-B/16**      | 68.6         | 68.7         |
| **ViT-L/14**      | 75.3         | 77.9         |
| **TaCA-ViT-L/14** | 74.3(+5.7)   | 76.5(+7.8)   |

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

| **Model**         | **MSCOCO**   | **Flickr**  |
| ----------------- | ------------ | ----------- |
| **ViT-B/16**      | 42.75        | 72.18       |
| **ViT-L/14**      | 46.42        | 75.08       |
| **TaCA-ViT-L/14** | 44.71(+3.67) | 73.97(+2.9) |

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Reviewer comment (Experiments):
While the idea of hot-plugging adapters is intuitively sound at first glance, this paper lacks quantitative evidence to support this motivation. Specifically, the motivation of this paper is: *When replacing the visual backbones, fine-tuning the proposed adapter is better than fine-tuning the downstream task-specific head*. Therefore, a comparison between these two fine-tuning methods should be presented, and such a comparison should be in a fair enough setting because, in my opinion, it should be the most important experiment for the whole manuscript. Specifically, the author should compare 1) the trainable parameter amounts, 2) training FLOPs, 3) The data amounts needed for fine-tuning, and 4) the fine-tuning schedule of these two fine-tuning paradigms. In addition, the results in Table 1 show that the *TACA-BetterModel*'s performance is inferior to directly fine-tuning the task-specific head with a *BetterModel*, i.g., ViT-H, which also shows the necessity of such a comparison.

Author response:
**Q1: About the motivation of hot-plugging adapters.** 

**A1:** (1) We acknowledge the confusion in our initial explanation. Our aim is to highlight **not** the superiority of adapter fine-tuning over task-specific head fine-tuning, but rather the **inefficiencies and inflexibility** of the latter. Hot-plugging adapters offer a **task-agnostic, generalized**, and **efficient** alternative, enhancing the performance of various downstream tasks without incurring training costs.

(2) As shown in Table 1, direct fine-tuning of downstream heads can outperform TaCA, but it remains task-specific. The fine-tuning schedules and inference details for these paradigms are presented in Appendix C.  In the video-text retrieval task, we fine-tune three distinct sets of downstream heads on the MSRVTT, MSVD, and DiDeMo datasets, resulting in the training of a total of 42M parameters. For the video classification task, fine-tuning the classification head requires nearly twice as many parameters as the TaCA model. However, TaCA requires training just once, with no additional fine-tuning costs, yielding a more parameter-efficient and training-free application in downstream tasks. This discussion will be incorporated into the final paper.



| **Strategy**   | **Learnable params.** | **Finetune data**      | **MSRVTT** | **MSVD** | **DiDeMo** |
| -------------- | --------------------- | ---------------------- | ---------- | -------- | ---------- |
| **Finetuning** | 14M                   | WIT+MSRVTT/MSVD/DiDeMo | 44.8       | 46.7     | 38.1       |
| **TaCA**       | 17M                   | No need                | 44.5       | 45.6     | 36.6       |

Table1: Comparison on Video-Text Retrieval datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

| **Strategy**   | **Learnable params.** | **Finetune data** | **K400** | **UCF-101** |
| -------------- | --------------------- | ----------------- | -------- | ----------- |
| **Finetuning** | 29M                   | Kinetics-400      | 87       | 85.7        |
| **TaCA**       | 17M                   | No need           | 83.6     | 83.1        |

Table2: Comparison on Video Classification datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

Reviewer comment (Experiments):
The manuscript should include more experiments to show its generalization ability to other non-CLIP models. For example, can the proposed method work on classic vision tasks, like detection or segmentation?

Author response:
**Q3: Generalization to Non-CLIP Models and Classic Vision Tasks. Can the proposed method work on classic vision tasks, like detection or segmentation?** 

**A3:** We have validated TaCA's effectiveness on non-CLIP models, such as Dinov2 and BEiT, in Table 9 (page 9). These models show slight improvements and meet the compatibility criterion for various tasks.

We have also applied our method to classical vision tasks, such as zero-shot image classification and zero-shot image-text retrieval, with results presented in Tables 1 and 2 demonstrating consistent improvements. The potential application of TaCA to detection and segmentation will be explored in forthcoming studies.





| **Model**         | **ImageNet** | **CIFAR100** |
| ----------------- | ------------ | ------------ |
| **ViT-B/16**      | 68.6         | 68.7         |
| **ViT-L/14**      | 75.3         | 77.9         |
| **TaCA-ViT-L/14** | 74.3(+5.7)   | 76.5(+7.8)   |

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

| **Model**         | **MSCOCO**   | **Flickr**  |
| ----------------- | ------------ | ----------- |
| **ViT-B/16**      | 42.75        | 72.18       |
| **ViT-L/14**      | 46.42        | 75.08       |
| **TaCA-ViT-L/14** | 44.71(+3.67) | 73.97(+2.9) |

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Reviewer comment (Experiments):
I am also curious if the method can be applied to replacing *different* foundation models. For example, can we use it to replace a DINO ViT with an ImageNet-22k Supervised ViT?

Author response:
**Q4: Does TaCA work well when replacing *different* foundation models? Can we use it to replace a DINO ViT with an ImageNet-22k Supervised ViT?** 

**A4:** TaCA's adaptability across diverse foundational backbones is discussed in Table 9 of the main paper. While it is possible to replace a DINO ViT with an ImageNet-22k Supervised ViT using TaCA, the performance may be suboptimal if the new foundation model's representational capabilities are inferior. The efficacy of downstream tasks is contingent upon the foundational models' improved representational quality.

Reviewer comment (Novelty):
The proposed head-fixing strategy makes me think about the head-inheriting trick that is commonly used in knowledge distillation [a][b]. Therefore, discussing it in the related work section will increase the comprehensiveness of this work.

Author response:
**Q5: Incorporating Knowledge Distillation in Related Work**

**A5:** We have included a new subsection on knowledge distillation in the related works of our revised manuscript, which can be found highlighted on page 18.