Paper: Consistency-guided Prompt Learning for Vision-Language Models (ICLR-2024)

Reviewer comment (Novelty):
While the paper is clear, and the numerical results are noteworthy, its novelty isn't entirely clear. The paper's self-consistency terms seem similar to those in self-supervised learning (SSL) methods. The authors' claim of differentiating their approach from SSL, where two perturbed inputs within a single encoder are used, doesn't fully convince.

Author response:
> The paper's self-consistency terms seem similar to those in self-supervised learning (SSL) methods. 

Our model does indeed use the principles of SSL for applying consistency constraint on the vision-language framework. While in general SSL frameworks have two *learnable encoders* which are both trained, in our method, one encoder is *frozen* while the other is learnable. This method effectively enables knowledge distillation from the frozen encoders to the learnable ones, thus maintaining the generalization strength of the pre-trained base model while tackling a new task in a few-shot scenario. In some sense, our approach is somewhat between SSL and distillation. We have included this discussion in the Introduction (Page 2, Paragraph 2).

Reviewer comment (Writing):
In SSL, typically there are two encoders: an online encoder and a momentum encoder. The paper’s pre-trained and trainable encoders appear analogous to SSL’s momentum and online encoders, respectively. The authors should clarify this similarity.

Author response:
> The paper's self-consistency terms seem similar to those in self-supervised learning (SSL) methods. 

Our model does indeed use the principles of SSL for applying consistency constraint on the vision-language framework. While in general SSL frameworks have two *learnable encoders* which are both trained, in our method, one encoder is *frozen* while the other is learnable. This method effectively enables knowledge distillation from the frozen encoders to the learnable ones, thus maintaining the generalization strength of the pre-trained base model while tackling a new task in a few-shot scenario. In some sense, our approach is somewhat between SSL and distillation. We have included this discussion in the Introduction (Page 2, Paragraph 2).

Reviewer comment (Novelty):
The paper omits recent relevant studies like Bayesian Prompt Learning [1] and Prompt Distribution Learning [2], which address overfitting in vision and language models. Discussing these in the related work and comparing them in sections like domain generalization are necessary.

Author response:
> Missing recent relevant studies. 

Thank you for pointing out two related works which we were not aware of before. We now provide a discussion on these two methods and compare the results as follows:

Bayesian Prompt Learning [1] is a prompt learning method that approaches the task from a Bayesian perspective, formulating it as a variational inference problem. ProDA [2] proposed a data-driven approach, that learns the soft prompts from a few downstream samples, discovering the task-related content with less bias than manual design. Our approach is quite different from these methods since we build over a multi-modal prompt tuning technique and propose three components aiming to improve both few-shot and zero-shot performance. 


Following, we show the comparison of our method with Bayesian prompt learning. We can't compare the results with ProDA since it adopted a different training protocol (only evaluated on few-shot learning) than most existing methods, including ours. 



| Method | Base Acc. | Novel Acc | HM |
| ------ | --------- | --------- | --- |
| Bayesian Prompt | 80.10 | 74.94 | 77.43 |
| CoPrmpt (ours)  | 84.00 | 77.23 | 80.48 |




| Method          | Caltech | Pets  | Cars  | Flowers | Food  | Aircraft | SUN397 | DTD   | EuroSAT | UCF   | Average |
|-----------------|---------|-------|-------|---------|-------|----------|--------|-------|---------|-------|---------|
| Bayesian Prompt | 93.67   | 90.63 | 65.00 | 70.90   | 86.30 | 24.93    | 67.47  | 46.10 | 45.87   | 68.67 | 65.95   |
| CoPrompt        | 94.50   | 90.73 | 65.67 | 72.30   | 86.43 | 24.00    | 67.57  | 47.07 | 51.90   | 69.73 | 67.00   | 




| Method          | ImNetV2 | ImNetS | ImNetA | ImNetR | Average |
|-----------------|---------|--------|--------|--------|---------|
| Bayesian Prompt | 64.23   | 49.20  | 51.33  | 77.00  | 60.44   |
| CoPrompt        | 64.25   | 49.43  | 50.50  | 77.51  | 60.42   |



As we find from the comparison of base-to-novel generalization, CoPropmt outperforms Bayesian Prompt by 3.05\% in harmonic mean over 11 datasets. It performs better than Bayesian Prompt on both base and novel categories. On the cross-dataset evaluation, CoPrompt outperforms Bayesian Prompt by 1.05\% on average over all datasets. Finally, on domain generalization, CoPrompt performs only 0.02\% lower than Bayesian Prompt on the average over all datasets, but looking into individual datasets, we find that CoPrompt outperforms Bayesian Prompt on all datasets except ImageNetA. 
We now discuss Bayesian Prompt and ProDA in our Related Work section (Page 3, Paragraph 2), and also expand the results in Tables 2 and 3 (Pages 6 and 7) to compare with Bayesian Prompt. We could not include Bayesian Prompt in Table 1 since it did not report the base, novel, and HM results like other methods.

Reviewer comment (Experiments):
In some cases, such as Bayesian Prompt Learning, these omitted methods outperform the methods in this paper. For example, in the domain generalization task, the Bayesian Prompt Learning method (%60.44) works better than the paper performance (%60.42).

Author response:
> Missing recent relevant studies. 

Thank you for pointing out two related works which we were not aware of before. We now provide a discussion on these two methods and compare the results as follows:

Bayesian Prompt Learning [1] is a prompt learning method that approaches the task from a Bayesian perspective, formulating it as a variational inference problem. ProDA [2] proposed a data-driven approach, that learns the soft prompts from a few downstream samples, discovering the task-related content with less bias than manual design. Our approach is quite different from these methods since we build over a multi-modal prompt tuning technique and propose three components aiming to improve both few-shot and zero-shot performance. 


Following, we show the comparison of our method with Bayesian prompt learning. We can't compare the results with ProDA since it adopted a different training protocol (only evaluated on few-shot learning) than most existing methods, including ours. 



| Method | Base Acc. | Novel Acc | HM |
| ------ | --------- | --------- | --- |
| Bayesian Prompt | 80.10 | 74.94 | 77.43 |
| CoPrmpt (ours)  | 84.00 | 77.23 | 80.48 |




| Method          | Caltech | Pets  | Cars  | Flowers | Food  | Aircraft | SUN397 | DTD   | EuroSAT | UCF   | Average |
|-----------------|---------|-------|-------|---------|-------|----------|--------|-------|---------|-------|---------|
| Bayesian Prompt | 93.67   | 90.63 | 65.00 | 70.90   | 86.30 | 24.93    | 67.47  | 46.10 | 45.87   | 68.67 | 65.95   |
| CoPrompt        | 94.50   | 90.73 | 65.67 | 72.30   | 86.43 | 24.00    | 67.57  | 47.07 | 51.90   | 69.73 | 67.00   | 




| Method          | ImNetV2 | ImNetS | ImNetA | ImNetR | Average |
|-----------------|---------|--------|--------|--------|---------|
| Bayesian Prompt | 64.23   | 49.20  | 51.33  | 77.00  | 60.44   |
| CoPrompt        | 64.25   | 49.43  | 50.50  | 77.51  | 60.42   |



As we find from the comparison of base-to-novel generalization, CoPropmt outperforms Bayesian Prompt by 3.05\% in harmonic mean over 11 datasets. It performs better than Bayesian Prompt on both base and novel categories. On the cross-dataset evaluation, CoPrompt outperforms Bayesian Prompt by 1.05\% on average over all datasets. Finally, on domain generalization, CoPrompt performs only 0.02\% lower than Bayesian Prompt on the average over all datasets, but looking into individual datasets, we find that CoPrompt outperforms Bayesian Prompt on all datasets except ImageNetA. 
We now discuss Bayesian Prompt and ProDA in our Related Work section (Page 3, Paragraph 2), and also expand the results in Tables 2 and 3 (Pages 6 and 7) to compare with Bayesian Prompt. We could not include Bayesian Prompt in Table 1 since it did not report the base, novel, and HM results like other methods.

Reviewer comment (Novelty):
The prompt learning and adapter learning method mentioned in this paper are introduced by MaPLe and CLIP-Adapter. The primary contribution of this paper only lies in the introduction of consistency constraint learning. Thus, regarding the method as prompt learning is a bit ambiguous in my opinion.

Author response:
Prompts are the main learnable parameters in our method, and the goal of our solution is to train the learnable prompt while ensuring effective adaptation to the target domain with few labelled samples without losing generalization. This is why we refer to it as a prompt learning method. Kindly note that existing methods in the literature (e.g. KgCoOp and ProGrad) that share similar conceptual frameworks are also considered prompt learning methods.

Reviewer comment (Experiments):
The comparison between CoPrompt and Zero-shot CLIP is not fair enought. The diverse text prompts generated by LLM can imporve the zero-shot classification ability of CLIP on downstream tasks. It is important to consider this aspect when evaluating the performance of Zero-shot CLIP.

Author response:
To fully investigate the answer to this question, we show the results for the following two scenarios: (1) removing the LLM-generated prompts from CoPrompt and (2) adding LLM-generated prompts to CLIP for its zero-shot evaluation.

The result of the first experiment is presented in Table 5(c) (first row), where CoPormpt obtains an accuracy of 80.09\%. While this is a 0.39\% drop from the optimal performance of CoPrompt, it is 8.99\% better than the zero-shot performance of CLIP (80.09\% compared to 71.1\%). We show the result of the second experiment in the Table below. The result from this experiment shows that LLM-generated text prompts do not improve the zero-shot classification ability of CLIP on downstream tasks, but rather drop the performance from 71.70\% to 69.14\%. This can be explained by the fact that using a more complex sentence instead of the simple template decreases the likelihood of the sentence embedding being aligned with the corresponding image, potentially leading to inaccuracies in CLIP's predictions. We have added this discussion to Appendix A.5 (Page 14).


| Prompt | Accuracy | 
| ------ | --------- |
| Hand-designed prompt | 71.10 | 
| LLM-generated prompt | 69.14 |

Reviewer comment (Experiments):
On small scale dataset like Eurosat, the higher value of λ leads to worse performance, accroding to Table 7. However, the analysis regarding this observation is missing from the paper. Can CoPrompt reaches better result on Eurosat if λ=0?

Author response:
Using λ=0 will effectively turn off the consistency constraint, which, according to the ablation study, leads to sub-optimal performance. In the table below (which is an expansion of Table 6), we show the results for all datasets with λ=0 and another smaller value of λ=0.01, which does not improve the performance of any dataset further. Accordingly, we have expanded Table 6 (Page 8) and added these results. 



| Lambda | ImNet | Caltech | Pets  | Cars  | Flowers | Food  | Aircraft | SUN397 | DTD   | EuroSAT | UCF   |
|--------|-------|---------|-------|-------|---------|-------|----------|--------|-------|---------|-------|
| 0.0    | 73.47 | 96.01   | 96.56 | 73.46 | 82.56   | 91.39 | 36.50    | 79.74  | 68.16 | 82.34   | 80.77 |
| 0.01   | 73.71 | 96.11   | 96.64 | 73.66 | 82.72   | 91.60 | 36.79    | 79.83  | 68.27 | 83.48   | 80.91 |
| 0.1    | 73.82 | 96.22   | 96.87 | 74.44 | 84.15   | 91.73 | 37.60    | 79.95  | 67.42 | 85.84   | 81.9  |

Reviewer comment (Experiments):
What if combined consistency constrain learning with other existing methods, like CoCoOP. Can CoPrompt improve the generalization of CoCoOP?

Author response:
Although we did not originally design CoPrompt as an add-on to other prompt tuning methods, the consistency constraint can be included in any method with a similar tuning mechanism. While CoCoOp constitutes a distinct prompt-tuning approach featuring a text branch conditioned on tokens generated by the image branch, the current form of the consistency constraint cannot be applied to it. However, it is feasible to integrate the consistency constraint into the CoOp method. We show the result of this experiment in the following table. Adding the constancy constraint to CoOp results in a considerable improvement in accuracy for novel classes (from 63.22 to 64.87), contributing to an overall increase in the harmonic mean (from 71.66 to 72.75).

| Method   | Base Acc. | Novel Acc | HM |
| --------- | --------- | --------- | --- |
| CoOp      | 82.69  | 63.22 | 71.66 |
| CoOp + Cons. | 82.80 | 64.87 | 72.75 |

We have added this discussion to Appendix A.6 (page 14).

Reviewer comment (Evaluation):
What is the training overhead in terms of time? What is training and test-time inference speed compared with prior methods?

Author response:
CoPrompt requires about 25\% more training time than the previous SOTA MaPLe. For example, one epoch of training on the Flower102 dataset takes 2 minutes and 9 seconds compared to 1 minute and 43 seconds for MaPLe on a single GPU. There is no overhead in the inference phase since the consistency constraint, the main component that adds the overhead, only affects the training process. We have updated the discussion on the parameters and computational complexity in Section 4.6 (Page 9) with the abovementioned information.

Reviewer comment (Novelty):
The idea of adaptors and prompt-tuning already exist in the literature. Merely combining the two ideas seems an incremental work and not novel.

Author response:
While adapters and prompt-tuning already exist in the literature (which we also acknowledged in the paper), combining them poses a considerable challenge that has not yet been solved by any prior works. This is due to the fact that the model can easily overfit the additional learnable parameters, thus losing generalizability. This phenomenon is has been reported by prior works. For example, MaPLe reported the optimal performance by adding prompts to 9 out of 12 layers of the encoder and not all 12. We highlighted the impact of the consistency constraint in our ablation study in Table 4. In the table below, we provide additional experiments emphasizing the difficulty of combining adapters with prompts for existing methods without our proposed consistency constraint.

| Method/ Setup | Accuracy |
| --------------| -------- |
| CoPrompt      | 80.48    |
| CoPrompt w/o Cons. const.| 78.45  | 
| MaPLe         | 78.55  |
| MaPLe (12-layer prompt) + Adapter | 77.61 |

As observed in this table, CoPrompt's performance of 80.48\% drops to 78.45\%  when prompts and adapters are trained without the proposed consistency constraint. Similarly, the optimal performance of MaPLe is 78.55\% , which drops to 77.61\%  when we increase trainable parameters in the form of prompts and adapters. This study signifies the fact that the adapters and prompts can not be combined to improve the overall accuracy without the consistency constraint proposed in our method. We have now added this discussion to Appendix A.4 (Page 14).

Reviewer comment (Novelty):
The idea of retaining the generalizability of the CLIP using consistency loss has already been explored in the paper "Self-regulating Prompts: Foundational Model Adaptation without Forgetting" (ICCV 2023) [1]. Hence, the consistency loss doesn't contribute towards the novelty.

Author response:
Kindly note that we were not aware of this work at the time of submission since this paper was accepted in ICCV 2023, which was around the same time that we were preparing for our submission to ICLR. It is worth noting that our paper had been publicly available on arXiv before PromptSRC. Our paper was posted on arXiv on Thu, 1 Jun 2023, while PromptSRC was posted on Thu, 13 Jul 2023. However, while both PromptSRC and our proposed CoPrompt share a common goal of ensuring consistency, the proposed solutions in the two methods differ significantly. These distinctions are as follows:


- While prompts are the only training parameters in PromptSRC, we introduced the concept of integrating adapters with prompts. It provides additional tunable parameters for adopting CLIP to a new domain, and training under the influence of consistency helps the model to improve generalization without over-fitting the few labelled samples.

- On the language branch, PromptSRC uses a pool of text templates to increase the number of text prompts, which are all hand-designed prompts. In contrast, our proposed solution uses an LLM to generate a more descriptive sentence. From the ablation study on PromptSRC and CoPrompt, we see a larger improvement with our approach. 

- For the consistency constraint, PromptSRC used L1 loss, whereas our CoPrompt utilizes a cosine loss. As opposed to L1 loss, cosine loss captures the angular similarity between vectors rather than relying solely on their magnitudes. Nonetheless, we already explored L1, and also MSE as the consistency loss, which was presented in Table 5(b). The results showed comparatively better performance for cosine loss over L1 or MSE. 

- PromptSRC utilized independent prompts on language and vision branches, while CoPrompt employs a multi-modal prompt with a coupling function between language and vision branches. 

- We now directly compare our results to PromptSRC in the following tables. On the base-to-novel generalization task, our method performs better than PromptSRC in the evaluation of 11 datasets, with an average HM of 80.48 vs.  PromptSRC's 79.97. Similarly, on the cross-dataset evaluation, CoPrompt outperforms PromptSRC by 1.19\%.

Reviewer comment (Experiments):
The authors have not compared their approach to the above paper and there is also no reference to the paper.

Author response:
Kindly note that we were not aware of this work at the time of submission since this paper was accepted in ICCV 2023, which was around the same time that we were preparing for our submission to ICLR. It is worth noting that our paper had been publicly available on arXiv before PromptSRC. Our paper was posted on arXiv on Thu, 1 Jun 2023, while PromptSRC was posted on Thu, 13 Jul 2023. However, while both PromptSRC and our proposed CoPrompt share a common goal of ensuring consistency, the proposed solutions in the two methods differ significantly. These distinctions are as follows:


- While prompts are the only training parameters in PromptSRC, we introduced the concept of integrating adapters with prompts. It provides additional tunable parameters for adopting CLIP to a new domain, and training under the influence of consistency helps the model to improve generalization without over-fitting the few labelled samples.

- On the language branch, PromptSRC uses a pool of text templates to increase the number of text prompts, which are all hand-designed prompts. In contrast, our proposed solution uses an LLM to generate a more descriptive sentence. From the ablation study on PromptSRC and CoPrompt, we see a larger improvement with our approach. 

- For the consistency constraint, PromptSRC used L1 loss, whereas our CoPrompt utilizes a cosine loss. As opposed to L1 loss, cosine loss captures the angular similarity between vectors rather than relying solely on their magnitudes. Nonetheless, we already explored L1, and also MSE as the consistency loss, which was presented in Table 5(b). The results showed comparatively better performance for cosine loss over L1 or MSE. 

- PromptSRC utilized independent prompts on language and vision branches, while CoPrompt employs a multi-modal prompt with a coupling function between language and vision branches. 

- We now directly compare our results to PromptSRC in the following tables. On the base-to-novel generalization task, our method performs better than PromptSRC in the evaluation of 11 datasets, with an average HM of 80.48 vs.  PromptSRC's 79.97. Similarly, on the cross-dataset evaluation, CoPrompt outperforms PromptSRC by 1.19\%.


| Method | Base Acc. | Novel Acc | HM |
| ------ | --------- | --------- | --- |
| CoPrmpt (ours)  | 84.00 | 77.23 | 80.48 |
| PromptSRC | 84.26 | 76.10 | 79.97 |



| Method          | Caltech | Pets  | Cars  | Flowers | Food  | Aircraft | SUN397 | DTD   | EuroSAT | UCF   | Average |
|-----------------|---------|-------|-------|---------|-------|----------|--------|-------|---------|-------|---------|
| CoPrompt        | 94.50   | 90.73 | 65.67 | 72.30   | 86.43 | 24.00    | 67.57  | 47.07 | 51.90   | 69.73 | 67.00   | 
| PromptSRC | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 | 65.81 | 
 
We now discuss the similarity of our method with PromptSRC in the Method section (Page 4, Paragraph 3), and also expand the results in Tables 1, 2 and 3 (Pages 6 and 7) to compare with PromptSRC.

Reviewer comment (Reproducibility):
Are the vision side prompts conditioned on text side? Do authors follow MaPLe settings or Independent VL prompting?

Author response:
For the implementation of the prompts, we followed the same approach as used in MaPLe.

Reviewer comment (Experiments):
It will be good to see the proposed method generalization for a newer V-L model. CLIP is relatively outdated and the authors are encouraged to show result on at least another recent CLIP variant. For example on EVA-CLIP[1] model.

Author response:
> It will be good to see the proposed method generalization for a newer V-L model. For example, on EVA-CLIP [1] model.

In the original submission, we only focused on CLIP since all existing methods we compared had reported their results on the same encoder. As per your suggestion, we have now investigated the performance of the proposed method, as well as the previous SOTA MaPLe, with the EVA-CLIP backbone. The results are presented below: 

| Method   | Backbone      | Base Acc  | Novel Acc | HM    |
|----------|---------------|-----------|-----------|-------|
| MaPLe    | CLIP-B/16     | 82.28     | 75.14     | 78.55 | 
| CoPrompt | CLIP-B/16     | 84.00     | 77.23     | 80.48 | 
| MaPLe    | EVA-CLIP-B/16 | 82.65     | 75.78     | 79.06 |
| CoPrompt | EVA-CLIP-B/16 | 84.29     | 77.50     | 80.75 |

This table demonstrates that CoPrompt also works well on the EVA-CLIP backbone and shows similar improvements over MaPLe. Overall, the EVA-CLIP encoder performs slightly better than the CLIP encoder. We have added this discussion to Appendix A.3 (page 14).

Reviewer comment (Novelty):
There is a recent prompt learning method PromptSRC [2], which also seems to introduce consistency constraints to prompt learning to improve generalization. How is the proposed method different from this work? Also, all fair comparisons should be added in the main paper.

Author response:
> There is a recent prompt learning method PromptSRC [2]. How is the proposed method different from this work? 

Kindly note that we were not aware of this work at the time of submission since this paper was accepted in ICCV 2023, which was around the same time that we were preparing for our submission to ICLR. It is worth noting that our paper had been publicly available on arXiv before PromptSRC. Our paper was posted on arXiv on Thu, 1 Jun 2023, while PromptSRC was posted on Thu, 13 Jul 2023. However, while both PromptSRC and our proposed CoPrompt share a common goal of ensuring consistency, the proposed solutions in the two methods differ significantly. These distinctions are as follows:


- While prompts are the only training parameters in PromptSRC, we introduced the concept of integrating adapters with prompts. It provides additional tunable parameters for adopting CLIP to a new domain, and training under the influence of consistency helps the model to improve generalization without over-fitting the few labelled samples.

- On the language branch, PromptSRC uses a pool of text templates to increase the number of text prompts, which are all hand-designed prompts. In contrast, our proposed solution uses an LLM to generate a more descriptive sentence. From the ablation study on PromptSRC and CoPrompt, we see a larger improvement with our approach. 

- For the consistency constraint, PromptSRC used L1 loss, whereas our CoPrompt utilizes a cosine loss. As opposed to L1 loss, cosine loss captures the angular similarity between vectors rather than relying solely on their magnitudes. Nonetheless, we already explored L1, and also MSE as the consistency loss, which was presented in Table 5(b). The results showed comparatively better performance for cosine loss over L1 or MSE. 

- PromptSRC utilized independent prompts on language and vision branches, while CoPrompt employs a multi-modal prompt with a coupling function between language and vision branches. 

- We now directly compare our results to PromptSRC in the following tables. On the base-to-novel generalization task, our method performs better than PromptSRC in the evaluation of 11 datasets, with an average HM of 80.48 vs.  PromptSRC's 79.97. Similarly, on the cross-dataset evaluation, CoPrompt outperforms PromptSRC by 1.19\%. 


| Method | Base Acc. | Novel Acc | HM |
| ------ | --------- | --------- | --- |
| CoPrmpt (ours)  | 84.00 | 77.23 | 80.48 |
| PromptSRC | 84.26 | 76.10 | 79.97 |



| Method          | Caltech | Pets  | Cars  | Flowers | Food  | Aircraft | SUN397 | DTD   | EuroSAT | UCF   | Average |
|-----------------|---------|-------|-------|---------|-------|----------|--------|-------|---------|-------|---------|
| CoPrompt        | 94.50   | 90.73 | 65.67 | 72.30   | 86.43 | 24.00    | 67.57  | 47.07 | 51.90   | 69.73 | 67.00   | 
| PromptSRC | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 | 65.81 | 

We now discuss the similarity of our method with PromptSRC in the Method section (Page 4, Paragraph 3), and also expand the results in Tables 1, 2 and 3 (Pages 6 and 7) to compare with PromptSRC.

Reviewer comment (Writing):
I think there is some writing logical errors in the paper. For example, in the Adapters heading in section 3, adapter based method are being mentioned but prompts have been written instead of the adapter blocks.

Author response:
> I think there are some writing logical errors in the paper. For example, in the Adapters heading in section 3, adapter based method are being mentioned but prompts have been written instead of the adapter blocks.

We checked the writing of the mentioned section, and are glad to inform you that there are no writing errors. However, we have revised the writing to resolve this confusion. The sentence mentioning 'prompt' referred to an existing work (Khattak et al., 2022) that discussed the fact the additional parameters (whether prompt or adapter) introduces the risk of overfitting. We have now revised the description on Page 5, Paragraph 1 to say: 'For instance, Maple achieved the best performance when adding learnable parameters to only nine out of twelve layers of the CLIP backbone.'