Paper: Don't Paint Everyone with the Same Brush: Adaptive Prompt Prototype Learning for Vision-Language Models (ICLR-2024)

Reviewer comment (Novelty):
As stated in the paper, Prototype learning traces its roots to classical models such as K-Nearest Neighbors (Peterson, 2009) and Learning Vector Quantization. Though some new aspects (adaptive attention, decorrelation, etc) are introduced in this paper, the technical novely seems stil limited.

Author response:
**Re W1**. Thank you for your comment regarding the technical novelty of our work in the context of prototype learning. We acknowledge the foundational roots of prototype learning in classical models, but we believe our approach brings significant advancements to this domain.

A key innovation in our work is the integration of Vision-Language Models (VLMs) into prototype learning. This represents a considerable shift from traditional methods that predominantly utilize visual data. By leveraging textual descriptions through VLMs, our model is able to generate prototypes that capture a more diverse and comprehensive representation of each class. This is particularly crucial in scenarios with limited visual data, such as few-shot learning, where traditional prototype models may exhibit biases towards certain styles or attributes due to sample scarcity. Furthermore, the introduction of adaptive attention and decorrelation in our approach addresses key limitations in existing prototype learning methods. Adaptive attention allows our model to dynamically focus on the most relevant aspects of each prototype, enhancing the accuracy and robustness of classification. Prototype decorrelation, on the other hand, ensures diversity among the prototypes, preventing overlap and redundancy that often plague traditional prototype-based methods.

These advancements are not merely incremental; they represent a significant leap in the field of prototype learning. Our method overcomes challenges inherent in few-shot scenarios and extends the capabilities of prototype models to handle a wider variety of classes more effectively.

In summary, the integration of VLMs for prototype generation, coupled with adaptive attention and decorrelation, constitute the core technical novelties of our work. These contributions significantly enhance the versatility and efficacy of prototype learning, marking a notable advancement in the field. We believe that these points, as elaborated in our paper, underscore the novelty and significance of our approach in pushing the boundaries of prototype learning.

Reviewer comment (Novelty):
The paper addresses the adaptive attention of prototypes. This does work but is also somewhat a straightforward point. The paper does not tackle the adaptive attention of words inside a prototype. The importance is verified in the failure case analysis in the experiments.

Author response:
**Re W2**. Thank you for your insightful suggestion regarding the exploration of word-level adaptive attention within prototypes. While this is indeed an intriguing aspect, our current approach using CLIP presents certain limitations in this regard. Specifically, the visual-language interaction in CLIP occurs at the embedding space level, which restricts our ability to dissect and analyze the semantic meanings of individual words within the prompts.

Furthermore, in our approach, we do not constrain the length of the prompts to preserve their generative nature. This variability in prompt length poses additional challenges in consistently analyzing word-level attention across different prototypes. Consequently, our research primarily focused on adaptive attention at the prototype level. This decision was guided by the objective to enhance the overall classification performance and robustness by dynamically focusing on the most relevant prototypes for each class.

However, we recognize the potential value of investigating word-level attention and its impact on model performance. In future research, we aim to explore methodologies that could allow for more granular attention analysis, perhaps by employing different models or techniques that can facilitate word-level semantic grounding in the context of visual-language tasks. We believe that such explorations could uncover new insights into the intricate interplay between textual and visual elements in prototype-based learning, further advancing the field.

Reviewer comment (Reproducibility):
What's the learnable part of prompt prototypes in Figure 2?

Author response:
**Re Q1**. Thank you for your query regarding the learnable part of prompt prototypes in our model, as depicted in Figure 2. The learnable aspect primarily consists of textual features. For instance, in the ViT-B/16 configuration, these textual features have a dimension of 512. The dimension of the entire learnable weights for prompt prototypes is calculated as the product of the class number, the number of prototypes per class, and 512. These textual features are updated and refined during the training process, dynamically adjusted to better capture the nuances of each class.

Reviewer comment (Experiments):
The experiments follow the base-to-new setting in CoCoOp. However, the base-to-new setting is more about generalization ability. Besides, the performance of the base-to-new setting is very sensitive to the hyperparameters, especially for epochs. It is because the performance of this setting requires a balance between alignment and generalization, which can be achieved by reducing the epochs. When tuning the training epochs of CoOp, it will also achieve good performance. It is suggested to use the few-shot learning setting in CLIP and CoOp, which is more fair and supportive to demonstrate the effectiveness of the proposed methods.

Author response:
**Re W1**. Thank you for your valuable suggestion regarding the incorporation of few-shot learning settings. We have indeed conducted experiments in this setting and reported the performance in our response to W2.

However, our primary focus on the base-to-new setting stems from our objective to critically evaluate the generalization ability of our methods, particularly in scenarios involving new classes. This setting is essential for understanding how well a model can adapt to novel categories, a key aspect of our research. While methods CoCoOp and MaPLe claim improvements in generalization on new classes, our findings indicate that with descriptive prompts, CLIP can already demonstrate superior performance on new classes than these context optimization methods. We provide a detailed discussion on this validation in our paper.

Furthermore, our analysis specifically addresses the issue of overfitting towards base classes and explores the potential of fine-tuning prototypes for both base and new classes with multiple prompts. This approach is aimed at enhancing generalization in new classes.

Regarding the impact of training epochs on CoOp, our experiments reveal significant insights. As shown in the tables for datasets like DTD, Flowers, and UCF, reducing the number of training epochs does not necessarily improve performance on new classes. In fact, we observe a consistent drop in new class performance with CoOp, highlighting a potential limitation of this method in achieving a balanced performance.

These findings are crucial as they demonstrate the comparative advantages of our approach, particularly in maintaining a balance between alignment and generalization across different class settings. We believe that these insights add significant value to the field and underscore the effectiveness of our methods.

**DTD**
| Methods | CLIP |CoOp200|CoOp150|CoOp100|CoOp50 | Co-CoOp| APPLe |
|:--------|:----:|:-----:|:-----:|:-----:|:-----:|:------:|:-----:|
|  Base   | 53.24| 79.44 | 79.25 | 78.82 | 79.98 | 77.01  | 82.41 |
|  New    | 59.90| 41.18 | 36.15 | 35.39 | 36.67 | 56.00  | 69.57 |
|  H      | 56.37| 54.24 | 49.65 | 48.85 | 50.28 | 64.85  | 75.45 |

**Flowers**
| Methods | CLIP |CoOp200|CoOp150|CoOp100|CoOp50 | Co-CoOp| APPLe |
|:--------|:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|
|  Base   | 72.08| 97.60 | 97.53 | 97.60 | 97.47 | 94.87 | 96.58  |
|  New    | 77.80| 59.67 | 58.59 | 60.17 | 61.72 | 71.75 | 78.58  |
|  H      | 74.83| 74.06 | 73.20 | 74.44 | 75.58 | 81.71 | 86.66  |

**UCF**
| Methods | CLIP |CoOp200|CoOp150|CoOp100|CoOp50 | Co-CoOp| APPLe |
|:--------|:----:|:-----:|:-----:|:-----:|:-----:|:------:|:-----:|
|  Base   | 72.08| 84.69 | 84.38 | 84.07 | 84.80 | 82.33 | 86.56  |
|  New    | 77.80| 56.05 | 50.82 | 55.22 | 58.36 | 73.45 | 81.99  |
|  H      | 74.83| 67.46 | 63.43 | 66.66 | 69.14 | 77.64 | 84.21  |

Reviewer comment (Experiments):
There are a series of methods for the class-wise LLM-generated prompts, such as [1-2]. It is suggested to add some discussions and comparisons with these methods. 
[1] Menon, Sachit, and Carl Vondrick. "Visual Classification via Description from Large Language Models." ICLR 2023.  
[2] Pratt S, Covert I, Liu R, et al. What does a platypus look like? generating customized prompts for zero-shot image classification. ICCV 2023.

Author response:
**Re W4**.  Thank you for directing our attention to the papers [1-2], which also explore the use of GPT-generated prompts for visual classification. We will include the two papers in the related work section and provide a comprehensive discussion and comparison with them. While there are similarities in leveraging GPT for prompt generation, our approach diverges in several key aspects.

Firstly, unlike the idea of leveraging the prompts for zero-shot CLIP inference in [1-2], we further propose an adaptive approach (APPLe) to fine-tune these prompts and provide the in-context ability for downstream tasks. 

Secondly, our approach in generating and utilizing prompts is distinct from [1]. [1] focuses on understanding an object from a compositional view. For example, the prompts for Cheeseburger in [1] are the attributes: "a burger patty", "cheese", "a bun", "lettuce", "tomato", "onion". In contrast, in our method, each prompt provides a different view/prototype of an object. For example, "A cheeseburger consists of a burger patty with cheese on top, placed between a hamburger bun". We believe that a comprehensive description of an object can help to match the image and prompts. 

Thirdly, we provide more insights into the descriptive prompts. As shown in the Experiment section, we show adaptive attention visualization, which helps us to understand which prompts are more important or noisy. We have prototype calibration, which balances the contribution of overall prompts and the most representative prompt. We also provide retrieval results to demonstrate the failure case and make abundant explanations. 

Fourthly, we compare our method with context optimization methods. One of our contributions is the observation that with GPT-3 prompts, CLIP performance is already better than the state-of-the-art context optimization method MaPLe. 

Lastly, in terms of performance, we have provided a performance comparison to [1,2] on ImageNet in the table below. The results show the superiority of our proposed method APPLe. Note that we use the same backbone ViT-L/14 for the comparison. 

|  [1]  |   [2]  |  ours | 
|:------|:------:|:-----:|
| 75.00 |  76.69 | 79.59 |

In conclusion, while there are thematic overlaps with the cited papers, our approach introduces an adaptive prompt prototype learning method for better leveraging GPT-3 prompts for VLMs, adding valuable insights and alternatives to the current body of research on visual classification using language models.

Reviewer comment (Presentation):
There is some confusion in the layout of tables and images.

Author response:
Thank you for pointing out the layout concerns. We understand that clarity in presentation is crucial for comprehensibility. We will thoroughly review our document to improve the placement and clarity of tables and images, ensuring they align intuitively with the corresponding text sections. We would also appreciate specific instances from the reviewer where the confusion is more pronounced, to better address these areas.

Reviewer comment (Theory):
Although using multiple prompts as category prototypes can help capture visual differences, in practice, not every visual sample closely matches each prototype.

Author response:
Thank you for the comment. We recognize that not every visual sample will closely match each of our multiple prototypes. However, this is an integral aspect of our design. The use of multiple prompts as category prototypes is intended to capture the broad spectrum of visual variations within a class.

In our method, the matching mechanism is designed to be flexible and robust. It assesses the affinity of a visual sample to the range of prototypes, rather than seeking a one-to-one precise match. This approach ensures that our model can generalize well across diverse samples within a class, even when direct matches to specific prototypes are not feasible.

Moreover, we have implemented strategies to further refine this matching process. These include the decorrelation loss, max/mean logits balancing that assesses the degree of similarity between samples and prototypes, ensuring that each sample is associated with the most representative prototypes for its class.

Reviewer comment (Reproducibility):
The article mentions the introduction of prototype decorrelation loss to suppress the co-occurrence of multiple confident prototypes. However, specific details on how the loss was designed and worked were not mentioned. This may affect the performance of the model in tasks with complex category distributions or a large number of categories.

Author response:
Thank you for bringing up the important aspect of prototype decorrelation loss. We will provide a detailed explanation of this concept and its role in our model in our revised paper. The prototype decorrelation loss is designed to reduce the co-occurrence of multiple confident prototypes, thereby ensuring that each prototype distinctly represents different aspects or variations of the categories.

This loss functions somewhat analogously to weight decay, but with a specific focus on prototype vectors. It operates by penalizing the model when prototypes for a category become too similar or correlated, encouraging a diverse and distinct representation for each prototype. This is crucial in scenarios with complex category distributions or a large number of categories, where the risk of prototype overlap is higher.

The calculation of this loss involves measuring the similarity between different prototypes within the same category and applying a penalty when this similarity exceeds a certain threshold. In our experiments, we observed that incorporating prototype decorrelation loss significantly improved the model's performance, particularly in tasks with intricate category distributions. For instance, in datasets with a large number of overlapping or closely related categories, this loss helped in clearly distinguishing between subtle variations, leading to more precise classifications.

Reviewer comment (Reproducibility):
It is not clear how to initialize these prototypes and how to obtain the base and novel class prompts.

Author response:
Thank you for your inquiry regarding the initialization of prototypes and the generation of prompts for base and novel classes. The prototypes in our model are initialized using textual features extracted by the CLIP text encoder. Specifically, we input descriptive prompts related to each class into the encoder. The encoder then translates these textual descriptions into feature vectors, which serve as the initial prototypes for each class. As for generating the prompts for both base and novel classes, we employ the pre-trained large language model GPT-3. The process involves feeding class-related keywords or descriptions into GPT-3, which then generates a series of prompts. 

These processes are detailed in Section 4, 'Prompt Construction', of our manuscript. In the revised version of the paper, we will provide more detailed examples and explanations of both prototype initialization and prompt generation to ensure that the methodology is clear and easily understandable.

Reviewer comment (Experiments):
The authors keep claiming that CLIP only uses one prompt, but in CLIP paper section 3.1.4, they discuss how they use 80 prompts to improve the performance without sacrificing test time speed (unlike APPLe which is slower with more prompts). The authors should definitely compare their method to CLIP with 80 prompts as a baseline.

Author response:
**Re W1**. We thank the reviewer for pointing out the usage of 80 prompts in the CLIP model. It is true that CLIP's performance on ImageNet benefits from these prompts. However, our work focuses on the limitations of these generic prompts, particularly for fine-grained datasets. These prompts, as limited in the official CLIP repository[1], are less effective for such datasets due to their lack of specificity. For example, prompts like  'a plastic {}' or 'a {} in a video game' do not capture fine-grained distinctions well. It is worth noting that the CLIP performance reported in our paper uses the customized prompts as indicated in the CLIP paper, e.g., “A photo of a {label}, a type of pet.” for OxfortPets.

We acknowledge that these context prompts can indeed improve the performance. We have evaluated the base-to-new setting on ImageNet in the table below (CLIP-80-emb). As for the embedding ensembling method that is raised in question 7. We have tested the performance between embedding ensembling and logits ensembling methods. It can be seen that without fine-tuning, the performance results between the two methods are relatively similar. However, if we want to further fine-tune the textual features, embedding ensembling method tends to overfit to base classes. This phenomenon has been discussed in Figure 3 (impact of prototype number). 

As for the test time speed, logits ensembling indeed causes more computation overhead, but only for dot product between the visual and textual features. Because the textual features of the 80 generic prompts also need to be inferred through the language transformer 80 times. The dot product between visual and textual features needs relatively less computational resources. 

[1] https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb

| Methods| CLIP  |CLIP-80-emb| CLIP-80-logits | APPLe*-50-emb | APPLe*-50-logits  |  APPLe-50-emb | APPLe-50-logits  | 
|:-------|:-----:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|
| Base   | 72.43 | 73.56  | 73.54 | 74.62 | 74.62 | 74.16 | 78.17 |
| New    | 68.14 | 69.97  | 69.99 | 71.79 | 71.94 | 71.93 | 72.12 |
| HM     | 70.22 | 71.72  | 71.72 | 73.17 | 73.26 | 73.02 | 75.02 |

Reviewer comment (Writing):
It needs to be clarified how training-free works. I think the authors should more explicitly describe it. My understanding is that training-free = 50 prototypes only (the second row in Table 4). Correct me if I am wrong.

Author response:
**Re W2**. Thank you for your valuable feedback regarding the clarity of our training-free approach and the overall training process. We agree that these aspects require more explicit descriptions and have revised our paper accordingly.

The training-free version indeed utilizes only 50 prototypes. These prototypes are initialized by the CLIP text encoder using specifically designed prompts that correspond to the categories or classes in our dataset. This initialization is critical as it provides a foundation for our zero-shot classification approach. This training-free method contrasts with our trained approach, where after the initial prototype initialization, we further fine-tune the prototype features. The fine-tuning process involves adjusting these features to better align with the specific nuances of our dataset, thereby improving the classification performance.

To address the reviewer’s concern about the vagueness in our training description, we have expanded Section 4 to include a detailed account of this process. This includes how the prototypes are initialized, the role of the text encoder and prompts in this initialization, and the subsequent fine-tuning steps, if applicable. Specifically, we explain how the text encoder and prompts serve as a starting point, and how image samples from our dataset are used to refine the prototypes for improved accuracy and applicability.

Reviewer comment (Experiments):
The authors primarily experimented with one CLIP model. It is unclear if this method can work with different CLIP variants, open-sourced CLIP replication or other VLM models. I'm curious if changing the model architecture, training data, or VLM format would yield different results.

Author response:
**Re Q1**. Thank you for highlighting the importance of testing our method's generalizability across different CLIP variants and other VLMs. We recognize the concern regarding potential overfitting to a specific model or dataset. To address this, we conducted extensive experiments with various CLIP models and other VLMs, as detailed in the revised table below.

| Variants | Set | CLIP  |CLIP-80| APPLe* | APPLe | 
|:---------|:----|:-----:|:-----:|:-------:|:-----:|
| ViT-B/16 | Base| 72.43 | 73.56 | 74.62  | 78.17 | 
|          | New | 68.14 | 69.97 | 71.94  | 72.12 | 
|          | HM  | 70.22 | 71.72 | 73.26  | 75.02 | 
| ViT-B/32 | Base| 67.43 | 67.52 | 69.40  | 72.66 |
|          | New | 64.04 | 65.84 | 67.78  | 67.83 | 
|          | HM  | 65.69 | 66.67 | 68.58  | 70.16 | 
| ViT-L/14 | Base| 79.20 | 79.96 | 81.03  | 83.51 |
|          | New | 74.02 | 76.43 | 78.06  | 78.13 | 
|          | HM  | 76.52 | 78.16 | 79.52  | 80.73 | 
|ViT-L/14@336|Base|80.25 | 81.04 | 82.00  | 84.26 |
|          | New | 75.50 | 77.60 | 79.08  | 79.15 | 
|          | HM  | 76.52 | 78.16 | 80.51  | 81.63 | 
|LAION ViT-B-32|Base|70.01| 70.07| 70.53  | 73.45 |
|          | New | 69.06 | 69.66 | 70.13  | 70.22 | 
|          | HM  | 69.53 | 69.86 | 70.33  | 71.80 | 
|BLIP ViT-B| Base| 43.63 | 50.10 | 54.40  | 67.56 |
|          | New | 48.42 | 59.16 | 63.38  | 66.28 |
|          | HM  | 45.90 | 54.25 | 58.55  | 66.91 | 

Our method's consistent performance improvement across different architectures, including ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/16@336, LAION's CLIP replication [1], and the BLIP model [2], attests to its robustness. These results underscore our method's adaptability to different model architectures and training datasets. 

[1] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.
[2] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Experiments):
While the method appears to be general, there is a concern about it potentially "overfitting" to a specific model and dataset.

Author response:
**Re Q1**. Thank you for highlighting the importance of testing our method's generalizability across different CLIP variants and other VLMs. We recognize the concern regarding potential overfitting to a specific model or dataset. To address this, we conducted extensive experiments with various CLIP models and other VLMs, as detailed in the revised table below.

| Variants | Set | CLIP  |CLIP-80| APPLe* | APPLe | 
|:---------|:----|:-----:|:-----:|:-------:|:-----:|
| ViT-B/16 | Base| 72.43 | 73.56 | 74.62  | 78.17 | 
|          | New | 68.14 | 69.97 | 71.94  | 72.12 | 
|          | HM  | 70.22 | 71.72 | 73.26  | 75.02 | 
| ViT-B/32 | Base| 67.43 | 67.52 | 69.40  | 72.66 |
|          | New | 64.04 | 65.84 | 67.78  | 67.83 | 
|          | HM  | 65.69 | 66.67 | 68.58  | 70.16 | 
| ViT-L/14 | Base| 79.20 | 79.96 | 81.03  | 83.51 |
|          | New | 74.02 | 76.43 | 78.06  | 78.13 | 
|          | HM  | 76.52 | 78.16 | 79.52  | 80.73 | 
|ViT-L/14@336|Base|80.25 | 81.04 | 82.00  | 84.26 |
|          | New | 75.50 | 77.60 | 79.08  | 79.15 | 
|          | HM  | 76.52 | 78.16 | 80.51  | 81.63 | 
|LAION ViT-B-32|Base|70.01| 70.07| 70.53  | 73.45 |
|          | New | 69.06 | 69.66 | 70.13  | 70.22 | 
|          | HM  | 69.53 | 69.86 | 70.33  | 71.80 | 
|BLIP ViT-B| Base| 43.63 | 50.10 | 54.40  | 67.56 |
|          | New | 48.42 | 59.16 | 63.38  | 66.28 |
|          | HM  | 45.90 | 54.25 | 58.55  | 66.91 | 

Our method's consistent performance improvement across different architectures, including ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/16@336, LAION's CLIP replication [1], and the BLIP model [2], attests to its robustness. These results underscore our method's adaptability to different model architectures and training datasets. 

[1] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.
[2] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Reproducibility):
How does the training process for cross-dataset transfer work? When training on the source data (e.g., ImageNet), the model learns prototype features for ImageNet classes and adaptive attention weights for them. How does this transfer to target datasets where prototypes and attention weights remain untouched during fine-tuning?

Author response:
**Re Q2**. In response to your question about the training process for cross-dataset transfer, let me provide a more detailed explanation. When fine-tuning on base classes from the source dataset ImageNet, we include prototypes for both base and new classes. This inclusion allows the model to learn and optimize features that are not only specific to the base classes but also relevant to the new classes in the target dataset.

The fine-tuning process involves optimizing the similarities between training samples and target datasets, particularly through the use of loss functions $L_{max}$ and $L_{dec}$. $L_{max}$ focuses on maximizing the relevant features for each class, while $L_{dec}$ works on decreasing the influence of irrelevant features. This dual approach ensures that the model not only retains the essential characteristics learned from the base classes but also adapts effectively to the nuances of the new classes in the target dataset.

For example, when transferring from ImageNet to a more specialized dataset like OxfordPets, our model uses the learned prototypes and attention weights as a starting point. It then fine-tunes these components, guided by $L_{max}$ and $L_{dec}$, to better represent and distinguish the unique features of pet classes.

Reviewer comment (Experiments):
Could you clarify the importance of the quality of prompts used in the experiments? What would happen if we used GPT-4 to generate the prompts? How does the quality of the input prompt to the GPT model impact the final performance?

Author response:
**Re Q3**. Thank you for your concern about the impact of prompt quality on our model's performance. The quality of prompts is a critical factor in our experiments, as they directly influence the model's ability to accurately interpret and classify images based on textual descriptions.

In our experiments, we observed that the performance with prompts generated by GPT-3 is relatively consistent with those generated by GPT-4. This could be attributed to the fact that generating category descriptions for our tasks may not require advanced reasoning capabilities, a domain where GPT-4 has more significant improvements over GPT-3. Hence, GPT-3's capacity appears to be sufficient for this specific task.

Regarding the column labeled 'mixed1' in our table, it represents a mix of prompts generated by both GPT-3 and GPT-4. We included this to examine the impact of using a heterogeneous set of prompts on model performance. The results indicate that there is not a significant deviation in performance when using mixed-quality prompts compared to those generated solely by GPT-3 or GPT-4.

We acknowledge the limitations in our current testing due to the time constraints of the rebuttal period. In the next version of our paper, we plan to conduct more comprehensive testing across all 11 datasets to demonstrate the consistency of performance regardless of the prompt generation source. This will provide a more complete picture of how different qualities of prompts impact the overall effectiveness of our model.


| Datasets | Set | GPT-3 | GPT-4 | mixed1| 
|:---------|:----|:-----:|:-----:|:-----:|
| DTD      | Base| 82.41 | 83.10 | 81.95 | 
|          | New | 69.57 | 70.41 | 70.72 | 
|          | HM  | 75.45 | 76.23 | 75.92 |
|OxfordPets| Base| 95.64 | 95.43 | 95.69 |
|          | New | 98.04 | 97.93 | 98.21 |
|          | HM  | 96.83 | 96.66 | 96.93 |
| Aircraft | Base| 44.66 | 45.14 | 45.20 |
|          | New | 43.13 | 39.41 | 41.81 |
|          | HM  | 43.88 | 42.08 | 43.44 |

Reviewer comment (Theory):
In Equation 7, the authors selected a method that can balance between all prototypes and the closest prototypes. Are there other balancing methods, such as using the Boltzmann operator or logsumexp, that could be considered?

Author response:
**Re Q5**. Thank you for suggesting the exploration of alternative balancing methods like the Boltzmann operator and logsumexp in Equation 7. We conducted additional experiments on ImageNet using these methods and present the results in the revised table below. We tested different Boltzmann temperatures (T=20, 10, 5) to understand their impact on performance. Our findings show that the mean/max balancing method we originally used outperforms both the Boltzmann operator and logsumexp. We hypothesize that this superiority is attributed to the specific way mean/max balancing integrates information from all prototypes while emphasizing the most relevant ones, which seems particularly effective for our model and dataset.

Regarding the Boltzmann operator, we observed that varying the temperature has a noticeable impact on performance. Lower temperatures (T=5) led to results closer to our method. However, none of the temperatures tested could match the performance achieved by our mean/max balancing. The performance with logsumexp, especially in the 'Base' class, was notably lower. We will include these findings in our revised paper, along with a comprehensive discussion of these observations.

| Methods| Mean/Max Balancing| Boltzmann T=20 | T = 10 |  T = 5 | Logsumexp |
|:-------|:-----:|:------:|:------:|:------:|:------:|
| Base   | 78.17 | 76.96 | 77.35  | 77.30 | 70.48 |
| New    | 72.12 | 71.96 | 71.96  | 72.07 | 71.64 |
| HM     | 75.02 | 74.27 | 74.56  | 74.59 | 73.73 |

Reviewer comment (Experiments):
While the authors aim for diverse prompts, it might be interesting to fix the prototype and only train the attention weights and forcing the attention weights to be a multinomial distribution with low entropy. This would be essentially learning to select the best prototype. It would be interesting to see if GPT-3 can produce a better single prompt than the hand-designed prompts used in CLIP.

Author response:
**Re Q6**. Thank you for the intriguing suggestion to experiment with training attention weights only and using low entropy to essentially select the best prototype. We conducted this experiment on ImageNet and compared the performance with CLIP’s hand-designed prompts and our APPLe* model.

As shown in the table below, the performance using the best prototype selected by the model was inferior to that of the hand-designed prompts used in CLIP. This outcome suggests that relying on a single prototype, even if it's the 'best' as determined by the model, may not effectively capture the diverse and complex nature of objects in images. Our results indicate that a single prototype is often biased towards a specific representation of an object, which limits its generalizability.

| Methods| CLIP | CLIP-closest prompt | APPLe* | 
|:-------|:-----:|:------:|:------:|
| Base   | 72.43 | 71.84 | 74.62  |
| New    | 68.14 | 67.90 | 71.94  | 
| HM     | 70.22 | 69.81 | 73.26  |

For example, as highlighted in Figure 1 of our paper, each prompt for an apple pie depicts a particular state or aspect of the pie, such as a round pie or a slice on a plate. While each prompt is accurate in its description, none can encompass all the variations of apple pies alone. This specificity is where the limitation lies. It becomes challenging for a single, even well-crafted prototype to represent the breadth of variations that an object can have in the real world.

Reviewer comment (Theory):
In Equation 7, stating the "average cosine similarity" is not entirely accurate because the cosine similarities are weighted by the attention weights.

Author response:
**Re Q8**. Thank you for pointing out the need for more precise terminology in describing our model's mechanics. We agree that the term 'average cosine similarity' in Equation 7 could potentially be misleading, as it does not fully capture the role of attention weights in the calculation. Therefore, we have revised our paper to use 'weighted average cosine similarity' in the relevant sections.

Reviewer comment (Experiments):
While the trend in Figure 5 is clear, it could be valuable to include settings with 0/1 and 1/0 to further illustrate the findings.

Author response:
**Re Q9**. We appreciate your suggestion to include settings with 0/1 and 1/0 ratios in Figure 5 to provide a more comprehensive understanding of our findings. We have revised Figure 5 accordingly, and the additional results are as follows:

| Ratio | Base  |  New  |   HM  |  
|:------|:------|:-----:|:-----:|
| 0/1   | 70.17 | 67.74 | 68.93 |
| 1/0   | 76.65 | 71.86 | 74.18 |