Paper: Prompting Language-Informed Distribution for Compositional Zero-Shot Learning (ICLR-2024)

Reviewer comment (Experiments):
The proposed method relies much on the quality of LLMs, and the transferability of the model is not reflected in the paper.

Author response:
**Rely on the LLM quality.** We would clarify that leveraging LLMs for the CZSL task is one of our proposed ideas in this paper, and it has never been explored in the CZSL literature. Therefore, whether the CZSL performance replies on LLM quality or not is an open research question, and interestingly, we found the performance is not always better if a better LLM is utilized (see Table 3). We believe in the future, there will be more research investigating how to better utilize LLMs to solve the challenging CZSL task.

Reviewer comment (Experiments):
According to the Ablation study, the experiment w/o VLPD does not change much (even the H_cw value decreases).

Author response:
**Impact of VLPD.** The VLPD primarily improves the open-world CZSL performance ($H_{ow}$ and $\text{AUC}_{ow}$), rather than the closed-world ones. This can be observed in Table 2 which indicates significant open-world performance gains. This observation is expected because the VLPD is developed to mitigate the overfitting issue on the visual data of seen compositions. Thus, in a much larger open-world decision space where most compositional class names might be infeasible, a less overfitting model w.r.t. the seen compositions is expected to achieve higher open-world performance.

Reviewer comment (Theory):
The proposed languageinformed distributions (LID) can effectively avoid the issue of intra-class variety. However, the authors would better also intepret how to solve the issue of inter-class correlation.

Author response:
**Inter-class correlation.** We clarify that the proposed LID is simultaneously minimizing the intra-class variance and maximizing the inter-class separability as stated in the Lines below Eq. (1) (the last 3 lines of Page 5). More specifically, the learning goal is to minimize the NLL upper-bound (r.h.s of Eq. (1)). This will encourage minimizing the $h_{k,y}$ term which is determined by the positive intra-class variance terms $\Sigma_{kk}$ and $\Sigma_{yy}$, and also the negative inter-class terms $-\Sigma_{ky}$ and $-\Sigma_{yk}$. It's thus clear that inter-class correlation is also optimized.

Reviewer comment (Experiments):
The gap between considering and not considering distributional-based margins in Eq. (1) and Eq. (4) is mild according to Table 4 (i.e., gap lower than 0.5 points except for harmonic mean OW). Fig. 5.a shows that increasing from 4 to 64 LLM-based sentences improves the overall results by less than 1 point and less in the more challenging OW setting. Given that querying LLM is costly in this setting (i.e., the number C of compositions might be in the order of thousands) and can be noisy (as per Appendix A), it is questionable whether LLMs and distributional semantics are crucial for the approach (as suggested by the title). The ablations are also conducted on the MIT-states dataset, which is known to be noisy (Atzmon et al. 2020), so it should be verified if the findings hold across datasets.

Author response:
- **LLM indeed has a noticeably positive effect while the cost is acceptable.** For the CZSL evaluation especially the very challenging open-world metrics, it is normal that the scales of numbers are very low and even 0.5 points are meaningful in practice (see other methods in Table 1). If removing LLM descriptions (w/o LLM, w/o LID), we present results in Table above (row (1)) which shows the closed-world performance decreases significantly ($-1.12$ points of $AUC_{cw}$). With LLM added but still without LID, the closed-world performance further improved ($+0.67$ points of $AUC_{cw}$). These results demonstrate that LLM and LID could collectively improve the CZSL performance. Regarding the concern on the cost of LLMs, in practice, the compositional descriptions are generated offline by frozen LLM and their features are pre-extracted by frozen CLIP so that the cost for our model training and testing is acceptable.

Reviewer comment (Experiments):
TFE and VFE are modules that refine visual/textual embeddings, but the article does not contain ablations on their number of parameters/design choices, and the improvement from the added views is mild (i.e., less than 0.5 points on Fig. 5b). Ablating variants of these modules, potentially removing the set of text embeddings/views and focusing on their parameters (e.g., even via MLPs, etc.), would strengthen the justification for implementing them as cross-modal blocks and for their specific input choices. This also applies to the specific implementations of the prediction modules \(f_s\) and \(f_o\) in Eq. (2).

Author response:
1. The MLP (x1) row keeps all others the same (uses SLM and VLPD) except for the TVE/VFE module, where we replace the cross-attention with MLP. Though it performs better, the improvement is rather small (less than 0.1 points). This is why we conclude that the type of the structure does not matter. For such structural exploration on the model without SLM and VLPD, we can do that but the inner structure of the module is NOT our claimed contribution and it would take more time at this moment. For your concern about our claimed contribution (b), we already have the variable-controlled experimental results in Table 2 (row (f)(g)) that validate the VLPD and SLM.

Reviewer comment (Experiments):
The SLM module should provide flexibility to the model regarding which predictions to trust. However, SLM is not compared to other aggregation strategies (e.g., average, max, product) and on the single scoring mechanism (compositional vs. primitive-based). Thus, it is hard to assess the need for this module.

Author response:
- **SLM is better than simpler (or without) logit aggregation.** In row (5), we simply use compositional logits without primitive ones, while in row (6), we use simple weighted averaging. We see that neither of these simple methods could achieve better performance and noticeably, only using compositional logits lags far behind our SLM strategy which indicates the importance of primitive-based logits in the CZSL task.

Reviewer comment (Writing):
From the abstract and introduction, it is unclear what the role of LLMs is and how they are used. It is implied that a language-informed class distribution is used, but not how this is achieved (or the information is kept generic in Section 1). While not a major weakness per se, given that the title focuses on this distribution, it would be helpful to give hints on how this distribution is estimated already at the beginning of the manuscript, clarifying the methodological idea to the reader. Fig. 1 serves this purpose well, but the text does not stress it.

Author response:
Thanks for providing the valuable writing suggestions! We will revise the Abstract and Introduction text part to convey clearly the methodological idea. For the notations, we have intended to precisely convey the technical truth of our method through the notations, while not being aware of their overloaded information for readers. Thanks to your suggestion, we will try our best to simplify them. We will also revise the figures in the final version.

Reviewer comment (Writing):
The notation in Section 3 is not straightforward to follow. The main challenge is not a lack of explanations (each term is properly defined) but the number of terms defined that the reader must remember to appreciate the method fully. While it may be difficult to simplify the notation given the multiple components, in some cases, the notation could be made simpler. For instance, the names of the text embeddings are detached from their inputs (e.g., S becomes D, [p:..] becomes q, x and X become v, etc.). The end of the VLPD part introduces \(\mathbf{h}\) elements and \(\mathbf{H}\), which could be replaced by directly stating that \(h^{rc}_y = h_s + h_o\) (even in Eq. (4)). These are arguable and possibly subjective examples of elements that could be undefined and/or simplified.

Author response:
Thanks for providing the valuable writing suggestions! We will revise the Abstract and Introduction text part to convey clearly the methodological idea. For the notations, we have intended to precisely convey the technical truth of our method through the notations, while not being aware of their overloaded information for readers. Thanks to your suggestion, we will try our best to simplify them. We will also revise the figures in the final version.

Reviewer comment (Evaluation):
What is the impact of the LLM and the cost to produce the compositional descriptions?

Author response:
- **LLM indeed has a noticeably positive effect while the cost is acceptable.** For the CZSL evaluation especially the very challenging open-world metrics, it is normal that the scales of numbers are very low and even 0.5 points are meaningful in practice (see other methods in Table 1). If removing LLM descriptions (w/o LLM, w/o LID), we present results in Table above (row (1)) which shows the closed-world performance decreases significantly ($-1.12$ points of $AUC_{cw}$). With LLM added but still without LID, the closed-world performance further improved ($+0.67$ points of $AUC_{cw}$). These results demonstrate that LLM and LID could collectively improve the CZSL performance. Regarding the concern on the cost of LLMs, in practice, the compositional descriptions are generated offline by frozen LLM and their features are pre-extracted by frozen CLIP so that the cost for our model training and testing is acceptable.

Reviewer comment (Experiments):
Do the structure of TFE and VFE matter more or less than the distribution/margins introduced?

Author response:
- **Both the structure of TFE/VFE and the introduced distribution matter.** Comparing the result row (2)(3)(4) with our full model (the last row), it shows that without the distribution modeling by LID (LLM, w/o. LID), the results (row (2)) are worse than models with LID but instead using simple MLP (row (3)), while better than the model with more cross-attention (XAttn$\times 3$) layers (row (4)). This indicates that the number of TFE/VFE layers matters more but the types (attn/mlp) matter less than the LID. Since LID has never been explored in CZSL literature, these findings could inspire future exploration on how to better utilize LLM-based distributional semantics.

Reviewer comment (Experiments):
Is SLM better than simpler aggregation strategies?

Author response:
- **SLM is better than simpler (or without) logit aggregation.** In row (5), we simply use compositional logits without primitive ones, while in row (6), we use simple weighted averaging. We see that neither of these simple methods could achieve better performance and noticeably, only using compositional logits lags far behind our SLM strategy which indicates the importance of primitive-based logits in the CZSL task.

Reviewer comment (Novelty):
The motivation for modeling class distribution is not new, i.e., it has been proposed and used in the vision-language model. Also, the augmentation to image input seems like a test-time adaptation strategy that is also explored in the community. These aspects degrade the contribution to the community.

Author response:
We thank the reviewer for viewing this paper.  
In terms of the novelty of our motivations and methodology, we would like to emphasize the key factors that might be ignored by the reviewer:
- On the general ZSL problem, we summarize the two aspects, i.e., context diversity addressed by modeling class distribution over soft prompts and the informativeness inherited from LLM-based language prompts, are only independently studied in the literature. In this paper, we advocate that they can be naturally bridged together to take the merits of both while overcoming their limitations. This points to a brand-new prompt learning paradigm, that is general to any vision-language foundation models on zero-shot classification problems.  
- Specific to the CZSL task, we are the first to model the class-wise distribution over both the compositional and primitive classes. Besides, our VLPD is different from existing CZSL literature as we decompose both image and text modality rather than simply decomposing text features in the DFSP and HPL (see text below Eq.(2) and Table 5). 

For the technical questions: 
 - Our augmentation to image input is NOT a test-time adaptation, as its relevant module VFE has to be learned in training.

Reviewer comment (Novelty):
The framework incrementally follows ProDA by introducing D^{(y)} and introduces VLPD by compositing v which is also widely used in CZSL. The framework seems like a combination of many existing techniques. This degrades its novelty.

Author response:
We thank the reviewer for viewing this paper.  
In terms of the novelty of our motivations and methodology, we would like to emphasize the key factors that might be ignored by the reviewer:
- On the general ZSL problem, we summarize the two aspects, i.e., context diversity addressed by modeling class distribution over soft prompts and the informativeness inherited from LLM-based language prompts, are only independently studied in the literature. In this paper, we advocate that they can be naturally bridged together to take the merits of both while overcoming their limitations. This points to a brand-new prompt learning paradigm, that is general to any vision-language foundation models on zero-shot classification problems.  
- Specific to the CZSL task, we are the first to model the class-wise distribution over both the compositional and primitive classes. Besides, our VLPD is different from existing CZSL literature as we decompose both image and text modality rather than simply decomposing text features in the DFSP and HPL (see text below Eq.(2) and Table 5).

Reviewer comment (Theory):
Why A in Eq. 1 is defined like that? Is the dimension shape of A correct by defining it as A_k,y? What’s the shape of A_k,y?

Author response:
For the technical questions: 
 - For the Eq. (1), the matrix $A$ is defined to represent pair-wise class correlation on each of $d$ features so that its shape is $d\times C\times C$ where $C$ is the number of classes. Therefore, the $A_{k,y}$ is a $d$-dimensional vector. Note in practice, before the marginal logit $v^\top A_{k,y} {v}$ is computed, $A_{k,y}$ is diagonalized to have the shape $d\times d$. To avoid misunderstanding, we will update the marginal logit as $v^\top \texttt{diag}(A_{k,y}) v$ in the main paper.

Reviewer comment (Experiments):
Some parameter analysis is shown, however, the results w.r.t. the value of N = 0 is not shown.

Author response:
For the technical questions: 
 - As requested, results w.r.t. the value of $N=0$ are shown in the Table below. It clearly shows the effectiveness of the VFE that takes an image with $N$ augmented views as input.

| Models      | $\text{H}_\text{cw}$ | $\text{AUC}_\text{cw}$ | $\text{H}_\text{ow}$  |  $\text{AUC}_\text{ow}$  |
| :---        |    :----:   |       :----:   |       :----:   |   :----: |
| $N=0$ |  37.89 | 21.07|19.37  | 6.78 |
| $N=8$ | $\textbf{38.97}$     | $\textbf{22.12}$       | $\textbf{20.41}$ | $\textbf{7.34}$  |