Paper: Feasibility with Language Models for Open-World Compositional Zero-Shot Learning (ICLR-2024)

Reviewer comment (Experiments):
According to the paper, the In-context Learning seems not to be trained in the process, which means that the performance relies on the LLMs. However, there existing some objects and states that are totally unknown to LLMs. In this situation, the proposed model cannot transfer the knowledge to the unknown compositions.

Author response:
LLMs have a vast vocabulary that includes all practically relevant states and objects considered in CZSL datasets (fine-grained shoe dataset UT-Zappos, common real-world objects in MIT states and C-GQA) and their language understanding is generally more comprehensive compared to GloVe and ConceptNet. Nonetheless, when we test the LLM with unknown words (e.g. made-up), it defaults to considering their pairs as infeasible unless their feasibility is very clear from the in-context example, i.e., it cannot as easily relate objects or states by their semantic similarity.

When new states and objects become available (e.g. neologism or recent product names), any model used for feasibility prediction (LLMs for FLM, GloVe, ConceptNet) would have to be updated with relevant training data. In the case of FLM, we would likely use the most recently available LLM. For anything in-distribution (e.g. real-world states/objects), we find that in-context examples are effective in adjusting the LLM to the specific task without requiring costly re-training.

Reviewer comment (Novelty):
The main concern of this work is its contribution. The paper basically uses the existing LLM to determine the feasibility of a state-object combination. This only shows that the existing LLM is able to determine the feasibility of a state-object combination, but what is the author’s contribution throughout the process?

Author response:
To the best of our knowledge, we are the first to leverage LLMs and in particular their in-context learning capability to enhance the CZSL performance. Though simple, our FLM substantially improves CZSL performance. Moreover, our FLM is an orthogonal work that can be integrated with any CZSL methods to boost performance in the open-world setting. By replacing an important step in the CZSL pipeline, our approach has an impact on both present and future CSZL methods. This approach also has been acknowledged as innovative by reviewer niw2.

Reviewer comment (Evaluation):
Since different threshold will affect the binary classification performance, wouldn’t a metric like ROC curve suits the tasks better?

Author response:
We report AUC of the CZSL performance in Table 1 which represents the ROC curve as we vary the calibration bias. In Table 2, we then report the feasibility accuracies for the fixed threshold that is used in Table 1 to make a direct connection to the downstream task of CZSL.

Reviewer comment (Presentation):
For Figure 2, it seems that both green and red block only show the feasible (s,o) pairs. The author is suggested to show some infeasible (s,o) pairs and the model prediction on those infeasible (s,o) pairs.

Author response:
In the CSP model on MIT-States dataset, our FLM scores “peeled sauce” and “molten milk”, which are infeasible pairs, as -0.46 and –0.30, respectively, while GloVe's scores are 0.1 and 0.18 and ConceptNet's scores are 0.09 and 0.05. However, there are also “failure cases” in predicting an infeasible pair. For example, “mashed tomato” (0.04, 0.28, 0.30 by FLM, GloVe, and Conceptnet, respectively) and “fresh apple” (0.04, 0.02, 0.07), though not included in the closed-world ground-truth pairs, are considered feasible by all methods. One could argue that those are correct predictions, but in the context of the dataset they become confusing pairs for the model as there are no related images in the dataset. It is difficult to find cases where a human would clearly consider a pair to be infeasible, but FLM predicts it to be feasible. Examples that come closest are “cracked shore” or “dented building”. As mentioned in Section 4.3, it is important to note that some realistically feasible pairs are simply not included in the dataset and, thus, we cannot show images for any infeasible pairs from the datasets.

Reviewer comment (Evaluation):
It is challenging to tell whether GloVe or the proposed FLM separates the feasible and infeasible better by only looking at the figures. The author is suggested to show some numerical results to support the claim.

Author response:
As shown by the numerical results in Table 2 that support Figures 2 and 3, our FLM performs the best on either feasible or infeasible accuracies. Considering both metrics together through arithmetic and harmonic means, our method best separates the feasible and infeasible pairs across all datasets.

Reviewer comment (Evaluation):
In the evaluation metric, the author mentioned that the calibration bias is varied. Does it mean that different calibration bias is used for different metric?

Author response:
Yes, we follow Nayak et al. (CSP) in that the calibration bias is different for every metric. The calibration bias, which is subtracted from the CZSL model outputs of the seen classes but not the unseen classes, is varied to plot the ROC curve of the seen class and unseen class accuracies. The seen class accuracy (metric S) corresponds to the minimal bias value, the unseen class accuracy (metric U) aligns with the maximum bias value, and the H metric is when the bias value reaches the best harmonic mean between the seen class and unseen class accuracies. The AUC metric, measuring the area under the ROC curve, remains unaffected by the value of the calibration bias as it takes all possible values into account.

Reviewer comment (Writing):
Typo: such “at” ChatGPT

Author response:
Thank you for pointing this out. We rectified the typo.

Reviewer comment (Novelty):
Is this the first paper to solve the CZSL problem by using the LLMs? If yes, I am curious about the motivation or some motivation experiments to demonstrate the effectiveness of LLMs? If no, I tend to see some differents compared with other published related works.

Author response:
To the best of our knowledge, we are the first to leverage LLMs and in particular their in-context learning (ICL) ability to enhance CZSL. Contextual understanding of LLMs would enhance their ability to predict feasibility, as illustrated by the example of “dark fire” in the Introduction section. This motivated us to use the ICL capability of LLMs rather than a simple prompt, resulting in a substantial improvement in CZSL performance. By replacing an important step in the CZSL pipeline, our approach has an impact on both present and future CSZL methods. This approach also has been acknowledged as innovative by reviewer niw2.

> I tend to see some differents compared with other published related works.

We tackle the issue of feasibility prediction while most other published works focus on changing the model pipeline (model architecture or loss function) to improve the performance. We argue that our FLM is an orthogonal work that can be integrated with any CZSL methods to boost performance in the open-world setting.

Reviewer comment (Novelty):
This method in this paper is not novel and performance improvement depends entirely on the language model. If the language model introduces biases, such as racial discrimination, during training, will this also affect downstream tasks?

Author response:
This is a valid concern. If biased attributes are involved in the composition, the LLMs’ response for feasibility prediction may be affected, leading to a wrong prediction. Understanding and addressing biases in the LLMs could be important for the CZSL downstream task. The general benchmarks in CZSL task, i.e. MIT-States, UT-Zappos, and C-GQA, involve common objects and states (e.g. shoes or animals, colors or conditions) without including any humans. These state-object pairs are not typically considered social biases, such as racial discrimination. Hence, these states might not be biased in the same way as human-related attributes, such that biases in LLMs could have a smaller effect on CZSL downstream tasks. Nonetheless, investigating the effect of biases in LLMs to the CZSL downstream task would be an intriguing open question.

Reviewer comment (Experiments):
Does a more powerful language model perform best in this paper?

Author response:
The general trend of CZSL downstream performance appears to align with the language model’s capabilities. In Figure 4, particularly on MIT-States, we observe that ChatGPT and Claude-2 outperform Vicuna-13B with LLaMA-2 being the last in the binary setup, reflecting a ranking similar to the LLM leaderboard. However, we also observed the exception where GPT-4 did not follow the expected trend in the CZSL downstream task.