Paper: Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR-2024)

Reviewer comment (Experiments):
The experiments only include one type of VLM, i.e., CLIP. The reviewer would like to know whether FuDD is compatible with more advanced VLMs.

Author response:
Thank you for your suggestion. To further evaluate FuDD, we run additional experiments with OpenCLIP, which is much stronger than CLIP, and BLIP2, which uses an entirely different training strategy. We observe similar performance improvement in this new set of experiments, which further supports the argument that FuDD benefits a wide range of VLMs.  
Please see our general comment to all reviewers for the results and further discussion.

Reviewer comment (Evaluation):
There is no analysis on the trade-off between performance gain by FuDD and additional (computational or financial) cost. Since generating pair-wise class description is more expensive than generating independent description for each class, such analysis is important.

Author response:
The number of LLM queries directly depends on the number of ambiguous classes, which is a hyperparameter set by the user. In Figure 3, we report the performance gains with different numbers of ambiguous classes (i.e., different values for k). As discussed in Section 4.1, the five most ambiguous classes account for most of FuDD’s performance gains. As a result, to further decrease the costs, one can use smaller values for k, while preserving most of FuDD’s benefits.  
Please also see our general comment to all reviewers for exact API costs and further discussion.

Reviewer comment (Theory):
The section 5: I wonder why would the Llama2 can generate informative sentences for satellite images? Especially given the model only been finetuned on the ImageNet dataset sentences.

Author response:
We believe that during fine-tuning, the LLM learns what types of features are useful for differentiating the target classes, while the world knowledge comes from the pre-training stage. As shown in Figure 4 and Figure 5, after fine-tuning, the LLM is describing more low-level visual features instead of higher-level semantic concepts. As another piece of evidence in support of this argument, in Table 3, we observe that the correctness of the descriptions increases moderately (i.e., fine-tuning has limited impact on LLM’s world knowledge), while their helpfulness (i.e., being differentiating) increases significantly, which we attribute the increased performance to.

Reviewer comment (Experiments):
What if the caption is so long and so complex that the image-text model could not understand? I wonder if the author encountered this scenario, if so, how to address it?

Author response:
That was exactly the case in our early experiments, where we got long, complex descriptions, sometimes expanding an entire paragraph. To alleviate this problem, we take advantage of the in-context learning capabilities of LLMs, and use two examples as part of our LLM prompt to guide the model to focus on concise visual features and follow the description style that is known to work well for VLMs, i.e., short photo captions that start with prefixes like "a photo of a."  

Inspecting the generated descriptions across all datasets, we find that, on average, each description contains 14 words and 99 percent of the descriptions start with simple prefixes like:  
"a photo of"  
"a photograph of"  
"a video of"  
"a close-up of"  
"a screenshot of"  
"a close-up photo of"  
"a microscopic image of"  

Excluding the prefix and the class name, there are approximately 10 remaining words in each description, which does not allow for much complexity in descriptions.

Moreover, to keep the descriptions simple but at the same time benefit from a diverse set of described attributes, we guide the model in both the task instruction and the in-context examples to generate multiple descriptions, and then we average over the generated descriptions to compute the class representations. On average, over all datasets, we use approximately 4 descriptions per class.

Reviewer comment (Novelty):
The method is incremental, as it is a small trick to boost zero-shot image classification results.

Author response:
We believe that our work contributes significantly to the field, especially as the need for efficient approaches to adapting VLMs to downstream tasks increases. We propose technical contributions that outperform previous methods, provide analysis of characteristics of effective natural language descriptions, study the utility of open-source LLMs and their fine-tuning,  and contribute to answering open questions in the field on the potential of natural language class descriptions for VLM adaptation compared to other sources of information (e.g., synthetic images) or alternative approaches like soft prompt tuning.

Please see our general comment to all reviewers for an extended discussion on our contributions.

Reviewer comment (Experiments):
The baseline is not strong enough. It is therefore not clear if the method can still improve state-of-the-art models on these tasks.

Author response:
To address the reviewer’s concern, we run additional experiments on two OpenCLIP model variants, which are stronger than their CLIP counterpart, as well as the BLIP2 model, which uses a different style of pre-training than CLIP. In both cases, we observe similar improvements, where the additional information provided by FuDD improves the performance over the naive LLM-generated descriptions. 

Please see our general comment to all reviewers for additional experiments and further discussion.

Reviewer comment (Experiments):
More diverse and challenging tasks are missing from the experiment, e.g. image-text retrieval. It will be nice to show wider applicability of the method.

Author response:
There are many interesting vision and language tasks that could benefit from additional semantic information. However, in keeping with most of the closely related work, here, we also focus on image classification, which remains a challenging and important problem (please refer to the manuscript for an extended discussion of previous work). Focusing on image classification allows more diverse comparisons and insights in this line of work, without confounding factors like task variation. For example, the comparison of image classification results in Section 6 uncovers the potential of natural language descriptions for VLM adaptation compared to other modalities or few-shot approaches proposed by previous work.

Although outside of the focus of our work, we would like to discuss FuDD in the context of information retrieval (IR). We draw an analogy between FuDD and reranking methods for IR, where for each query, a low-cost approach selects a subset of potentially related documents (similar to Section 3.2 in FuDD), and a more complex method reranks this smaller subset of documents (similar to Section 3.3 and 3.4 in FuDD). In the context of IR, because of the large number of documents, it is financially and computationally impossible to generate alternative descriptions for all documents regardless of choosing naive or differential descriptions. In such a setting, focusing on ambiguous documents is the only feasible option (we have extended discussion and experiments on ambiguous classes in Sections 3.2 and 4.1).
Now the question is, "How effective are naive descriptions for reranking compared to FuDD?." Constrained by the rebuttal time window, we design new experiments to simulate text retrieval with our current descriptions and datasets. In these new experiments, we use test images as equivalent for queries and classes as the documents to be retrieved in a text-retrieval setup.
We first retrieve the ambiguous classes as described in Section 3.2 and then use differential descriptions and naive descriptions to rerank these ambiguous classes and compare the results. We report the accuracy and nDCG@10, which represents the improvement in the ranking of the documents (i.e., class names). Compared to classification, it shows how much closer the true class label is to the top of the list, even if it is not the top prediction.


|-|Flowers102|Flowers102|Places365|Places365|
|---|---|---|---|---|
|-|Acc|nDCG@10|ACC|nDCG@10|
|Naive|75.98|85.21|41.58|59.37|
|FuDD|78.76|86.35|43.95|60.89|


As expected, we observe that better representations for each class coming from differential descriptions actually improve the performance in this case, where both naive and differential descriptions are used for reranking.

Reviewer comment (Novelty):
The idea of prompting LLM to give more differentiating descriptions is effective but not necessarily a significant innovation.

Author response:
We respectfully disagree with the reviewer as our work makes technical contributions, achieves superior results, provides analysis on characteristics of effective natural language descriptions, studies the use of off-the-shelf and fine-tuned LLMs in the process, and positions natural language descriptions as a promising approach for adapting VLMs. This is especially important for future work as it advances the debate over promising approaches for VLM adaptation in favor of natural language descriptions compared to other sources of information like synthetic images or alternative approaches like soft prompt tuning.  
Please find our extended response in the general comment to all reviewers for further discussion.

Reviewer comment (Evaluation):
The paper focuses only on getting differential descriptions; however, it is not guaranteed that CLIP can accurately interpret them. The effectiveness of such models hinges critically on and is bounded by the vision-language models. I suspect that this could be why the performance improvement is not very significant and why different CLIP variants could exhibit huge performance variations.

Author response:
Here, we focus on generating class descriptions that provide the most useful information for the target set of classes and how we can use LLMs to do so effectively and efficiently. Of course, different VLMs use this information differently based on their capabilities and pre-training datasets. However, as shown in our experiments, VLMs are capable of taking advantage of this information to improve performance.
Please see our general comment to all reviewers for experiments with additional VLMs, where we observe similar improvements.
Clearly, VLMs are able to take advantage of this information. However, we believe an additional attempt to improve VLMs’ understanding of natural language is a different question and is out of the scope of this work.

Respectfully, considering the challenging nature of the task, we believe our method makes significant improvements in most cases (e.g., 13.95 and 5.58 percentage points for EuroSAT and Oxford Pets). Especially, considering that without using any labeled data, FuDD’s performance as a zero-shot method is comparable to other few-shot learning methods that use labeled images for adapting VLMs.

Reviewer comment (Evaluation):
I would appreciate an error analysis: how many of the mistakes are due to insufficient / erroneous differential descriptions and how many of the mistakes are due to CLIP’s insensitivity to descriptions? This could be informative for future work.

Author response:
As an attempt to understand how the correctness and helpfulness (i.e., being differentiating) of descriptions impact the performance, we perform a manual evaluation as reported in Table 3, and observe that improving the helpfulness has a significant impact on performance even if the level of correctness of descriptions stays roughly the same. Another related experiment is described in Section 4.2, where we show that differentiating attributes are superior to generic attributes and that VLMs are sensitive to the described differences and can use them to their advantage.

As explained in Section 3.3, we use an ensemble of descriptions for each class, and often the ensemble is a mixture of helpful and correct and unhelpful or incorrect descriptions. As a result, it is challenging, if at all possible, to decide if the wrong prediction is caused by incorrect or unhelpful information or by the model’s inability to understand the correct and helpful descriptions. Therefore, this hinders such a precise analysis of the cause of the error for each image as suggested by the reviewer.

Reviewer comment (Writing):
For the equation in 3.2, what does the summation mean? I guess the equation simply means getting the top-k classes with the highest similarity with the image features? Then in this case, do we need to sum over c_i?

Author response:
As mentioned by the reviewer, this equation selects the k-most similar classes to each image. In other words, the argmax is over sets of classes, and to score a set, we have to sum the similarities of all classes in each set.

Reviewer comment (Experiments):
Would it be possible to give examples of differential descriptions for each of the evaluation datasets? It is especially interesting to see the huge performance on EuroSAT.

Author response:
Thank you for your suggestion. We include examples of differential descriptions for all datasets in the updated manuscript. It is interesting to see that differential descriptions are more effective in cases that describe low-level visual characteristics than high-level semantic features, which is consistent with our previous observations described in Section 5.

Reviewer comment (Experiments):
Would it be possible to prompt the LLM to come up with differential attributes given >2 classes in a single prompt?

Author response:
We think it is really interesting to investigate these different approaches, and we do not see an immediate reason that it does not work. Since this is the first work on generating differentiating, dataset-specific class descriptions, we focused on two classes per query. But, this is an interesting direction for future work.