Paper: Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity (ICLR-2024)

Reviewer comment (Novelty):
[Hierarchical Labels vs. Coarse-grained Labels] There is a paper (Chils: Zero-shot image classification with hierarchical label sets in ICML 2023) that shows that using fine-grained/ subclass-level labels helps zero-shot classification. It seems the conclusions of this work and ICML work are different. Please discuss this.

Author response:
* The differences between our observations and the CHiLS [1] paper as follows:
CHiLS demonstrates that mapping predictions from leaf classes across granularity levels 1-6 can enhance results for ImageNet classification (Table 1 in [1]), aligning with our findings in a multi-label classification context (Figure 3 in our paper). However, our analysis extends to all levels of the ImageNet hierarchy, up to 16. We observed distinct behaviors when the leaf classes are extremely fine-grained. In such cases, propagating their predictions to parent classes may not be beneficial. For instance, certain classes at level 16 exhibited negative changes when predictions were propagated from children at level 17, as illustrated in our Figure 3-Right.

Reviewer comment (Novelty):
[Specificity of Text] I think the observation that VLMs could be sensitive to the specificity of text is straightforward. When training VLMs, they are given specific texts. If we use the proposed modified version for training, I would expect them to perform better. Please comment on my thoughts.

Author response:
* Regarding the potential improvement in Specificity through modified training, our Appendix (Section C) explores this. We finetuned CLIP-B on COCO with diverse text samples generated by our benchmarking strategy. While fine-tuning with challenging text samples shows promise, it falls short of a comprehensive solution. Notably, the models struggle with cases outside the augmentation strategy's scope, underscoring the limitations of relying solely on fine-tuning with hard samples. Moreover, achieving certain augmentation goals, like enhancing alt-text to increase correct information, presents significant challenges.

Reviewer comment (Experiments):
Please discuss the different observations between your work and ICML work. Also, comment on my thoughts for the bias caused by the training manner.

Author response:
* The differences between our observations and the CHiLS [1] paper as follows:
CHiLS demonstrates that mapping predictions from leaf classes across granularity levels 1-6 can enhance results for ImageNet classification (Table 1 in [1]), aligning with our findings in a multi-label classification context (Figure 3 in our paper). However, our analysis extends to all levels of the ImageNet hierarchy, up to 16. We observed distinct behaviors when the leaf classes are extremely fine-grained. In such cases, propagating their predictions to parent classes may not be beneficial. For instance, certain classes at level 16 exhibited negative changes when predictions were propagated from children at level 17, as illustrated in our Figure 3-Right.

* Regarding the potential improvement in Specificity through modified training, our Appendix (Section C) explores this. We finetuned CLIP-B on COCO with diverse text samples generated by our benchmarking strategy. While fine-tuning with challenging text samples shows promise, it falls short of a comprehensive solution. Notably, the models struggle with cases outside the augmentation strategy's scope, underscoring the limitations of relying solely on fine-tuning with hard samples. Moreover, achieving certain augmentation goals, like enhancing alt-text to increase correct information, presents significant challenges.

Reviewer comment (Writing):
Section 4.2 (Impact of Pre-training Data) is not clear to me. Please elaborate your points here.

Author response:
* To clarify Section 4.2 (Impact of Pre-training Data), we compared the performance of UniCL models trained on distinct datasets. These datasets include ImageNet21K, featuring classification prompts (e.g., “a photo of {}”) and class names, and image alt-text data (YFCC-14M and GCC-15M).

  * UniCL trained solely on image alt-text data (UniCL_YFCC) outperforms UniCLIN21K in FG leaf label classification (35.75 vs. 26.28 mAP).
  * UniCL_IN21K, trained on ImageNet21K, excels in CG ancestor raw performance due to the inclusion of all CG classes in IN21K, surpassing CLIP and OpenCLIP trained on larger-scale alt-text data.
  * Training UniCL on a combination of IN21K with Alt-text Data (YFCC14M and GCC15M) improves FG classification but slightly worsens CG classification, leading to a larger CG-FG discrepancy.
  * Scaling up alt-text training data: Larger models, such as OpenCLIP-H-2B trained on LAION2B, show inferior CG ancestor raw performance and a more pronounced CG-FG discrepancy than UniCL_IN21K, indicating that merely scaling up the training data or model sizes is insufficient to resolve this discrepancy effectively.

  We will modify our draft to make this section clearer.

Reviewer comment (Novelty):
The evaluation on the technical side is rather straightforward, and I am not sure how surprising the results are.

Author response:
Dear Reviewer stdj,

Thank you for your time for reviewing our submission. 

We acknowledge your concern regarding the straightforward nature of our technical evaluation. However, we wish to emphasize that the straightforwardness of our approach was crucial for ensuring the clarity and replicability of our results. Our methodology, while seemingly intuitive, was designed to rigorously test VLMs in scenarios that have not been explicitly showcased or examined in existing literature.

Reviewer comment (Novelty):
Although this study may provide few valuable insights, I do not think it meets the ICLR novelty threshold.

Author response:
Regarding the novelty of our work, we believe our contribution is significant in several aspects:
Empirical Evidence and Novel Benchmarks: We have developed new benchmarks and conducted comprehensive analyses, providing empirical evidence of the VLMs' performance across different granularity and specificity levels.
Development of a New Evaluation Protocol: A major novelty of our research lies in the creation of a new evaluation protocol and the modification of existing datasets to fit this protocol. This effort was non-trivial and has not been previously undertaken to our knowledge. We would greatly appreciate it if the reviewer could provide references to previous works that are dedicated in the same topic of our work. 
We also provide deep analysis on how the training data impact the observed granularity issue and if finetuning can solve the specificity issue.

We believe that our research fills a crucial gap in the current understanding of VLMs, and we hope that our responses clarify the novelty and depth of our contributions to this important area of study.

Reviewer comment (Evaluation):
Textual Bias in Specificity Benchmark. As the authors point out in the supplement, language-only methods are competitive on this benchmark, suggesting that the protocol should be amended to more accurately evaluate vision-language capabilities, rather than just language understanding. In fact, many VLM benchmarks face the same problem of not requiring vision to solve the task.

Author response:
- **Textual Bias in Specificity Benchmark**: 
  - We would like to clarify a critical aspect of our study that seems to have been misunderstood. Our language-only study was not aimed at demonstrating the competitiveness of language-only methods on the VLM benchmark. Instead, its purpose was to investigate whether the granularity issue observed in VLMs also manifests in language modeling alone.
  - We specifically designed a text-classification task based on two-level granularity concepts, derived from our VLM benchmark, to assess if language models exhibit similar granularity issues. Our findings revealed that while text encoders in VLMs and embedding-type LMs struggle with this task, larger generative Language Models (LLMs), such as GPT-3.5, show better performance. However, it remains unclear whether this improved performance is attributable to the generative modeling approach or simply the larger scale of these models.

Reviewer comment (Evaluation):
On the Importance of Granularity from Vision. The granularity benchmark actually tests a model's alignment with word-net, which may not be universally accepted. For example, searching the web for "road-cone" and "barrier" show two visually distinct items. Although people may agree that road-cone and barrier are related, it is unclear if road-cone is a child class of barrier. Therefore, simply training a VLM in alignment with word-net rather than the broader internet may artificially boost performance on this benchmark, but perform worse in practice.

Author response:
- **On the Importance of Granularity from Vision**
WordNet for its structured and widely-accepted semantic relationships, which provide a clear framework for evaluating VLMs' understanding of granularity. We believe overall it defines the most widely accepted conceptual relations. To clarify, the example “road-cone” and “barrier” we used to explain the granularity issue is not part of WordNet. We believe WordNet’s semantic relationships provide a solid foundation for evaluating VLMs' understanding of granularity while it may not capture every nuance of real-world visual categories. We will include the discussion in the limitation section.

Reviewer comment (Evaluation):
What is the Summary Metric? Although the in-depth analysis is appreciated, it is difficult to identify the best and worst performing methods (as is common in a benchmark). How can the methods be ranked for both benchmarks?

Author response:
- **Summary Metric:** our study does not employ a single summary metric as it evaluates VLMs across two separate benchmarks: granularity and specificity, each testing different aspects of VLM capabilities. Consequently, each benchmark has its specific set of metrics that appropriately capture the distinct properties we aim to assess. This approach ensures a more nuanced and accurate evaluation of VLM performance in diverse scenarios.

Reviewer comment (Writing):
Please revise the text of this paper, there are many spelling and grammar errors (e.g. Figure 5 poisoneded).

Author response:
We will carefully review our draft to correct any spelling or grammatical errors.

Reviewer comment (Novelty):
Going Beyond Common Knowledge: The paper highlights that performance in granularity and specificity improves when the testing scenario mirrors the training set, a well-known fact in the field. A more substantial contribution could be made by delving deeper into this issue, perhaps by providing baseline solutions or strategies to enhance granularity and specificity in pre-trained models.

Author response:
We understand your concern that our findings about performance improvement under training-like scenarios may appear intuitive. Nevertheless, our contribution lies in the empirical evidence and systematic exploration of this phenomenon, which has not been explicitly explored in previous research. Our benchmarks provide a foundation for future in-depth studies in this area. Furthermore, we explored the efficacy of simple fine-tuning with hard examples to address the specificity issue. Detailed in Appendix (Section C), we fine-tuned the CLIP-B model on COCO using various text samples generated through our benchmark strategy. While fine-tuning shows some promise, it falls short as a comprehensive solution, particularly in instances outside the augmented sample scope. This finding underscores the limitations inherent in relying solely on fine-tuning with hard samples.

Reviewer comment (Experiments):
Connection to Specific Vision Tasks: While the paper provides a high-level motivation for the need of such benchmarks, it stops short of connecting the dots to specific vision tasks. A more robust argument could be made by demonstrating how improvements in the proposed benchmarks translate to advancements in other vision-language tasks, such as open vocabulary object detection, open vocabulary tracking, and text-to-image generation. At least, a high-level discussion is needed in this regard.

Author response:
We are grateful for your suggestion to link our benchmarks to specific vision-language tasks. Downstream VL tasks like open-vocabulary detection/tracking and text-to-image generation often leverage pre-trained VL models assessed in our benchmarks. They often leverage the pretrained VL models tested in our benchmarking by knowledge distillation or apply similar image-text embedding alignment strategies on task-specific models with comparable image-text data. Consequently, they are subject to the same granularity and specificity issues we identified. Our improvements in these benchmarks should, therefore, translate to advancements in these tasks, yielding more robust pre-trained VL models and improved training methodologies. We will incorporate this discussion into our manuscript to underscore our benchmarks' practical implications in enhancing VLM performance across various applications.

Reviewer comment (Evaluation):
Evaluation Using COCO: Why are COCO's original captions more effective than detailed ones? Is there any assurance that the model hasn't been exposed to any COCO images or captions during training?

Author response:
We believe the original captions are more effective due to their similarity in information content with the alt-text in VL training data, which is typically shorter than Localized Narratives captions. While there is overlap between COCO images and web-crawled datasets like LAION and YFCC, COCO's distinct annotation process separates its captions from these datasets. Recognizing your concern about potential model exposure to COCO images and captions during training, we note that such exposure is a common challenge in VL model evaluations. However, given the vast scale of pretraining data (billions of image-text pairs) compared to the size of the COCO dataset (hundreds of thousands), the models' behavior is predominantly shaped by the broader training data spectrum, where COCO's portion is minimal.

Reviewer comment (Experiments):
Apart from the analyses, it'd be great to explore some simple ideas to mitigate the granularity and specificity issues in these VLMs by light-weight finetuning. Some ideas include: augment the alt-text with LLM to increase the amount of information; generate hard-negatives by replacing the nouns; adjust the granularity by replacing the fine-grained concepts with their parent in the hierarchy. Since the analysis points to the mismatch between training and test data distribution, it'd be natural to bridge the distribution gap by data augmentation.

Author response:
We appreciate your suggestion to explore lightweight fine-tuning methods. Our appendix (Section C) investigates this for the specificity issue, where we finetuned CLIP-B on COCO using various text samples generated by our benchmarking strategy. While fine-tuning with hard text samples shows some promise, it’s not a comprehensive solution. In particular, we found that the models tend to fail on cases not covered by the augmentation strategy. This highlights the inherent limitations of relying solely on fine-tuning with hard samples. It’s also important to note that some augmentation goals, such as "augmenting the alt-text with LLM to increase the amount of information", can be challenging to achieve. There is no guarantee that the LLM-generated information will be well aligned with the image, posing a significant challenge for ensuring model accuracy and relevance. We concur that the ideas you have suggested are intriguing and will discuss them as potential future directions, acknowledging the complexities involved.

Reviewer comment (Experiments):
It's not obvious how to translate the findings of this work into improvements in the downstream application of CLIP. Take the granularity study for example. Let's say we want to build an open-vocabulary detector on some super categories of LVIS. To achieve the best performance, we'd need a way to generate the fine-grained categories associated with those super-categories, where the super-categories may not be in the WordNet hierarchy. One idea is to use the LLMs to help generate fine-grained categories so that we can apply the max-score aggregation in Fig 2 (c).

Author response:
Our study provides essential insights into the limitations of current VLMs in terms of granularity and specificity, forming a basis for enhancing these models for practical applications such as open-vocabulary object detection. We acknowledge the potential of applying our findings to practical applications, as you kindly suggested, and will include a discussion in our revised draft on this promising yet extensive topic, which requires further investigation beyond our current scope.

Reviewer comment (Novelty):
Although the analyses are comprehensive and interesting, the findings are not very surprising. It seems to boil down to the limitations of the data that naturally exists at large-scale at the end of the day. I'd recommend to take the discussion section outside of conclusion and expand on it more to address the limitations.

Author response:
We believe that the seemingly intuitive nature of findings does not diminish their importance. Our work, like recent studies on VLMs' compositional understanding limitations, provides vital empirical evidence, despite some findings being in line with expectations. These insights are crucial for advancing the field. We have discussed the limitations in Appendix Section E and will expand this discussion to further articulate the significance of our findings.