Paper: Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy (ICLR-2024)

Reviewer comment (Evaluation):
The efficacy of the proposed method presented in this study is heavily dependent on the performance of the image classifier. Both the In-Subtree Probability (ISP) and Subtree Coverage Score (SCS) metrics rely on the probabilities generated by the classifier. To achieve reliable results, it is crucial to obtain a highly accurate classifier capable of effectively covering a wide range of real-world object classes. The metrics' effectiveness is contingent on the classifier's performance, meaning that if the classifier demonstrates low prediction accuracy, the reliability of the ISP and SCS metrics may be compromised.

Author response:
> The efficacy of the proposed method presented in this study is heavily dependent on the performance of the image classifier.

We agree, which is why **we use an accurate and robust pretrained ImageNet classifier** (ViT-B/16) in our evaluation and in the reference implementation of the metrics we propose. For a detailed discussion about the validity of using pretrained classifiers, please see the **general response**; overall, we argue that this dependence should not be viewed as a disadvantage of our approach.

Reviewer comment (Experiments):
Pretrained generic large language models (e.g., T5, Llama2), trained on text-only corpora, demonstrate proficiency in text encoding for image synthesis. These models inherently possess semantic understanding and effectively acquire knowledge of hypernymy. More experiments of applying the proposed metrics into text-to-image models equipped with LLMs need to be conducted.

Author response:
> Pretrained generic large language models (e.g., T5, Llama2), trained on text-only corpora, demonstrate proficiency in text encoding for image synthesis
> Please consider extending the experiments with more LLMs-equiped text-to-image models, e.g. Imagen

Thank you for this suggestion! We would like to note that **Imagen is not publicly available**; therefore, it is not possible to run experiments on this model. Moreover, large language models **do not necessarily have much better hypernymy knowledge**: as shown in [1], models like ChatGPT can fall behind BERT, CLIP, or other smaller neural networks in terms of hypernym discovery, and GPT embeddings are only marginally better when compared to CLIP (which is used in the majority of the models that we study).

Still, to address your concern, we conducted experiments on DeepFloyd IF XL (which has **T5-XXL as its backbone**) and included results in the updated revision of the paper. We kindly ask you to see the new versions of **Tables 1, 9, and 10** and tell us if these results address your concern. In the camera-ready version, we will improve the fidelity metrics of DeepFloyd with super-resolution (that was harder to run within the limits of the response period), but other results should remain similar.

[1] Concept Understanding in Large Language Models: An Empirical Study. Liao et al., ICLR 2023 Tiny Papers track.

Reviewer comment (Experiments):
Please consider extending the experiments with more LLMs-equiped text-to-image models, e.g. Imagen (Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding) (but not limited to.).

Author response:
> Pretrained generic large language models (e.g., T5, Llama2), trained on text-only corpora, demonstrate proficiency in text encoding for image synthesis
> Please consider extending the experiments with more LLMs-equiped text-to-image models, e.g. Imagen

Thank you for this suggestion! We would like to note that **Imagen is not publicly available**; therefore, it is not possible to run experiments on this model. Moreover, large language models **do not necessarily have much better hypernymy knowledge**: as shown in [1], models like ChatGPT can fall behind BERT, CLIP, or other smaller neural networks in terms of hypernym discovery, and GPT embeddings are only marginally better when compared to CLIP (which is used in the majority of the models that we study).

Still, to address your concern, we conducted experiments on DeepFloyd IF XL (which has **T5-XXL as its backbone**) and included results in the updated revision of the paper. We kindly ask you to see the new versions of **Tables 1, 9, and 10** and tell us if these results address your concern. In the camera-ready version, we will improve the fidelity metrics of DeepFloyd with super-resolution (that was harder to run within the limits of the response period), but other results should remain similar.

[1] Concept Understanding in Large Language Models: An Empirical Study. Liao et al., ICLR 2023 Tiny Papers track.

Reviewer comment (Novelty):
This paper focuses on a very specific aspect of text-to-image generation.

Author response:
While we agree that we study a particular aspect of generation abilities, we believe that the **hypernymy relation is a sufficiently abstract concept** that can be broadly useful. As we notice in the paper, there are multiple applications of hypernymy that encompass different questions faced by text-to-image generation practitioners. Moreover, as shown by prior work [1], **even specific aspects could be of significant interest** to the community. Our metrics offer a general foundation for a more semantics-driven approach to evaluation, which makes it possible for the research community to explore nuances not captured by existing methods.

Reviewer comment (Evaluation):
The proposed evaluation framework relies on a well-trained image classifier. Its performance depends on the accuracy and coverage of the image classifier. Particularly, its usefulness may be limited by the coverage of existing image classifiers.

Author response:
Thank you for this observation! We believe that the existence of many **high-quality and robust pretrained ImageNet classifiers** makes our approach viable; for more details, see our **general response**.

Reviewer comment (Evaluation):
The results of some experiments (e.g., the influence of the classifier-free guidance scale, the number of diffusion steps, and the number of generated samples) seem to be self-evident and provide little new insights.

Author response:
**These experiments were designed to have unsurprising results:** their goal is to show that our metrics **behave as expected** when sampling hyperparameters are changed and that the default hyperparameters we use for evaluation provide sensible results. In other words, these experiments are **necessary to validate the design choices** of our evaluation protocol, which, in our opinion, should be helpful for intended users of ISP and SCS.

Reviewer comment (Evaluation):
For Subtree Coverage Score, why not use simpler formula such as the entropy of the average distribution.

Author response:
Thank you for this question! In general, our motivation is the same as in the Inception Score paper: using the entropy of the average distribution **does not allow us to distinguish** between diverse images of specific objects and images covering mixed objects in one sample. More concretely, we have measured the entropy of the average distribution for setups from Section 4.3 and added its correlation with human preferences to **Appendix H**. As you can see there, measuring entropy in each synset has lower agreement than SCS.