Paper: Lost in Translation: Conceptual Blind Spots in Text-to-Image Diffusion Models (ICLR-2024)

Reviewer comment (Writing):
The paper is difficult to read. For instance: "In this category, one concept (A or B) demonstrates a dominant relationship, either with an underlying C or" -- it is unclear what A, B, and C represent.

Author response:
**"Concept Pairs" denotes two main objects A and B in a text segment asked by users. C, while not explicitly indicated, is often bound together with B (or A) in the training dataset.** For example, in the case of a teacup (A) and iced coke (B), the frequent association of iced coke (B) with a transparent glass (C) in datasets leads to the replacement of the teacup (A) by the transparent glass (C). The inclusion of this hidden object C is the main focus of our study.

Reviewer comment (Experiments):
It is unclear how the categories and patterns in Table 1 are determined.

Author response:
**Categories and patterns in Table 1 are classified by human experts through meticulous observation.**

    2.1 Specifically, Category I represents the misalignment issues involving the hidden object C, as mentioned above.  Category II, on the other hand, represents traditional issues involving only A and B, such as requesting an image of a frog and a turtle, but the final image contains only two frogs.

    2.2 **Each pattern represents more specific scenarios within each category** , such as "Beverage and Erroneous Container", "Local Cuisine and Non-native Location", ... The existence of patterns can better provide GPT with few-shot guidance.

Reviewer comment (Reproducibility):
"Initially, 259 concept pairs were identified through the collaborative efforts of human researchers, supported by GPT." -- How does GPT help here? What were the guidelines for human researchers?

Author response:
"What GPT helps, and human researchers guide?" 
**In this process, GPT helps to come up with more misalignment concept pairs, and human researchers provide instruction prompt, few shot and evaluation on images.**
Specifically, GPT generates new examples that it deems likely to exhibit misalignment issues, based on given misalignment cases. Human researchers are responsible for designing the instruction prompts and providing examples for GPT's few-shot learning. We have included the complete instruction prompt in the appendix of our paper. Furthermore, the examples provided by GPT are sent to Midjourney and SDXL-1.0 for image generation, and human researchers then assess whether misalignment issues occur.

Reviewer comment (Reproducibility):
"After rigorous screening, 159 concept pairs attain a Level 5 rating, representing the pinnacle of quality." -- Can you elaborate the screening process?

Author response:
**"Elaborate screening process"**
Of course, GPT first expands each concept pair with five information-rich sentences, and each sentence undergoes four rounds of image generation. This results in a total of 20 images, from which human experts select based on the number of images correctly representing both A and B. A Level 5 rating means all 20 images exhibit misalignment issues, indicating the highest quality samples. All complete instruction prompts are included in the appendix of our paper.

Reviewer comment (Writing):
The presentation of the paper is bad. Some important terms such as "conceptual blind spots," "concept pairs," and "Socratic reasoning/questioning" are not clearly defined, which hinders the understanding of the paper.

Author response:
1. "Missing the clear definition of important terms." Thank you very much for pointing out that some important terms are not clearly defined.

    1.1 **"Concept Pairs" denotes two main objects A and B in a text segment asked by users.** In text-to-image generation, when a text segment contains multiple objects, some may not be accurately represented in the resulting image.  In our study, we focus on scenarios where a text segment contains only two objects.  We refer to these two objects as A and B, collectively termed a "concept pair."

    1.2 **"Conceptual Blind Spots" (CBS) denotes object A (or B) is encroached upon and consequently disappears in the image.** In text-to-image generation, "Conceptual Blind Spots" represents the occurrence of object encroachment. Specifically, this denotes the disappearance of object A (or B), which represents the user's requirement, in the final image. Previous studies primarily focused on scenarios where A is encroached upon by B. However, we have observed that in many instances, A can also be encroached upon by a hidden object C, which often appears bound with B in datasets. For example, in the case of a teacup (A) and iced coke (B), the frequent association of iced coke (B) with a transparent glass (C) in datasets leads to the replacement of the teacup (A) by the transparent glass (C). The inclusion of this hidden object C is the main focus of our study.

    1.3 **"Socratic Reasoning"[1] denotes the process of problem identification, understanding and reasoning using Large Language Models (LLMs), particularly GPT.** We utilize GPT to help us come up with concept pairs that are likely to encounter issues of "Conceptual Blind Spots". The specific procedures for this approach are detailed in Appendix of our paper.

Reviewer comment (Experiments):
A large portion of the paper is describing the data collection process, but the overall goal of the dataset is not clearly explained, and the reason behind each round is also not explained.

Author response:
2. **Missing detailed explanation of the process of data collection** Thank you very much for pointing out that we miss clear explanation for the data collection.

    2.1 **"The overall goal of the dataset" is that we aim to collect CBS samples with inclusion of a hidden object C** mentioned above, a focus not emphasized in previous work. Typically, earlier studies concentrated on scenarios involving only A and B mutually encroaching upon each other.

    2.2 **"The reason behind each round" is that we use a divide-and-conquer strategy, leveraging the extensive knowledge possessed by GPT to expand our dataset.** Specifically, human mind has its limits, hence the need for GPT's assistance. However, GPT often cannot complete the task in one step, leading us to divide the data collection process into several rounds. Initially, we provide CBS examples conceived by human experts and use these as few shot for GPT to generate more examples. Subsequently, human experts categorize these examples into patterns, creating stronger few shot for further rounds. Here, human experts are involved only in the initial two stages. Afterwards, GPT operates like a snowball effect, continually expanding the scope of data collection.

Reviewer comment (Presentation):
The methodology section is confusing. It is unclear how the proposed metric $D$ is used during generation, and why the binary search algorithm is used for generation. An algorithm table could help explain the method more clearly.

Author response:
3. **Missing detailed explanation of our method** Thank you very much for pointing out that we miss clear explanation for the our method. We provide a detailed explanation here, and will give an algorithm table in the updated version of our paper.

    3.1 **A smaller value of D indicates a higher likelihood of objects A and B appearing together in the image.** Specifically, during N-step diffusion sampling, GPT initially suggests the drawing sequence for A and B. Taking the case where A is drawn first, we provide the model with text prompts only for A in the initial x steps and then supply prompts for both A and B in the remaining N-x steps. After the drawing process, we adjust x based on the magnitude and sign of D to find a more optimal x.

    3.2 **We employ a binary search algorithm because we believe that the absolute value of D has a linear relationship with x.** For concept pairs A and B prone to CBS issues, if x is too small, CBS issues will still occur, whereas if x is too large, it limits the time steps available for drawing B. Therefore, we use a binary search algorithm to quickly find the appropriate value of x.

Reviewer comment (Experiments):
The experiment setting is problematic because only the standard stable diffusion model is used as a baseline, and no comparison is provided for related works mentioned in Section 2.

Author response:
4. **Problematic experiment setting** Thank you very much for pointing out that the experiment setting is problematic.

    4.1 **We have supplemented our research with experimental results from two baseline models, AAE[2] and DALLE3[3]** The results of human evaluations are presented here and will be updated in the paper.

    4.2 **We greatly appreciate your feedback regarding the evaluation metrics.** Given the tendency of automatic scoring to be ineffective in addressing CBS issues, we will focus on human evaluation, encompassing both the experiments already included in the paper and the additional ones we have supplemented.

| Method | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Average Level |
|-------|-------|-------|-------|-------|-------|-------|
| SDXL-1.0 Baseline | 0 | 0 | 0 | 0 | 173 | 5 |
| AAE | 0 | 0 | 6 | 10 | 157 | 4.87 |
| DALLE3 | 14 | 24 | 31 | 23 | 81 | 3.77 |
| MoCE (ours) | 11 | 25 | 40 | 71 | 26 | **3.43** |

Reviewer comment (Writing):
Some notations are wrong. For example, in the Background of Section 4, $I$ shouldn't denote the input image.

Author response:
5. Thank you very much for pointing out that some notations are wrong. We will make modifications and improvements in the updated version of our paper.

Reviewer comment (Novelty):
Several related works are missing, such as [1], [2], [3], and [4].  

[1] Feng et al., 2022. Training-free structured diffusion guidance for compositional text-to-image synthesis.  
[2] Lian et al., 2023. LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models.  
[3] Bali et al., 2022. eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers.  
[4] Wu et al., 2023. Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis.

Author response:
6. Thank you very much for pointing out that several related works are missing. We will make modifications and improvements in the updated version of our paper.