Paper: LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR-2024)

Reviewer comment (Experiments):
Missing information on human study. Detailed information on human study could be provided to assess the reliability of the outcome. How do you recruit and select them based on what qualification? Isn't there any conflict of interest for the subjects and authors? How many subjects are recruited? What was the confidence of the votes? This issue is the major reason for leaning toward rejection. I am eager to see the author's feedback for reassessment.

Author response:
Our approach is designed to overcome the limitations observed in diffusion models when confronted with lengthy and intricate textual prompts that involve multiple objects. Given the absence of an efficient metric tailored for this specific challenge, we conducted a thorough human study to offer a more comprehensive evaluation, as outlined in the main paper. The subjects for human study were recruited from a pool of academic students with backgrounds in computer science and engineering.
To ensure an unbiased evaluation and maintain objectivity, the subjects participating in the evaluation remained anonymous with no conflicts of interest with the authors.  Furthermore, we prioritized user confidentiality by refraining from collecting personal details such as names, addresses etc.

The entire survey process was anonymized, ensuring that there was no leakage of data to any user. Following a 2-AFC (two-alternative forced choice) design, each participant was tasked with selecting one image out of a pair of images, which best aligns with the given textual prompt and faithfully captures all the objects. Each pair of images was randomly sampled from a pool containing images from both baselines as well as our approach. This system mitigates biases which can originate from user-curated surveys. Further, the pool of images was shuffled before sampling each time. The system was further equipped with the feature of randomly flipping the position of the images, to remove the user-specific position bias in case any pair of images is repeated. Additionally, a 2-second delay between questions was introduced to facilitate more thoughtful decision-making.

The system yielded approximately 350 responses from 50 subjects, with each user answering an average of 7 questions. The results, as presented in Fig. 4 of the main paper, indicate that users predominantly selected images generated from our approach. This observation is further supported by qualitative analyses (Fig. 5 of the main paper and Figs. 9 and 10 of the appendix) and quantitative comparisons in Section 4 of the main paper and Table 1 of the revised appendix. We believe that this comprehensive approach and the subsequent results contribute significantly to the robustness and reliability of our outcomes.

Reviewer comment (Novelty):
This framework relies on recently proposed models like a strong LLM, CLIP, and an image composition model. The collection of previous works provides shallow techniques compared with them.

Author response:
As per our understanding of the question, our methodology leverages the complementary strengths of diverse existing approaches, enhancing and refining each component in the process. Furthermore, our approach incorporates specific modifications to optimize the performance of each individual element. To prevent the Large Language Model (LLM) from generating results that lack coherence, we implement an interpolation mechanism. Additionally, we address potential issues in the output of the first-stage model through an iterative refinement scheme. In the context of the composition model, we introduce multi-modal guidance. The the impact of these components is visually demonstrated in Figures 6 and 7 of the main paper. Thus, our proposed layout generation, iterative refinement and multi-modal guidance solves for the weaknesses of the existing components and enhances the overall generalization towards complex textual prompts.

Reviewer comment (Presentation):
In Sec. 3.1, Models. -> Models (Please exclude the period in the section title.)

Author response:
We understand the reviewer's concerns. Thank you for pointing it out. We have removed the ChaTGPT logo in the new revised version of the paper. We have further removed the period from Sec. 3.1 title.

Reviewer comment (Evaluation):
I think the major limitation of the proposed approach is efficiency. The complexity of the proposed approach increases as the number of objects increases for complex scenes. This might also be the major issue presenting this approach to be applied in real usage.

Author response:
Our approach operates as a two-stage framework, relying significantly on the efficacy of the underlying models it employs. While some existing state-of-the-art methods exhibit faster performance, they encounter challenges in faithfully capturing every detail of lengthy and intricate textual prompts. Notably, these methods often fall short in generating all the objects specified in the prompt, as illustrated qualitatively in (Figs. 1 and 5) of main paper and (Figs. 9 and 10) of appendix. Our approach demonstrates a high abjectness score expressed as Prompt Adherence Recall (PAR) rate, indicating a notable alignment between the number of objects mentioned in the prompt and those actually present in the generated image (please see Sec. 4 in the main paper). Therefore, a discernible trade-off emerges between addressing complexity and ensuring the faithful reproduction of images. We provide a comparison of our method with existing approaches in terms of time efficiency and PAR score below.

Table 2. Quantitative comparison of our approach with baselines in terms of PAR score and inference times. Kindly note that all inference times are measured on a single Nvidia A100 GPU.

| Method|PAR score |Inference Time (min) |
|-----|-----|---|
| Stable diffusion | 49\%  | 0.18 |
| GLLIGEN |  57\% | 0.5 |
| LayoutGPT | 69\%   | 0.83 |
| DeepFloyd |  60\% | 8.33 |
| DenseDiffusion | 52\%  | 2.5|
| Ours  | 85\%  | 3.16  |

Recent works, such as those introduced by [1] and [2], have presented methods capable of scaling up diffusion inferences. Looking ahead, we anticipate incorporating these advancements in speed-boosting techniques into our ongoing research as a future work.

Reviewer comment (Experiments):
It seems that although the generation of objects can be iteratively refined, the bounding box locations cannot be refined. If the LLM predicts unreasonable bounding box layouts at the first stage, it cannot be corrected. Have the authors think of introducing the refinement of bounding box locations into the pipeline?

Author response:
Our method is based on two stage process of layout generation and bounding box refinement. The spatial faithfulness of final generated image depends on the quality of layouts generated in the first stage.  We observe that ChatGPT is effective in generating faithful layouts  (please refer to Sec. A.7 of appendix) which exactly follow the details from the textual prompt. [3] further verifies this finding. However, in extreme cases, such as in Fig. 6 of the main paper and Fig. 12 of the revised appendix where object positions are not explicitly defined in the prompt, ChatGPT produces bounding boxes that do not coherently align with the prompts. We tackle such cases by generating multiple layouts, selecting the relevant ones via clustering and then interpolating them to a single optimal layout to lessen the influence of any outlier bounding boxes. We further provide a user-specific hyperparameter $\eta$ (refer to Fig. 3 of main paper) which can be controlled to vary the location of bounding boxes. Refinement of bounding boxes along with object refinement is an interesting future research idea, but is likely to further scale up the compute cost which may not be ideal for user experience.

Reviewer comment (Experiments):
How many layouts are needed to interpolate? How does the number of layouts for interpolation affect the results?

Author response:
While there is no restriction on the number of layouts needed for interpolation, our final results are generated with the interpolation of 3 layouts. Please refer to Figures 11 and 12 (Sec A.6) of the revised appendix for a visualization of generated images with different number of layouts. We conclude that while interpolation plays a significant role on the final generated image in extreme cases where spatial locations are not clear from the prompt, it does not have a drastic impact on the final generated image when object locations are well-defined.

Reviewer comment (Writing):
While the writing is satisfactory, it can still be improved! The authors should provide more information in the paper on how Eq. (5) is used to guide the sampling process.

Author response:
**1. Information on how Eq. (5) is used to guide the sampling process and clarity in writing.**

We kindly refer the Reviewer to section A.8 of revised appendix for a detailed mathematical explanation on sampling process using guidance. 

We will refine the writing for better clarity and ease of understanding in our final manuscript.

Reviewer comment (Writing):
Can the authors provide more intuition on the interpolation step?

Author response:
**2. More intuition on the interpolation step.**

We observed that proprietary LLMs such as ChatGPT (used in our approach) are good at following the details from the prompts and generating image blueprint in the form of box proposals as per the prompt. The same has been verified in [1]. However, we observed that in the multi-object settings where the spatial position of objects in unclear, ChatGPT has certain inherent bias for such cases. We conducted an experiment where we instructed ChatGPT to provide boxes for the prompt "In a living room with a dog and a cat sitting one on each side". Note that, in this prompt, the relative position of cat and dog is not clear. We observed that 60\% of the times ChatGPT generated bounding boxes towards the left for the dog and only 30\% times the dog was on the right. For the remaining 10\% of the times, it had arbitrary position.
To avoid such errors and to account for this bias, we instead prompt ChatGPT to output $K$ bounding boxes per object proposal. To exploit the above bias, we perform density clustering and choose the cluster containing the majority of the boxes. After this, we introduce the interpolation step to merge these filtered bounding boxes into a single optimal box. This potentially removes the effect of the outlier boxes if present. We have further presented a visual demonstration of the effect of such irrelevant boxes on the final generated image in Fig. 6 of the main paper and Fig. 12 of the revised appendix. Kindly note that this is the extreme case where the position of the objects is not clear from the text. So to further facilitate the user-specific control, we provide a hyperparameter $\eta$ which can be controlled to adjust the position of the boxes. Kindly refer to Sec. 3.3 of the main paper for the description of $\eta$ and Fig. 3 (main paper) for qualitative visualization of the effect of $\eta$.

Reviewer comment (Experiments):
Given that there are stronger open-source diffusion models (e.g., DeepFloyd) — the authors should provide some context on how long prompts work in those cases, as they use a stronger text-encoder like T5.

Author response:
**3. Comparison with DeepFloyd.**

As recommended, we compare our method with the DeepFloyd and present visual comparisons in Fig. 10 (section A.5) of the revised appendix. Additionally, we also present quantitative comparisons in terms of Prompt Adherence Recall (PAR) below in Table 1. We observe that even with a strong T5 text encoder, DeepFloyd struggles to generate coherent images with complex compositional prompts. We conclude that our scene blueprint augmented with iterative refinement is effective for generating coherent images from complex compositional prompts.

Table 1. Quantitative comparison of our approach with DeepFloyd.

| Method| PAR score |Inference Time (min)|
|-----|-----|---|
| DeepFloyd |  60\% | 8.3 |
| Ours  | 85\%  | 3.16 |

Reviewer comment (Evaluation):
While the authors comment that the size of the tokens (77 in CLIP) is one of the potential reasons why SD cannot generate compositional prompts — I believe this is only partially true. Even for non-complex compositional prompts, SD is not able to generate coherent images. Can the authors comment in general on some potential reasons why these text-to-image models are not able to generate images corresponding to simple compositional or complex prompts? I think both are related somehow and it will be beneficial to provide some context regarding it.

Author response:
**4. Potential reasons why these text-to-image models are not able to generate images corresponding to simple compositional or complex prompts?**

We agree with the reviewer's comments that current text-to-image diffusion models struggle to generate the image content even for non-complex compositional prompts. Experiments on DeepFloyd with a strong text encoder show that even stronger models struggle to generate coherent images. As per our intuition, there could be several factors that could contribute to inconsistencies in text-to-image diffusion models to generate images corresponding to simple compositional or complex prompts.

- Lack of generalizability towards diverse textual prompts can lead to images with missing details.

- The limitation could also stem from the lack of compositional prompts in the training dataset of text-to-image diffusion model inhibiting it from generalizing and generating diverse and accurate images for those prompts.

- Lack of location-aware training of diffusion models on complex compositional prompts can limit their ability to capture intricate relationships between objects, scenes, or attributes, leading to incoherent images.