Paper: R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR-2024)

Reviewer comment (Novelty):
It is unclear to me how the presented work should be viewed in comparison with ZestGuide [1]. The authors cite it and use the method in [1] to compute an aggregated attention map. However, there is no visual/quant. comparison and a discussion on the strengths/weaknesses are missing. From a high-level, it seems that [1] is able to solve the same issues using a simpler method, namely a zero-shot segmentation approach to guide the diffusion process on a per-object level. Furthermore, [1] is able to use free-form masks instead while the presented approach is limited to boxes.

Author response:
Firstly, bounding boxes and segmentation maps are two forms of spatially conditioning input at different granularity. Bounding boxes are coarse but more user-friendly, while segmentation maps provide more spatial information but are harder to acquire. We think both two settings have research and application value. Secondly, we compare **ZestGuide** with our proposed **R&B** qualitatively in **Appendix L**. We observe when conditioned on grounded input like bounding boxes, which only coarsely describe the size and location of corresponding objects, Zest tends to fill the entire boxes, leading to poor generative results. Please refer to the paper for more analysis.

Reviewer comment (Experiments):
There is recent work that tackles poor attribute-noun binding in Text-to-Image models. Can the authors provide examples where one object has an additional attribute such as a color and visualize how the generated image is changed when varying the attribute or noun (e.g. "blue car" -> "orange car" -> "orange fox")? See [2] examples.

Author response:
We conduct the visual experiments following your suggestion in **Appendix I (Figure 16)**, illustrating how the proposed **R&B** copes with the attribute binding issues. Given consistent layout, we vary the corresponding phrases to adopt different visual outcomes. Results show that **R&B** helps to bind the nouns and their corresponding attributes across different variations with good robustness.

Reviewer comment (Evaluation):
Although the LIMITATIONS AND DISCUSSION section acknowledges that the model's capacity may be exceeded when dealing with a high number of objects, providing a more in-depth analysis of this phenomenon would enhance the discussion. Notably, the BoxDiff model appears to yield more favorable results in generating a large number of objects.

Author response:
Generating multiple objects is challenging for T2I models due to the problem of object dominance, where the saliencies of objects compete with each other and some objects are hindered. In grounded generation setting, object saliencies are encouraged within the specified bounding boxes, mitigating the object dominance problem to some extent. Intuitively, the more accurate the spatial guidance is, the better performance on multiple objects will be.  Notably, this problem is alleviated when multiple objects and their box instructions have strong semantic consistency, i.e. the composition of these objects are common, which explains the delicate results of multiple-object examples presented in **BoxDiff**. Some of the visual examples from our revised manuscript include multi-objects, for example, **Figure 1 (last column)**, **Figure 11-13**, **Figure 13**, **Figure 18 (upper half)**, **Figure 20**, **Figure 22 (bottom half)**, **Figure 23 (bottom half)**, and **Figure 24 (last row)**. We run the same examples of BoxDiff in **Figure 17** for comparison. We use same random seeds (annotated at the bottom) for different methods. Our proposed **R&B** shows comparable or even better generative results. HRS dataset consistutes of compositions of 2, 3 and 4 objects. The proposed R&B method surpasses all training-free methods including BoxDiff on the HRS dataset, proving the ability of handling multiple objects of R&B. R&B provides more accurate spatial guidance through the proposed two losses, leading to the performance gain.

Reviewer comment (Experiments):
To enhance the comprehensiveness of the study, including additional visual comparisons with other methods would be advantageous. For instance, an evaluation of how our model performs with consistent boundary and prompts when compared to BoxDiff or other comparable methods could provide valuable insights.

Author response:
We conduct a qualitative comparison using cases extracted from the **BoxDiff** paper between our proposed **R&B** and other methods in **Figure 17**. Please refer to **Appendix J** 'Qualitative' part for more analysis. We fix the random seeds and annotated them below the images for better comprehension. We also observe that the generative layouts of **BoxDiff** are unstable and sensitive to the initial latents due to the top-k optimization strategy. On the contrary, our method show robustness to different prompts and seeds.

Reviewer comment (Evaluation):
It is imperative to incorporate more detailed analyses, such as color and object counts. Additionally, there is a concern regarding the last row in the rightmost image of Figure 1, where the text "A bundle of" does not seem to be adequately performed.

Author response:
For object color, we present additional qualitative comparisons in **Figure 22** and **Figure 23**, showing that our propsed **R&B** is able to handle attribute binding correctly and generate the color according to text prompt, even when the attribute is not common. We also show the robustness of our method by varying nouns and attributes (e.g., color) in **Appendix I (Figure 16)**. For object counts, we show visual comparison in **Figure 18**, we witness that the spatial accuracy of images generated by our method is even comparable with the training-based method. Please refer to **Appendix J** 'Qualitative' part for more analysis. Our proposed layout constraints not only provide accurate layout guidance, but also cope with semantic issues simultaneously, which helps the model to bind attributes with right nouns effectively, and helps to generate objects with correct number according to the layout instructions. As for the question that the text "A bundle of" does not seem to be adequately performed, we find that when the bounding box is associated with only the word "balloon," the model tends to generate only a single balloon inside the box. One simple solution is to incorporate 'a bundle of' into the object phrase, letting spatial guidance imposing on the whole phrase, and multiple balloons are observed. You can refer to **Figure 1 (last row, last column)** and **Figure 24 (last row, last column)** , by changing the input form, we can intuitively observe that the semantics of "A bundle of" can be adequately performed.

Reviewer comment (Experiments):
How does this model perform when provided with more intricate prompts, such as "a pink rabbit with a white head," with the boundary pointing to both the "rabbit" and "head"?

Author response:
We present visual results on intricate prompts in **Appendix J**, where the prompt "a pink rabbit with a white head" is included. Three more similar intricate prompts are provided for visual comparison. Please refer to **Appendix K** 'Performance on intricate prompts' part for detailed discussion.

Reviewer comment (Evaluation):
I find the result in the last row of the rightmost image in Figure 4 perplexing. The image seems to exhibit incomplete deformation. What factors may have led to this particular outcome?

Author response:
We think the model tries to synthesize view from the airplane porthole since the prompt specifies background as runway. Yet finally only a portion of the windows is generated, resulting in strange distortions in the upper half. The regions of the "running horse" guided by layout constraints do not exhibit any undesirable distortions. Commonly, when the layout guidance completely fails, the appearance of images may seem like **Figure 4 (last row, column 3)**, which seems notably out-of-distribution.

Reviewer comment (Novelty):
My main concern is unclarity in the conceptual differences of the proposed method to other methods. Training-free modification of attention map is not a new idea as other methods like (Chen et al., 2023; Xie et al., 2023; Phung et al., 2023) also design objectives that encourage the model to shift objects towards specified bounding boxes. These methods generally exhibit same motivation (align objects with bounding boxes) with similar implementation (shift cross-attention maps). It is not clear what makes the proposed approach conceptually different and where the performance gain comes from.

Author response:
**Q1: Unclarity in the conceptual differences of the proposed method to other methods.**

A1: Most diffusion-based T2I models inject external textual condition into the generative process through the cross-attention, the most effective approach for controllable generation with diffusion models is to leverage cross-attention manipulation. From this perspective, most works share the same idea to achieve zero-shot grounded generation, that is, to align the cross-attention maps of diffusion models with the grounding input. In this work, our motivation is primarily based on an observation of the two critical issues previous zero-shot grounded generation methods face with: (1) previous methods fail to provide accurate spatial guidance, (2) they inherit inconsistencies from the original T2I model. Detailed discussions on previous methods **Layout-guidance** [1], **BoxDiff** [2], and **Attention-refocusing** [3] can be found in the **Appendix B**.

 **As for the spatial accuracy**, **BoxDiff** and **Attention-refocusing** borrow insights from previous work [4] to amplify the top values of cross-attention maps within the bounding boxes, and reduce the top values of the outer regions to control the generative layout of different concepts. Yet the min-max optimization makes the layout guidance unstable and sensitive to the starting latents. See **Figure 9**, **Figure 17 (bottom half)**, **Figure 18** for example, we observe that some samples align with the box constraints, while others present inaccuracy of shape and localization. And optimizing top values only ensures that the most discriminative parts of the object are contained within the bounding boxes, while other parts of the object may not adhere to the layout constraints. The score function of **Layout-guidance** encourages the concentration of attention within the bounding boxes, which may lead to tricky solution, i.e., the objects are much smaller than the boxes, but the value of the score function are close to 0. Examples can be found in **Figure 4 (column 5, row 4)**, where the cup is much smaller than the box, **Figure 9 (column 3 and 6, row 4)**, where the globe is much smaller than the box. Moreover, when peak attention values appear within the bounding boxes, the value of score function experiences a sudden drop, which may affect the accuracy of guidance. See the upper half of **Figure 9 (row 3)**, the trunk is within the bounding box but the branches and leaves are outside the box. Quantitative results (**Table 5 in Appendix F**) on box alignment are consistent with aforementioned discussions. 

**As for the semantic consistency**, one key insight is that if the attention map is too smooth, the object may lack discriminative regions, which can lead to missing objects or attribute leakage. Thus, previous works [4,5] essentially increasing the variance of cross-attention maps to tackle the semantic issues. Both **Layout-guidance** and **BoxDiff** enhance the response of cross-attention within the layout equally, which do not effectively boost the total variance, thus lead to missing objects (**Figure 4, 9** and **18**) and concept fusion (**Figure 9** and **18**). The self-attention refocusing introduced by **Attention-refocusing** aims to ensure semantic consistency within the bounding box but does not consider the conditional textual information, it may have negative effects on guidance when the generative layout deviates significantly from the ground truth, resulting in tough semantic issues (**Figure 4, 9** and **18**). Results in **Table 1** in **Section 4.2** reveal that previous methods suffer from fatal semantic issues.

(**Continue from A1**) By comparison, by directly addressing two key issues (spatial, semantic), our proposed **R&B** achieves significant performance improvements. (1) The region-aware loss aims to bridge the gap between the consecutive attention maps and the binary masks. Different from previous methods, we directly model the divergence between generative layout and ground truth by dynamic thresholding and box selection, and gradually optimize the predicted boxes towards the target boxes. The IoU between pseudo boxes and ground truth modulates the guidance scale, which mitigates the tricky solution encountered in **Layout-guidance**. The discrete sampling makes each foreground elements contribute equally to the score function, avoiding the affect of peak response.  Hence, the generative outcomes better align with the ground truth layout. (2) The boundary-aware loss helps refine boundaries of attention maps in object regions, this essentially enhance the variance within the layout, and enhances the semantic consistency to the textual prompts, and further ensures adherence to layout constraints. We achieve high spatial alignment and semantic consistency simultaneously, making our method distinct from previous methods.

Reviewer comment (Theory):
Could the authors please elaborate their explanations in intro: 1) "previous methods fail to provide accurate spatial guidance" - why does this happen based on their design, and why does this not happen in R&B method based on its design? 2) "inherit inconsistencies from the original T2I model" - doesn't this also happen in R&B? Or what part of the design helps to avoid them?

Author response:
(**Continue from A1**) By comparison, by directly addressing two key issues (spatial, semantic), our proposed **R&B** achieves significant performance improvements. (1) The region-aware loss aims to bridge the gap between the consecutive attention maps and the binary masks. Different from previous methods, we directly model the divergence between generative layout and ground truth by dynamic thresholding and box selection, and gradually optimize the predicted boxes towards the target boxes. The IoU between pseudo boxes and ground truth modulates the guidance scale, which mitigates the tricky solution encountered in **Layout-guidance**. The discrete sampling makes each foreground elements contribute equally to the score function, avoiding the affect of peak response.  Hence, the generative outcomes better align with the ground truth layout. (2) The boundary-aware loss helps refine boundaries of attention maps in object regions, this essentially enhance the variance within the layout, and enhances the semantic consistency to the textual prompts, and further ensures adherence to layout constraints. We achieve high spatial alignment and semantic consistency simultaneously, making our method distinct from previous methods.

Reviewer comment (Novelty):
Similarly, it is not clear to me what is the technical novelty of the proposed method. The narration of the method is structured as the adaptation of existing methods (Li et al., 2023b, Li et al., 2023b) to the task of interest without much new analysis. Could the authors please elaborate what could be a broader-impact technical lesson or insight for the community from Sec. 3?

Author response:
**Q2: The technical novelty of the proposed method.**

A2: Some technical novelties can be found in the summary of our method in A1. The previous work **Divide&Bind** [5] brings a valuable insight that enhancing the variance of cross-attention can effectively address semantic issues related to the corresponding concepts. Inspired by this, we design the boundary-aware loss for zero-shot grounded generation. Indeed, our common goal is to address semantic issues, but from the implementation perspective, we are fundamentally different from [5]. [5] simply enhances the overall variance of the cross-attention map. In contrast, our method adopts and sharpens the Sobel boundaries within the boxes while reducing the variance of objects outside the boxes. Additionally, our guidance is modulated by the predicted IoU to prevent excessive optimization that could impact accuracy and quality. Furthermore, our method employs two loss functions, which can not be structured as an adaption of [5]. 

In summarry, (1) the clear motivation of the proposed two layout constraints (region-aware loss for layout accuracy, boundary-aware loss for semantic consistency), (2) the exhaustive analysis and discussions on the effects of each operation in **Section 3**, illustrated in **Section 4.3**, **Appendix C** and **E**  could have a impact for the community from the technical perspective. Further, and the promising improvement over existing methods both quantitatively and qualitatively (even comparable with training-based methods) and the vital problem (grounded generation without any data) may have a broader-impact for controllabe image generation with user-interface.

Reviewer comment (Experiments):
Why are there no comparisons to some other baselines, like GLIGEN?

Author response:
**Q4: Why are there no comparisons to some other baselines, like GLIGEN?**

A4: We consider our method to be applicable without training data, and therefore, we compare it with other methods under the same setting. However, we notice that many reviewers are interested in a comparison with training-based methods. In response, we include comparison with a typical training-based method, **GLIGEN** [6], in the revision to enhance the completeness of the paper. We add quantitative and qualitative comparisons with **GLIGEN** in **Appendix J**. You can see **Figure 17-19**, **22**, **23** and **Table 7** in our revised version for details of comparison results and discussions.

Reviewer comment (Experiments):
Comparison with training-based methods are lacking. For example, GLIGEN (which the paper cites) is a training-based method for grounded text-to-image generation. As they solve the same task, it would be helpful to compare with those methods qualitatively and quantitatively to reveal the strength and weakness of both approaches (and also for the sake of completeness).

Author response:
We thank the reviewer for the useful suggestion. For sake of completeness of our experiments, we add quantitative and qualitative comparisons with GLIGEN [1] in **Appendix J**. See **Figure 17-19**, **22**, **23** and **Table 7** in our revised version for details. One key observation is that training-based methods (like GLIGEN) tend to exhibit strong biases according to the training data, that is, it behaves well on common prompts (higher accuracy on spatial and size categories in **Table 7**), while fails to generate images properly on unusual prompts (poor accuracy on color categories in **Table 7**, semantically entangled concepts in the generated images). These implies that although well-trained on annotated data, GLIGEN still suffers from severe semantics issues inherited from the original T2I Diffusion model, and struggles to generate out-of-domain images (**bottom half of Figure 17**, **Figure 22-23**). Further, images generated by GLIGEN sometimes present undesirable watermarks inherited from the COCO training-set, for example, **Figure 18 (row 3 of the bottom half)**, **Figure 19 (row2 and row 6)**. These indicates that poisons in the training data may also harm the quality of training-based grounded generation methods.

Reviewer comment (Experiments):
I want the authors to comment on the runtime of the proposed method. How is it compared to the vanilla generation process (i.e., without guidance). In addition, how sensitive is the method to hyper-parameters?

Author response:
The number of guidance steps and optimization iterations per step are the critical factors that affect computational latency. By default, we adopt the DDIM scheduler with 50 denoising steps. We perform layout-guidance at the first 10 steps, and optimize the noisy latent for 5 iterations per step. Essentially, we add 50 forward processes and 50 backward process to the vanilla denoising procedure (50 forward steps). We list the practical runtime of several training-free methods mentioned in our manuscript on a single NVIDIA RTX-4090 GPU as below:

| &emsp; Stable Diffusion &emsp; | Layout-guidance | &emsp; BoxDiff &emsp; | Attention-refocusing | &emsp; R&B &emsp;|
| :------: | :---: |  :---: |  :---: |  :---: |
|     7.9s    |    25.3s     |  31.6s|  51.4s | 29.7s  |

Additional time cost is relatively acceptable because, with vanilla diffusion alone, it may not be possible to generate an image that aligns with the user's grounding input, even after generating dozens of images. Nonetheless, further analysis regarding the trade-off between guidance iterations and the image quality for zero-shot grounded generation is meaningful, and may be comprehensively discussed in our future work.

Reviewer comment (Experiments):
Is the method able to handle overlapping objects? Most examples in the paper show non-overlapping bounding boxes. I am curious about how cross-attention guidance behaves when two or more objects compete for the same pixel.

Author response:
Some of the visual examples in our original manuscript present objects with overlaps, **Figure 1 (columns 2-4)**, **Figure 3**, **Figure 6 (row 1)**, **Figure 8 (row 1-2)**, **Figure 11** and **Figure 13-14**. We also illustrate more samples in our revised version, see **Figure 17 (upper half)**, **Figure 18 (upper half)**, **Figure 24 (row 1, 4, and 5)**. Some hard cases are also presented, for example, **Figure 20** (overlaps between multi objects), **Figure 22-23** (intricate prompts). One intuitive attention map visualization is shown in **Figure 2** of **Section 3** with prompt "A handbag placed on the wooden bench", we observe that normalized attention map of "bench" is hollow because the corresponded region is overlapped by the "handbag". Detailed discussion can be found in **Appendix K**. When users provide reasonable grounding inputs, our method effectively activates the diffusion priors on object position and relations, resulting in desirable generative outcomes.

Reviewer comment (Experiments):
I want to understand the performance of the method with respect to object size. Since the attentions maps have relatively low resolution (down to 16x16), I am wondering whether the losses (especially the boundary-aware loss) are still effective in the presence of small objects. Some qualitative results and visualizations would be helpful.

Author response:
Generating small objects is relatively difficult for the original Stable Diffusion, because the resolution of the noisy latents is 64$\times$64. In our implementation, we adopt the Stable Diffusion V-1.5 as our base model, which injects external conditional information with cross-attention at resolution 8, 16, 32, 64 respectively. Most cross-attention control methods [2, 3, 4] design score functions on the cross-attention maps of resolution $16\times16$, this may results in object shifts from the layout constraints when scaling up to the image space. As mentioned in **Section 3 (Page 4)**, we upscale the cross-attention maps to resolution $64\times64$ and aggregate them to a unified map, we then calculate the score functions for layout-guidance. This helps mitigate the aforementioned issue of object shift. Some visual examples on small ojbects are shown in our revised manuscript, and can be found in **Figure 18**, **Figure 22**, **Figure 23 (bottom half)**, **Figure 24 (row 4)**, we also accordingly discuss our performance on small objects in **Appendix K** and **Figure 21**. Generating grounded small ojbects is indeed a tough setting for training-free methods, although showing robustness in handling smaller boxes, our method fails when the size of given layout constraints is too small (**last column in Figure 21**).