Paper: Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis (ICLR-2024)

Reviewer comment (Reproducibility):
Correct me if I have misunderstood. I'm confused about the details of the soft guidance. I understand that the work wants to leverage the mask map to selectively impose the foreground embedding and background embedding for the cross-attention mechanism. However, I'm uncertain as to whether this process should take place during the computation of similarity or subsequent to it. I would appreciate it if the author could elucidate the specific dimensions and computational details related to both the cross-attention mechanism and the soft guidance technique, to enhance reader comprehension. Please refer to the questions for additional context on my confusion.

Author response:
[W1] Firstly, we would like to address your questions on soft guidance and why the mask is utilized on the calculated similarity rather than during the actual computation of similarity. Assuming “during the computation of similarity” means **before** $S$ is rescaled with softmax and multiplied with $V$ (eg. $S'=(QK^T/\sqrt{d})\otimes M'$), and “calculated similarity” means **after** S is rescaled with softmax and multiplied with $V$ (eg. $S'=\operatorname{softmax}(QK^T/\sqrt{d})\otimes M'$), we would like to clarify that soft guidance is **indeed applied during the computation of similarity, or before the softmax operation.** It seems that this confusion stems from us omitting the softmax operation in the paper, where we state that “…the attention operation $S' \cdot V$ is completed.”, and whereas we intended $S$ to be understood as the raw similarity scores, readers would assume that $S$ means the scaled similarity score (or attention score, depending on sources). We thank you for your detailed insight on this matter. We’ve updated the equations to be clear that soft guidance is indeed applied during the similarity computation, and the reshaped, flattened, repeated boolean mask $\varphi(M) \in \{0,1\}^{i \times N}$ to $\varphi(M) \in \mathbb{B}^{i \times N}$ to avoid confusion.

[W1] Secondly, we would like to illuminate the specific dimensions used in the soft guidance technique. We bring up an exact example of how soft guidance is applied in the first cross-attention layer latents pass through, when generating an image of size $512 \times 512$.
In our implementation, we use 8 attention heads, and set $N=4$, or the number of tokens $\mathbf{y}\_\text{fg}$, $\mathbf{y}\_\text{bg}$ are projected to. Since we use the CLIP text encoder, the given text prompt is encoded into $77$ tokens with a dimension of $768$. By $\mathbf{y}\_\text{full} = \operatorname{concat}(\mathbf{y}\_\text{text}, \lambda\_\text{fg}\mathbf{y}\_\text{fg},\lambda\_\text{bg}\mathbf{y}\_\text{bg})$, $\mathbf{y}\_\text{full}$ becomes a tensor of shape `[1, 85, 768]`. The latent $\mathbf{z}\_t$ is a tensor of shape `[1, 4096, 320]` because it has been processed by the subsequent self attention layer. $\mathbf{z}\_t$ and $\mathbf{y}\_\text{full}$ are then projected into $Q$, $K$, $V$, with respective dimensions of `Q=[8, 4096, 40]` , `K=[8, 85, 40]`, and `V=[8, 85, 40]`. The raw similarity scores are calculated by $S=QK^T/\sqrt{d}$, resulting in `S=[8, 4096, 85]`. We then reshape, flatten, repeat the mask $M$ used for soft guidance (with an initial dimension of `[1, 512, 512]`, to a shape of `fg_mask=[1, 4096, 4]`, which serves as the mask for the foreground token part of $\mathbf{y}\_\text{full}$. We then take the inverse of the mask, `bg_mask=[1, 4096, 4]`, as the mask for the background token part of $\mathbf{y}\_\text{full}$. We then create a base mask of only ones `base_mask=[1, 4096, 77]`, and concat the three masks to create $M'$ of shape `[1, 4096, 85]`. Then the operation of $S\otimes M'$ is conducted across all attention heads (eg. masking), to get $S'$. Then, the rest of the attention operation, $\operatorname{softmax}(S')\cdot V$ is performed.

Reviewer comment (Evaluation):
Apart from the standard metrics used for evaluating generative models, I wonder if there exist specific metrics that can accurately assess the model's capability to control the three-dimensional placement of objects and localize global semantics, as these are the primary objectives of this study. While the Mean Absolute Error (MAE) between the ground truth depth map and the depth maps derived from the generated images may offer some insight into the model’s proficiency in 3D object placement, I am curious about how we might effectively gauge its ability to localize global semantics. Could there be other metrics or methods of evaluation that address this second capability?

Author response:
[W2] We have added an ablation study on a metric that gauges our model's ability to localize global semantics in our revised paper, section [A.4] of our appendix. To provide a brief summary: given two exemplar images, a mask used during soft guidance, and the resulting generated images, we take the cosine similarity between the CLIP image embeddings of the masked generated image and its corresponding semantic, and take an average value. We've included an additional experiment in section [A.4], Additional results on conflicting semantics, that demonstrates how the average cosine similarity directly correlates with the generated image, in terms of how well each semantic is localized to their corresponding regions. If the average cosine similarity is high, it implies that each semantic in each region is very similar to its exemplar image. This metric can be thought of as a variant of CLIPScore, which is the cosine similarity between the CLIP image embedding of an image and the CLIP text embedding of a prompt.

Reviewer comment (Theory):
In my understanding, the size of S is $i\times j$, where i is the number of queries, and j is the number of keys; the size of $W_K \dot y_{full}$ is $j \times C$ and the size of $W_Q \dot z_t$ is $ i \times C$. However, I am puzzled as to why $j$ needs to be greater than $2N$.

Author response:
[Q1, Q2] In your first question mentioning the dimensions of $Q$ with a size of $i \times C$, and $K$ with a size of $j \times C$, $C$ would equal to $40$, and it can be seen that the mask isn’t applied across $C$. Regarding to your question about why $j$ has to be greater than $2N$, this can be thought of as a notational choice since by concatenating $\mathbf{y}\_\text{text}, \mathbf{y}\_\text{fg},\mathbf{y}\_\text{bg}$, we expand the key space $j$, and the notion that $j$ has to be greater than $2N$ is used to define the all ones matrix $J \in \mathbf{1}^{i \times j-2N}$, which is the `base_mask=[1, 4096, 77]` mentioned above. We hope that this clarifies soft guidance, and we thank you for your detailed questions on its specifications. We appreciate your attention to this detail, and we’re glad to answer any additional questions.

Reviewer comment (Reproducibility):
Additionally, I am uncertain about whether the mask should be applied along the dimension of $C$. It perplexes me that the mask is utilized on the calculated similarity rather than during the actual computation of similarity.

Author response:
[W1] Firstly, we would like to address your questions on soft guidance and why the mask is utilized on the calculated similarity rather than during the actual computation of similarity. Assuming “during the computation of similarity” means **before** $S$ is rescaled with softmax and multiplied with $V$ (eg. $S'=(QK^T/\sqrt{d})\otimes M'$), and “calculated similarity” means **after** S is rescaled with softmax and multiplied with $V$ (eg. $S'=\operatorname{softmax}(QK^T/\sqrt{d})\otimes M'$), we would like to clarify that soft guidance is **indeed applied during the computation of similarity, or before the softmax operation.** It seems that this confusion stems from us omitting the softmax operation in the paper, where we state that “…the attention operation $S' \cdot V$ is completed.”, and whereas we intended $S$ to be understood as the raw similarity scores, readers would assume that $S$ means the scaled similarity score (or attention score, depending on sources). We thank you for your detailed insight on this matter. We’ve updated the equations to be clear that soft guidance is indeed applied during the similarity computation, and the reshaped, flattened, repeated boolean mask $\varphi(M) \in \{0,1\}^{i \times N}$ to $\varphi(M) \in \mathbb{B}^{i \times N}$ to avoid confusion.

Reviewer comment (Theory):
Calling the model "depth-aware" is misleading. I would rather say it learns to compose two spatial layouts and generate a coherent image. Using the teaser figure as an example, the cat can appear either in front of or behind the cake given the same depth maps, and similarly, the castle can either occlude or be occluded by the mountain. In other words, the exact ordering of objects is not induced by the depth maps. This phenomenon is likely because the depth produced by MiDaS is scale and shift invariant (i.e., it is not metric depth, and background can appear closer than foreground).

Author response:
[W1] We agree on your insight stating that the exact ordering of objects are not induced by the depth maps. The ability to order objects into their successive foreground/background stems from the proposed depth disentanglement training (DDT), where information about the otherwise occluded background is retrieved via our image triplets and distilled into our local fuser. The depth maps produced by MiDaS are indeed scale/shift invariant, but they do hold attractive properties that lead us to choosing them as our local condition, instead of other representations (eg. segmentation maps). Because depth maps are single channel representation of an object's shape and holds the relative depth representation of an object in the scope of the image, they hold an inductive bias that we sought attractive to leverage. To summarize, the "depth-aware" property of our model doesn't stem from the depth maps themselves (although we do leverage its inductive biases), but its ability to process each depth map in their respective foreground/background stream in our local fuser and place them accordingly.

Reviewer comment (Experiments):
Since all that matters is generating a coherent image, I would imagine that other types of spatial conditions (e.g., segmentation masks for both foreground and background) can work equally well if used for training. I encourage the authors to test this hypothesis, and design additional ablation experiments to fully reveal the behavior of their model.

Author response:
[W2] We agree that different spatial conditions can work equally well during training. We have added an additional experiment that elucidates this case in our revised paper, section [A.4] Ablation Study, of the appendix. We explore the use of canny edges instead of depth maps, due to the fact that canny edges hold very different inductive biases to that of depth maps. Canny edges are inherently binary, and they hold much more fine grained detail about an object while trading off the ability to represent the relative depth of a pixel. The added figure 13 of page 18 showcases the results, where we find that our model generalizes well to canny edges. As mentioned above, this ability to generalize well stems from our depth disentanglement training, where information about the background is distilled to the model, even if the representation used doesn't hold inductive biases about the relative depth of an object. However, the inductive biases of each representation does come into play when generating images. In figure 13 (a) and (c), depth maps generate images that capture the sense of depth present in the conditioned images, while canny edges result in the generated images often looking flat. The opposite holds, for in figure 13 (b) and (d), it can be seen that canny edges generate better fidelity samples to that of depth maps when conditioned on flat, illustration based images. This leads to the conclusion that while DDT is successful in distilling information of what object to place in front of and cover, our model leverages the inductive biases of the representations used. If the representation inherently holds information about the relative depth of an image, DDT also distills this information into our model, resulting in the ability to compose images in a depth aware fashion.

Reviewer comment (Presentation):
The illustration of soft guidance in Figure 2 is confusing. I personally prefer color coding of the attention maps to highlight the regions influenced by different tokens.

Author response:
[Q1] We have updated Figure 2 in the revised paper, and we agree that color coding the influenced regions provide a much clearer description. Thank you for the recommendation!

Reviewer comment (Evaluation):
Details about the reconstruction experiments (Figure 6) are lacking. It is unclear from the text what is the exact evaluation procedure. Also the MAE values reported in Table 2 is not meaningful, again because MiDaS does not predict metric depth. Please include qualitative comparison between the input and reconstructed depth maps.

Author response:
[Q2-1] The evaluation procedure of reconstruction experiments are as follows: given a ground truth image from the validation set, image triplets are constructed. Subsequently, background/foreground depth maps and CLIP image embeddings are extracted from the triplets, and are fed into our model as done during training. Since other baseline models do not accept either two depth maps or CLIP image embeddings, the depth maps and CLIP image embeddings from the ground truth images are extracted, and fed into said models. We thank you for pointing out the ambiguity, and we've added additional details to the revised paper, in section [4.2].

[Q2-2] To address the choice of including MAE of depth maps for our reconstruction experiments: the depth maps produced by MiDaS indeed doesn't predict metric depth, but we've included the MAE values as a metric of how our model "predicts" depth maps (extracted from the reconstructed images) given the ground truth depth maps, since our model and other baseline models were trained on said MiDaS outputs. We've also included an experiment that further elucidates the choice of including MAE in our revised paper, in section [A.3] of our appendix. The experiment portrays pairs of ground truth/reconstructed images and their extracted depth maps, randomly chosen from the top 50 pairs of the lowest/highest MAE values. In the experiment, it can be seen that lower MAE values directly correlate to how close the ground truth and reconstructed images are, and higher MAE values result in unfaithful reconstruction, and an overall decrease in sample fidelity. Although we agree that MiDaS depth maps do not predict metric depth, we find it suitable in terms of a metric that gauges the quality of reconstruction.

[Q2-3] We've included a qualitative comparison figure between the input and reconstructed depth maps in our revised paper, in section [A.3], additional details on Reconstruction of our appendix. The figure shows that while other models hold the overall shape of the objects, the depth maps of our model succeeds in capturing the relative depth of areas relative to the salient object. We thank you for your recommendation, for the added comparison sheds light on how the actual ground truth depth maps and reconstructed depth maps interact.

Reviewer comment (Experiments):
Type of conditions. (1) Is the model capable of applying different types of conditions? (2) Is the model capable of applying two different conditions simultaneously while recognizing the 3D relations? I am rather interested in these different situations especially considering that the authors only employ depth in the submitted paper. More examples or scenarios would be appreciated.

Author response:
[W1-1] Yes, our model indeed is capable of applying different types of conditions. We have added a section in our revised paper, section [A.4] Ablation Study of the appendix. We explore the usage of canny edges as an alternative to depth maps, due to the fact that while canny edges are spatial representations of an image, it trades off the ability to represent depth with more fine grained details. We include qualitative samples comparing generated images from canny edges and depth maps, in Figure 13. It can be seen that our model generalizes well to canny edges, a trait that stems from our depth disentanglement training where information about the salient object, and what's behind said object is distilled into our local fuser. However, canny edges hold inherently different inductive biases to that of depth maps. This can be seen playing a role in the generated samples, where canny edges yield better results for "flat" images (such as  graphics), but often fail in capturing a sense of depth. The opposite holds for depth maps, so we find that special cases might call for variants of our model trained on different conditions. We have also included a quantitative evaluation in the same section, where we compare the canny edge variant of our model with other baseline models, and find the results to be comparable. [W1-2] Because our model generalizes well to different conditions, we find no reason for our model to fail in generalizing different conditions for the foreground and background. While due to limitations in our computational resources at the moment, we plan to explore this possibility, starting with canny edges for the foreground and depth maps for the background. This initial choice comes from our newly found insights while experimenting with the canny edge variant of our model, where we find that canny edges provide more control over the intricate details of an object. We hypothesize that leveraging depth maps as background representations with canny edges as foreground representations might give us the best of both worlds; fine grained control over the salient object while additionally providing a sense of depth.

Reviewer comment (Experiments):
Image triplets. There is no visualization of the prepared image triplets for training. I am curious regarding the quality of foreground image, background image and foreground mask. It's also especially important to analyze the inpainted background image and how it would negatively affect the training and final outcomes.

Author response:
[W2] We kindly refer the reviewer to section [A.2] of the appendix of our paper, where we detail the process and provide visualizations of preparing the image triplets. We agree that the quality of the synthetic image triplets does influence the quality of the outcomes, especially the background image. Figure 9 of section [A.2], Details on CnC, of our appendix elucidates this factor about the image triplets, where we initially try out [1]LaMa, a widely utilized inpainting model. We show that the depth maps of the background images extracted by LaMa holds artifacts that don't provide information of what is behind the salient object, whereas background images extracted by SD don't. For depth disentanglement training to be fully effective, the background image has to distill information that is initially occluded by the salient object. If the quality of the background image were to be on par with the samples extracted by LaMa as in figure 9 (i.e. doesn't provide any information about the initially occluded background), it would hold the same effect of training our model with the same depth maps, as shown in figure 3 of our paper.

We also find the process of binary dilation to be crucial in inpainting with SD, because the masks $M$ extracted by the Salient Object Detection model we utilize are usually pixel-perfect. SD's inpainting module tends to leave certain edge artifacts when the masks just barely cover the targeted object, so we additionally use binary dilation to expand the mask. These extra steps while utilizing a much computationally demanding module (SD over LaMA) are taken to ensure the quality of our image triplets, for without it would negatively effect the quality of final outcomes, as you've pointed out.

[1]Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." _Proceedings of the IEEE/CVF winter conference on applications of computer vision_. 2022.

Reviewer comment (Evaluation):
Qualitative results. (1) What are the prompts for examples in Figure 3? (2) Through the visualization in Figure 3, 4 and 5, it's interesting to see that the final generated images do not fully reflect the foreground depth condition. Meanwhile, the background depth map is often ignored through the qualitative results.

Author response:
[W3-1] The examples in figure 3 are generated from the validation set of Pick-a-Pic and COCO-Stuff, from which the prompts were utilized. The prompts are:
3(a) - A beautiful dessert waiting to be shared by two people (COCO-Stuff)
3(b) - diamond minecraft block facing frontwards on a green background (Pick-a-Pic)
3(c) - cat knight, portrait, finely detailed armor, intricate design, silver, silk, cinematic lighting, 4k (Pick-a-Pic)
We have added the prompts to the revised paper, in figure 3. Thank you for highlighting this aspect.

[W3-2] A Depth map can be considered to be reflected in a generated image if the local placements (x, y axis) are reflected. In our case, we also consider the z-axis, so a depth map can be considered to be reflected in a generated image if the foreground depth map effectively occludes the background depth map in regions the foreground depth map is present. Figure 3 and 5 demonstrates these capabilities, and in figure 4, we only visualize a single depth map since the baseline models we compare our samples to only accept a single depth map condition. As detailed in section [4.2] of our paper, we condition the same depth map for our foreground/background streams, hence the visualization of a single depth map in figure 4.