Paper: Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive (ICLR-2024)

Reviewer comment (Evaluation):
As mentioned in the Failure Cases, when editing the attribute of one object, it could affect the other objects as well. It is claimed to be inherited from Stable Diffusion.

Author response:
We believe that the attribute/concept binding issue, i.e., editing the attribute of one object could affect the other objects as well, is an inherited problem from Stable Diffusion (SD), as also been reported in prior SD based text-to-image works [1,2,3,4]. Using words to describe detailed changes at specific locations can be lengthy and inaccurate, therefore realizing such edit via text can be difficult. SD still cannot accomplish the task perfectly, either ignoring the text or mapping the text description to different objects. With the further scaling of the denoising UNet, e.g., SDXL [5], such issue might be mitigated but still exists as mentioned in their limitation section. One hypothesis for the binding issue lies in the pretrained text encoders, which tend to compress information into a fixed number of tokens. Employing inference time optimization [2,3] could resolve the problem to a certain extent. Nevertheless, we believe this is an interesting yet open research question. 

[1] Feng, Weixi, et al. "Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis." ICLR. 2022.

[2] Chefer, Hila, et al. "Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.

[3] Li, Yumeng, et al. "Divide \& bind your attention for improved generative semantic nursing." arXiv preprint arXiv:2307.10864 (2023).

[4] Rassin, Royi, et al. "Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment." arXiv preprint arXiv:2306.08877 (2023).

[5] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).

Reviewer comment (Evaluation):
Despite thorough ablation tests, the paper does not give any insights on the experiments. The text in the experiments only describes the results instead of analyzing the phenomenon. Thus, I think it shows limited contribution to the community. I suggest to delete the plain description of the results, as we can all see from the tables and figure captions, but add more analysis and insights of why it can work.

Author response:
We thank the reviewer for pointing this out to help us further improve the manuscript quality. We will reiterate some of the insights presented in the Introduction and Method sections, and connect it better with the experiments. To provide a short summary of the important insights: ALDM improves upon the other methods from the training pipeline: the adversarial supervision explicitly leverages the label map condition; and the unrolling strategy can consider more sampling steps during the learning process, bridging the gap between training and the inference time sampling. The combination of both enables the model to well comply with the condition consistently over a time window, leading to the improvement on the alignment in the final result.

Reviewer comment (Evaluation):
It is mentioned in the limitations that editing one object may also affect others. However, in Figure 4, it shows better local controllability than ControlNet and FreestyleNet. Any insights why? Because of better pixel alignment? If that's the case, is better mIoU always means better local controllability? For example, can the one trained on frozen segmenter exhibits better local controllability?

Author response:
Our ALDM can improve the alignment between the synthesized image and the label map, which is beneficial for achieving more accurate local editing on a specific object area. Yet, the commonly observed binding issue from Stable Diffusion [1,2,3,4], i.e., editing one object may affect another object simultaneously, is still inherited. Despite ALDM improves on the pixel alignment due to the adversarial supervision and unrolling, the inherited binding issue cannot be eliminated completely, as shown in the limitation (Fig. 17). It can be seen that although the cars have been properly synthesized within the correct label map regions, extra editing such as color changes via text cannot be accurately enforced to the specific car. Improving alignment with the label map alone is not sufficient to improve the binding. One may seek to link the text and location in the label map or incorporate additional instructions. When using a pretrained frozen segmenter, the generator learns a mean class mode and heavily loses the diversity, which influences its text controllability. Our intuition is aligned with the quantitative results: we measured the text editability via the TIFA score when using a frozen segmenter, where it only achieves TIFA=0.556, which is significantly lower than ALDM with TIFA=0.888 as presented in Table 2.

Reviewer comment (Experiments):
Why larger number of unrolling steps is always better? Have you tried any K>9? Do all steps contribute the similar gradient magnitude or some of them is more important? What if we omit some of the earlier steps?

Author response:
Larger unrolling step K can involve more sampling steps in the learning objective, better imitating the inference time sampling and encouraging more consistent adherence to the label map condition. Due to the linear increase of the unrolling overhead, we limit the number of steps to $K=9$ in our experiments. Nonetheless, post submission, we also experimented with more unrolling steps on ADE20K, using $K=15$, where mIoU was further improved (see Table 5 in Appendix, and listed below).

| ControlNet | K=0 | K=6 |K=9 |K=15 |
|--|--|--|--|--|
| 30.4 |34.0|34.5|36.0|36.9|

We simply average the discriminator loss across different unrolling steps, i.e., equal contribution, as formulated in Eq. (10), similar to simplified diffusion MSE loss proposed in DDPM [6]. Nevertheless, it is an interesting future direction to develop more advanced time-dependent weighting to further improve the algorithm. We thank the reviewer for the inspiring feedback, and we have included this in our newly added "Future Work" section (see Appendix E).
During training, the initial step $t$ before the unrolling is thus randomly sampled, since early denoising steps are also important for determining the content of the image, as observed in prior work [7]. 

[6] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." NeurIPS (2020): 6840-6851.

[7] Hertz, Amir, et al. "Prompt-to-Prompt Image Editing with Cross-Attention Control." ICLR. 2022.

Reviewer comment (Experiments):
it's unclear if careful tuning of a frozen segmentation network's impact on the total loss wouldn't be competitive to the proposed adversarial approach. I.e. in Table 3, the frozen UperNet achieves the best mIoU, but much worse FID - consistent with the hypothesis that the impact of the segmentation network is just too strong. Verying the impact of the segmentation loss relative to the diffusion loss while fine-tuning would clarify this.

Author response:
Our initial assumption was the same, however, we couldn't manage to resolve the issue after tuning the hyperparameters. As shown below, when decreasing the weighting $\lambda$ of the frozen segmenter loss, the mIoU drops heavily while the FID has minor improvement. The training loss is the same as formulated in Eq. (7), which is a combination of diffusion MSE loss and the segmenter loss. The only difference is for our proposed ALDM the segmenter as a discriminator is also updated with the diffusion model instead of being frozen. In the adversarial game, it is important that the discriminator and generator strike a balance to continuously improve each other. While using a frozen segmenter, the discriminator loses the chance to update itself, and the powerful generator (Stable Diffusion UNet) can freely find a cheating path, leading to the mode collapse issue, as observed in GANs [1]. The mean mode yields limited diversity thus high FID, and it's easier for the segmenter to classify the class mean mode thus the high mIoU. 

| Method| mIoU | FID|
|--|--|--|
| $$\lambda=1e-2$$ |50.8|40.2|
| $$\lambda=1e-4$$ |39.2|39.8|
| ALDM (Ours)|36.0|30.2|

[1] Thanh-Tung, Hoang, and Truyen Tran. "Catastrophic forgetting and mode collapse in GANs." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

Reviewer comment (Theory):
The introduction gave a good intuition for the whole method. However, the combination of diffusion models with adversarial methods remains ad-hoc, and there's no theoretical justification for the validaty of the approach given. It should be either worked out, or at least added for future work that propoer understanding of diffusion-adversarial coupling should be investigated (to make these methods work best together).

Author response:
We thank the reviewer for acknowledging that the introduction provides a good intuition, and for providing the constructive suggestion. We acknowledge the value of having a carefully derived mathematical framework for the problem. In the proposed adversarial training, the denoising UNet of the diffusion models can be viewed as the generator, the segmentation based model acts as the discriminator. For the diffusion model, the discriminator loss is combined with the original reconstruction loss, to further explicitly incorporate the label map condition. Prior works [2,3] have combined VAE and GAN, and hypothesized that they can learn complementary information. Since both VAE and Diffusion Models (DMs) are latent variable models, the combined optimization of DMs with an adversarial model follows this same intuition - yet with all the advantages of DMs over VAE.

The combination of the latent diffusion model with the discriminator is thus, in principle, a combination of a latent variable generative model with adversarial training. In our submission, we  specified the adversarial loss such that relates our model to optimizing the expectation over ${x}_0$ in Equation (6), while for the diffusion model, we refer to the MSE loss defined on the estimated noise. In the original DDPM paper [4], it has been shown how the MSE loss in our Equation (2) relates to optimizing ${x}_0$ with respect to the approximate posterior $q$ by optimizing the variational lower bound on the log-likelihood (refer to their equations (1)-(14), where our Equation (2) corresponds to their Equation (14)). For the relationship of the MSE loss to the approximate posterior optimization, we can in particular refer to Equation (5) in DDPM [4]. Our resulting combination of loss terms in Equation (7) can thus be understood as to optimize over the weighted sum of expectations on ${x}_0$ given in our Equation (6) and Equation (5) in DDPM [4]. 

We thank the reviewer for pointing this out and help us to improve the manuscript quality. We rewrote our Equation (2) such that its relationship to optimizing an expectation over $x_0$ becomes clearer, and we have included the above discussion of the theoretical framework in the new section "Discussion \& Future Work" (see Appendix E). May we know if this addresses the concern?

[2] Larsen, Anders Boesen Lindbo, et al. "Autoencoding beyond pixels using a learned similarity metric." International conference on machine learning. PMLR, 2016.

[3] Xian, Yongqin, et al. "f-vaegan-d2: A feature generating framework for any-shot learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[4] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.

Reviewer comment (Presentation):
Figure 1 lowest right image: this example shows that the method doesn't yet work perfectly. The top part of the truck is a building (see the windows and the 'roof structure'). It should be pointed out in the paper that some failed cases still persist (even if harder to see).

Author response:
We are grateful for the reviewer's valuable suggestions and have incorporated the modifications accordingly to improve the manuscript. As mentioned in (W.4), we agree that "more attention has been devoted to leveraging pretrained knowledge for the L2I task and using diffusion models" is an important point, and have highlighted the relevant part in the "Introduction" in blue now. To address (W.3), we additionally point out that ALDM is not yet perfect in the failure cases section (see Appendix D).

Reviewer comment (Presentation):
In "Related Work" you write "more attention has been devoted to leveraging pretrained knowledge for the L2I task and using diffusion models"; I think this is an important point, and should already be part of the motivation of the general method in the Introduction.

Author response:
We are grateful for the reviewer's valuable suggestions and have incorporated the modifications accordingly to improve the manuscript. As mentioned in (W.4), we agree that "more attention has been devoted to leveraging pretrained knowledge for the L2I task and using diffusion models" is an important point, and have highlighted the relevant part in the "Introduction" in blue now. To address (W.3), we additionally point out that ALDM is not yet perfect in the failure cases section (see Appendix D).

Reviewer comment (Experiments):
why don't you train a diffusion model from scratch for the task of L2I and T2I simultaneously? Recent segmentation models are very powerful, and can produce the segmentation maps needed to augment the dataset (e.g. LAION-5B). I'd expect that to work best.

Author response:
We agree that training a diffusion model for L2I and T2I jointly is a promising idea, and the recent segmentation models such as SAM have demonstrated impressive zero-shot performance, and in many cases can provide reliable label maps for training. Unfortunately, in our lab we have very limited computational resources, thus large-scale training, e.g., 256 A100 GPUs with 150000 hours reported for Stable Diffusion, is far beyond our computing capability. In contrast, fine-tuning only requires 2 A100 GPUs, which is affordable for us. As not having enough resources to train Stable Diffusion from scratch is not an uncommon status for many labs in the world, developing fine-tuning techniques is valuable as well. Nevertheless, we thank the reviewer's valuable suggestion, and we have incorporated this into the newly added "Future Work" discussion in Appendix E.

Reviewer comment (Experiments):
for the case of a frozen pre-trained segmenter: did you train a diffusion model plus an additional loss for the segmentation model? That is surprising, because I wouldn't have expected the diffusion model to collapse in such case (the same training method/loss is still there).

Author response:
Our initial assumption was the same, however, we couldn't manage to resolve the issue after tuning the hyperparameters. As shown below, when decreasing the weighting $\lambda$ of the frozen segmenter loss, the mIoU drops heavily while the FID has minor improvement. The training loss is the same as formulated in Eq. (7), which is a combination of diffusion MSE loss and the segmenter loss. The only difference is for our proposed ALDM the segmenter as a discriminator is also updated with the diffusion model instead of being frozen. In the adversarial game, it is important that the discriminator and generator strike a balance to continuously improve each other. While using a frozen segmenter, the discriminator loses the chance to update itself, and the powerful generator (Stable Diffusion UNet) can freely find a cheating path, leading to the mode collapse issue, as observed in GANs [1]. The mean mode yields limited diversity thus high FID, and it's easier for the segmenter to classify the class mean mode thus the high mIoU. 

| Method| mIoU | FID|
|--|--|--|
| $$\lambda=1e-2$$ |50.8|40.2|
| $$\lambda=1e-4$$ |39.2|39.8|
| ALDM (Ours)|36.0|30.2|

[1] Thanh-Tung, Hoang, and Truyen Tran. "Catastrophic forgetting and mode collapse in GANs." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

Reviewer comment (Novelty):
The technical contributions of this manuscript are limited. Adversarial supervision on the semantic maps is identical as in OASIS. Multistep unrolling is computationally very expensive (scales linearly and hence can take up x9 longer), and has a small effect on the performance.

Author response:
> (1)  The technical contributions of this manuscript are limited. 

To the best of our knowledge, our adversarial supervision with the multistep unrolling strategy has not been proposed before for improving layout-to-image diffusion models. Our approach is indeed inspired by OASIS, which we have adequately acknowledged in the paper, i.e., cited it in multiple sections (introduction, related work, method, etc.) in the main paper.  However, incorporating the broad concept of discriminator loss into a completely different type of generative model, i.e., diffusion model, is by no means trivial. 

The traditional diffusion model suffers from poor alignment due to the suboptimal training objective without explicit supervision based on the label map, and its sampling characteristics also make consistent adherence to the conditional layout with time difficult. Little attention has been paid to improve the original diffusion reconstruction loss. Therefore, the proposed adversarial supervision and multistep unrolling is a novel training design for diffusion models, to explicitly encourage consistent alignment to the label map over a time window, closing the gap with inference time sampling. **We believe our work focuses on the important yet underexplored aspects of diffusion models, i.e., the training strategy**, and we demonstrate the effectiveness of our proposal on different recent diffusion model-based methods, e.g., OFT (NeurIPS'23), T2I-Adapter (ArXiv'23-Mar) and SOTA ControlNet (ICCV'23 Best Paper) (see Table 1 and below). Especially, the multistep unrolling is  tailored for diffusion models, considering its sampling characteristic, which is significantly different from GANs.

| Method| Cityscapes mIoU | ADE20K mIoU|
|--|--|--|
| OFT |48.9|24.1|
| OFT + Ours |58.8|31.8|
| T2I-Adapter|37.1|24.0|
| T2I-Adapter + Ours|50.1|29.1|
| ControlNet |55.2|30.4|
| ControlNet + Ours |63.9|36.0|

Regarding the computing overhead, as mentioned in the "Implementation details", we only apply the unrolling every 8 optimization steps, such that the additional overhead is manageable. It is important to mention that multistep unrolling is a training strategy that can largely improve the alignment with the label map, coming at no additional cost at inference time, which is more important for most applications.

Reviewer comment (Evaluation):
The results (while improving over the diffusion baselines) are behind OASIS, a GAN from 2020.

Author response:
> (2) The results (while improving over the diffusion baselines) are behind OASIS, a GAN from 2020.

Our goal is to improve text-to-image (T2I) diffusion models for semantic image synthesis.  We agree with the reviewer that this class of generative models underperforms GANs in terms of alignment, which we have shown in Table 8 in the Appendix. However, there are many benefits of using large-scale T2I diffusion models for this task, such as powerful text controllability. Our ALDM can naturally synthesize novel samples, which can be used for downstream applications such as improved domain generalization (see Table 4 and Fig. 5). In contrast, GAN alone can only synthesize samples resembling the trained domain. We believe our proposed training strategies can inspire the community and can be combined with advancing future diffusion model architecture designs, which are orthogonal to our work, leading to better results in the future.

Reviewer comment (Novelty):
The motivation to use a strong text-to-image model and adapt it for semantic image synthesis is flawed. The paper motivates it by enabling text-conditioned content and style transfer, but the semantic mask completely specifies the content. Hence, the application is reduced to text-guided style and color transfer.

Author response:
> (3 & 5) Motivation to use a strong text-to-image model and adapt it for semantic image synthesis is flawed. The paper motivates it by enabling text-conditioned content and style transfer, but the semantic mask completely specifies the content. Hence, the application is reduced to text-guided style and color transfer. A simple baseline combining OASIS and a state-of-the-art style transfer model should be considered. 

Our ALDM is not only capable of global style changes but also local editing via text, as presented in Fig. 1, 4, 9-13. We have verified the effectiveness of synthesized novel samples on domain generalization task (in Table 4 and below), where we compare against GAN-based style transfer ISSA [1]. We thank the reviewer for the inspiring comment, we additionally compare ALDM against text-guided editing approach PnP-Diffusion [2] below as well. ALDM outperforms both methods and largely improves the generalization performance of the segmenters. A visual comparison between ALDM and the style transfer method has been added in the revision in Fig. 14, where it can be seen that ours can produce more realistic results with faithful local details, e.g., mud, snow. In contrast, the majority of the style transfer methods mix two images on the global color style, while the local details cannot be faithfully transferred (as also seen from Fig. 14). 

| Method| ACDC - HRNet | ACDC  - SegFormer|
|--|--|--|
| Baseline (Cityscapes) |41.48|47.04|
| ISSA (GAN-based style transfer) |50.05|52.45|
| PnP-Diffusion (Text-guided editing) |48.74|48.71|
| ALDM (Ours)|53.03|53.78|

[1] Li, Yumeng, et al. "Intra-& extra-source exemplar-based style synthesis for improved domain generalization." International Journal of Computer Vision (2023): 1-20.

[2] Tumanyan, Narek, et al. "Plug-and-play diffusion features for text-driven image-to-image translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
A simple baseline combining OASIS and a state-of-the-art style transfer model should be considered.

Author response:
> (3 & 5) Motivation to use a strong text-to-image model and adapt it for semantic image synthesis is flawed. The paper motivates it by enabling text-conditioned content and style transfer, but the semantic mask completely specifies the content. Hence, the application is reduced to text-guided style and color transfer. A simple baseline combining OASIS and a state-of-the-art style transfer model should be considered. 

Our ALDM is not only capable of global style changes but also local editing via text, as presented in Fig. 1, 4, 9-13. We have verified the effectiveness of synthesized novel samples on domain generalization task (in Table 4 and below), where we compare against GAN-based style transfer ISSA [1]. We thank the reviewer for the inspiring comment, we additionally compare ALDM against text-guided editing approach PnP-Diffusion [2] below as well. ALDM outperforms both methods and largely improves the generalization performance of the segmenters. A visual comparison between ALDM and the style transfer method has been added in the revision in Fig. 14, where it can be seen that ours can produce more realistic results with faithful local details, e.g., mud, snow. In contrast, the majority of the style transfer methods mix two images on the global color style, while the local details cannot be faithfully transferred (as also seen from Fig. 14). 

| Method| ACDC - HRNet | ACDC  - SegFormer|
|--|--|--|
| Baseline (Cityscapes) |41.48|47.04|
| ISSA (GAN-based style transfer) |50.05|52.45|
| PnP-Diffusion (Text-guided editing) |48.74|48.71|
| ALDM (Ours)|53.03|53.78|

[1] Li, Yumeng, et al. "Intra-& extra-source exemplar-based style synthesis for improved domain generalization." International Journal of Computer Vision (2023): 1-20.

[2] Tumanyan, Narek, et al. "Plug-and-play diffusion features for text-driven image-to-image translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Writing):
Layout-to-image is usually referred to as a task that transforms a list of bounding boxes and class labels into an image. This paper tackles semantic image synthesis where the input is a label mask (each pixel is labelled).

Author response:
> (side remarks 1) Layout-to-image is usually referred to as a task that transforms a list of bounding boxes and class labels into an image. This paper tackles semantic image synthesis where the input is a label mask (each pixel is labelled)

We thank the reviewer for the references. We agree that the terminology is not well-defined, and we acknowledge that the semantic image synthesis is also correct [3,4]. Since we focused on diffusion models, as also clearly stated in the title, we have chosen to inherit the terminology from the recent FreestyleNet: Freestyle Layout-to-Image Synthesis (CVPR'23) which is a diffusion model conditioned on label maps. Nevertheless, we thank the reviewer for pointing this out, and we have now incorporated this in the related work in the revision.

[3] Schönfeld, Edgar, et al. "You Only Need Adversarial Supervision for Semantic Image Synthesis." International Conference on Learning Representations. 2020.

[4] Wang, Yi, et al. "Image synthesis via semantic composition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Reviewer comment (Experiments):
What is the difference between Control-Net + Adv Supervision and multi-step rolling in Table 1 compared with ALDM?

Author response:
There is no difference between Control-Net + Adv Supervision + multi-step rolling and ALDM in other tables. In general, ALDM (adversarial supervision plus multi-step unrolling) can be seen as an improved training pipeline for L2I diffusion models, which can be combined with different adaptation architecture designs as shown in Table 1. In other tables, we use ControlNet as the default model combined with adversarial supervision and multistep unrolling, to represent ALDM.  We thank the reviewer for pointing this out, and we have clarified accordingly in the revision (on Page 7 after Table 1).

Reviewer comment (Evaluation):
Can you show some outputs of the discriminator? Since it works in the latent space, we need to be more careful about what is happening inside.

Author response:
We have added some discriminator predictions in Fig. 15. The discriminator takes the input of the decoder, which actually turns the latents into the pixel images, as mentioned in the "Implementation details". From Fig. 15 the discriminator can categorize some regions as "fake" class (in black), meanwhile, it is also fooled by some areas, where the discriminator produces reasonable segmentation predictions matched with ground truth labels. Therefore, the discriminator can provide useful feedback to the generator (diffusion model) that it should produce realistic results meanwhile complying with the given label map.