Paper: Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints (ICLR-2024)

Reviewer comment (Experiments):
I would appreciate it if the authors give more details of the proposed framework given different conditions.

Author response:
Thank you for your feedback. We have added detailed information about our model architecture, hyper-parameter settings, and training details in Appendix B.3. In short, our model is a four-layer transformer with a modified adaptive layer normalization for time variable injection. This architecture is adopted from the source code of LayoutDM. We added two FNNs to encode and decode element vectors. The model size remains comparable. We use only one set of parameters for different conditional tasks. Known entries are fixed throughout the generation process using a binary mask as described at the end of section 2.2. For refinement task, we treat given noisy layout as a noisy latent states at a certain time step and finish the generation process as described at the end of section 4.2.

Reviewer comment (Experiments):
The ablation study can be extended to illustrate the contribution of two constraints. For example, Figure 3 only shows the visualization result of global alignment and overlap constraints. I would recommend studying the effectiveness of all sub-constraints, e.g., local and global alignment, quantitatively and qualitatively.

Author response:
Thank you for your suggestion. We have added more results in our ablation study and Appendix A. We observe that global alignment constraint performs comparably to local alignment. However, in theory, it should be less effective because local alignment constraints directly minimize the alignment metric. In addition, the global constraint is needed in post-processing for better visual quality.

Reviewer comment (Evaluation):
The limitations of the proposed method should be discussed. This can give good guidance for the application in real-world situations.

Author response:
Thank you for your suggestion. We have added the limitation in the revised conclusion section. The primary limitations include (some also apply to related methods): 1. It generates layouts as boxes, not segmentations. 2. the model is not background-aware or content-aware. 3. The maximum number of elements is limited. 4. It relies on a closed-vocabulary label set. 

Despite these limitations, the method shows promise in applications like layout-guided image generation, realistic scene text generation through TextDiffuser, and automated poster design, offering valuable contributions.

Reviewer comment (Experiments):
The key idea of this paper is to enable constraint optimization by formulating layout generation as conditional generation processes in continuous space. However, a post-processing step is still conducted. I was wondering about the advantage of the proposed method compared to those that directly apply the post-processing algorithm to the discrete diffusion models.

Author response:
A: Thank you for your suggestion. Appendix A includes the required results and discussion, comparing our model with LayoutDM with post-processing. In addition, we include results of LayoutGAN++, which also uses constraints in training and applies post-processing. 

We observe a noticeable FID increase after post-processing for the two competing methods. In contrast, LACE without post-processing outperforms LayoutDM with post-processing. Its performance further increases after post-processing while maintaining a stable FID. These differences are due to the challenge in post-processing, as discussed in section 2.3 of the main text, which is thresholding the misalignment in raw outputs. A small threshold for the alignment mask is insufficient to select severely misaligned pairs, while a large one damages the subtle layout structure, leading to a noticeable increase in FID. These additional results have further enhanced the quality of our paper.

Reviewer comment (Theory):
For the overlap constraint, the current loss pushes elements away from each other. However, in some cases, designers would intentionally use overlap to make the top element look closer to the viewer. For example, text elements are often located above the image elements in banner ads. The usefulness of such a constraint would be questionable by directly applying it to all elements in graphic designs.

Author response:
A: We apologize for any confusion regarding the overlap constraint. As noted in the global response, we acknowledge that the overlap constraint is unsuitable for training on the Rico dataset. Thus, we exclude it in the training of the model for Rico, which was noted in section 4.2. However, our method provided a way to add differentiable constraint functions in the diffusion training. The constraint could be replaced or removed as needed.

Reviewer comment (Evaluation):
Why the evaluation metrics do not consider overlap, while the aesthetic constraints contain the overlap in layout?

Author response:
A: We apologize for the confusion. As mentioned in our global response, we have added the requested alignment and overlap results in Appendix A. We believe these additional results have further enhanced the quality of our paper.

Reviewer comment (Reproducibility):
How to determine the time-dependent constraint weight and the threshold used in the post-processing stage?

Author response:
A: We have now included a detailed explanation with a supporting figure in Appendix B.1 to demonstrate our empirical approach for determining the time-dependent constraint weights. This weight is empirically set to the point, as demonstrated in the figure, where the corruption process has not introduced too much overlap. Thus, in the reverse process, the coarse structure of the layout has emerged. 

The post-processing threshold is empirically set to 1/64 of the scaled relative canvas size. This setting aligns with real dataset observations, where only ~0.5% of unaligned coordinate pairs has a smaller difference. We have included a more detailed explanation in Appendix B.2.

Reviewer comment (Reproducibility):
*Incomplete details for reproduction*: It appears that no training details of the models are reported in the paper and the appendix. It would be extremely helpful for future research for the author(s) the report details such as model architecture, noise schedule, loss schedule, constraint weights, etc, given that this continuous state-space of diffusion for layout generation was previously unexplored.

Author response:
Thank you for your feedback. We have added detailed information about our model architecture, hyper-parameter settings, and training details in Appendix B.3 and Appendix B.4. In short, our model is a four-layer transformer with a modified adaptive layer normalization for time variable injection. This architecture is adopted from the source code of LayoutDM. We added two FNNs to encode and decode element vectors. The model size remains comparable.

Reviewer comment (Evaluation):
*Limited inclusion of qualitative results and human-rater experiments*: It would further improve the paper if the author(s) can provide more qualitative results that demonstrates the differences between the proposed method and existing SoTA, and preferably include a lightweight human-rating study given that the metrics might not fully reflect human preference for layout generation.

Author response:
Thank you for recommending a lightweight human-rating study. We acknowledge its significance and aim to incorporate it in the final version of our paper. We have included demo qualitative results in Appendix C, comparing our model with LayoutDM and real data, which could be used for future human-rating studies. However, due to the time constraints of the rebuttal process, it is hard for us to complete the participant recruitment and execution of the study in time.

Reviewer comment (Evaluation):
*Missing common metrics*: While FID and MaxIoU is quite representative and established in the layout generation literature, it will be helpful if the author(s) can include alignment and overlap indices, which can further show the effect of the proposed aesthetic constraints.

Author response:
Thank you for your suggestion. As mentioned in our global response, we have incorporated the requested alignment and overlap indices results in Appendix A. We believe these additional results have further enhanced the quality of our paper.

Reviewer comment (Reproducibility):
Can the author(s) provide more details of model training (e.g., loss schedule, model architecture), given that the introduced method applied diffusion in a novel and complex state-space that is non-trivial?

Author response:
Thank you for your feedback. We have added detailed information about our model architecture, hyper-parameter settings, and training details in Appendix B.3 and Appendix B.4. In short, our model is a four-layer transformer with a modified adaptive layer normalization for time variable injection. This architecture is adopted from the source code of LayoutDM. We added two FNNs to encode and decode element vectors. The model size remains comparable.

Reviewer comment (Reproducibility):
Will the code be open-sourced if this paper is accepted?

Author response:
Yes, we plan to include a GitHub link of the source code and trained checkpoints in the final version of the paper. And we have provided source code through an Anonymous GitHub link in the revised version.

Reviewer comment (Experiments):
Can you further discuss the importance of the time-dependent constraint weight (preferably empirically)? This appears to be a very important parameter in incorporating aesthetic constraints.

Author response:
The time-dependent constraint weight is critical for effective model convergence and output quality. Without this weight, the model struggles to converge, leading to a high Fréchet Inception Distance (FID) score, typically remaining above 100, which indicates poor layout quality. We determine the weight empirically as explained in Appendix B.1.