Paper: LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The performance on RICO and PubLayNet is unsatisfactory. As shown in Table 2, the advantage of the proposed method over SOTA methods is marginal on average. This is somewhat below expectation given the fact that the proposed method is based on large pre-trained models and has a chance to observe more training data (under the DA setting).

Author response:
**Question 2:** About the performance of LayoutNUWA on different datasets. 

**Response 2:** This is a very astute observation. We will address your confusion from two aspects:

1. **The RICO and PublayNet datasets are large.** In previous works, even with not very large model parameters, the models perform well when trained on a large training dataset and then validated on a test set with a similar distribution. However, for datasets with less data, such as Magazine, the performance is poor. Therefore, our model improves more on the Magazine dataset. However, **we can't specifically look at the improvement on a certain dataset.** Specifically, we conducted Domain-Agnostic experimental settings, and LayoutNUWA has significant improvements on all datasets under this setting, which previous work could not achieve.

2. One of the success points of LayoutNUWA's good performance is that it uses the prior Code knowledge of LLM (Section 5.2, page 9). It's not surprising that it performs best on low-resource datasets (Magazine), which also proves our point that **LayoutNUWA makes full use of the prior code and layout knowledge of LLM.**

Reviewer comment (Writing):
In Eq.(3), what does M_j^t denote? What is the definition of L on the right hand side of the equation?

Author response:
**Question 1:** About Equation Issue

**Response 1:** In Equation 3, $M_{j}^{(t)} $represents the mask corresponding to the $j$-th element in a Layout for the $t$-th task.

Reviewer comment (Evaluation):
From the results in Table 1 and 2, the performance improvement on RICO and PubLayNet is not as significant as on the Magazine dataset. What are possible reasons behind this?

Author response:
**Question 2:** About the performance of LayoutNUWA on different datasets. 

**Response 2:** This is a very astute observation. We will address your confusion from two aspects:

1. **The RICO and PublayNet datasets are large.** In previous works, even with not very large model parameters, the models perform well when trained on a large training dataset and then validated on a test set with a similar distribution. However, for datasets with less data, such as Magazine, the performance is poor. Therefore, our model improves more on the Magazine dataset. However, **we can't specifically look at the improvement on a certain dataset.** Specifically, we conducted Domain-Agnostic experimental settings, and LayoutNUWA has significant improvements on all datasets under this setting, which previous work could not achieve.

2. One of the success points of LayoutNUWA's good performance is that it uses the prior Code knowledge of LLM (Section 5.2, page 9). It's not surprising that it performs best on low-resource datasets (Magazine), which also proves our point that **LayoutNUWA makes full use of the prior code and layout knowledge of LLM.**

Reviewer comment (Presentation):
The authors mention that LayoutNUWA suffers from the error propagation problem. Some visualizations/examples might be useful to illustrate this point.

Author response:
Thanks for your valuable comments.

**Question 1:** About the issue of understanding cases of error propagation.

**Response 1:** We have updated the example diagram of error propagation (Appendix A, Figure 4, page 14) in the updated version of the PDF.  You can also click on this link: [https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/error_propagation.pdf](https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/error_propagation.pdf) to view it on the anonymous website. *( If a PDF rendering error occurs, manually refreshing the page can resolve it )*

Specifically, possible error propagation is that the history layout generated by the AR model contains an unreasonable Layout, which will affect the layout generation of subsequent elements, such as overlapping, unreasonable size/location, etc. However, these all happen under the condition of insufficient training. **All models in this paper have been fully trained, so the impact of this error propagation is minimal.**

Reviewer comment (Evaluation):
*Potential limited diversity and overfitting*: The qualitative generation results appear very similar to the real design, and it is unclear whether the model can generate diverse layouts given a single condition, or if the model has overfitted to a certain specific layout. I suggest the author(s) to include the most similar training example to the generation (by DocSim) in a figure in the appendix. Please also see questions below for addressing my concerns.

Author response:
This is a great question. We construct the same input and take the top 4 paths with the highest probabilities from the model output. Our model can generate diverse layouts in the position and size of each element while keeping the overall position of elements unchanged. You can see the generated cases on our provided anonymous website in  [https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md](https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md), and this phenomenon is quite obvious. In the paper, all real designs are sampled from the test and development datasets (which is completely consistent with previous work such as LayoutDM). To further address your concerns, in the updated version of the paper, we used the DocSim metric in Appendix D (page 16) to show the distribution of training data similar to the test set, which also proves that our model is not overfitting on some specific layout distributions.

Reviewer comment (Experiments):
Regarding the potential diversity and overfitting, I wonder if the author(s) can show whether the model can generate diverse layouts during the sampling process, given the same input conditions. Moreover, I wonder if the 'real design' in the Figures (e.g., Fig 5) are taken from the training or validation set? It could indicate overfitting given the high similarity between real designs and the generated designs in LayoutNUWA, especially given the LLMs are significantly larger in size over prior work.

Author response:
This is a great question. We construct the same input and take the top 4 paths with the highest probabilities from the model output. Our model can generate diverse layouts in the position and size of each element while keeping the overall position of elements unchanged. You can see the generated cases on our provided anonymous website in  [https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md](https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md), and this phenomenon is quite obvious. In the paper, all real designs are sampled from the test and development datasets (which is completely consistent with previous work such as LayoutDM). To further address your concerns, in the updated version of the paper, we used the DocSim metric in Appendix D (page 16) to show the distribution of training data similar to the test set, which also proves that our model is not overfitting on some specific layout distributions.

Reviewer comment (Experiments):
Have the author(s) also explored unconditional generation? It would be good to understand the capability of the models to generate diverse layouts unconditionally, following the point above.

Author response:
That's a great suggestion. In the updated paper, we also explored the unconditional generation in Appendix E (page 16~17). Here is the model performance:  
| Models | RICO Align. | RICO FID | PubLayNet Align. | PubLayNet FID | Magazine Align. | Magazine FID |  
|--------|-------------|----------|------------------|---------------|----------------|--------------|  
| LayoutTrans | 0.008 | 12.286 | 0.015 | 33.416 | 0.043 | 32.083 |  
| MaskGIT | **0.003** | 60.724 | **0.004** | 28.836 | 0.003 | 157.560 |  
| DiffusionLM | 0.019 | 23.997 | 0.016 | 18.720 | 0.043 | 37.951 |  
| LayoutDM | 0.010 | *6.456* | 0.011 | 13.437 | 0.037 | 59.507 |  
| LayoutNUWA-CL-DS (ours) | *0.008* | **5.674** | *0.007* | **8.914** | **0.092** | **24.108** |  
| LayoutNUWA-CL-DA (ours) | 0.014 | 6.932 | 0.013 | *9.208* | *0.073* | *28.930* |  
| Real Data | 0.004 | 6.250 | 0.001 | 1.850 | 1.693 | 6.695 |

Reviewer comment (Evaluation):
While the domain-agnostic model is impressive, I wonder if the author(s) have observed domain confusion? Such as the model generating a magazine layout when prompted to generate a mobile layout (from RICO).

Author response:
This is a great question, and we specifically discussed this issue in Appendix F (page17~18). As shown in the table below:  

| Tasks | RICO -> Magazine Align. | RICO -> Magazine FID | Magazine -> RICO Align. | Magazine -> RICO FID |  
|--------|------------------------|----------------------|------------------------|----------------------|  
| C + S -> P | - | - | 0.001 | 8.027 |  
| w/o Domain-Confusion | **0.472** | **6.755** | **0.007** | **2.870** |  
| C -> S + P | 0.196 | 10.623 | 0.014 | 4.569 |  
| w/o Domain-Confusion | **0.359** | **8.791** | **0.004** | **2.524** |  
| Completion | 0.217 | 9.283 | 0.013 | 9.756 |  
| w/o Domain-Confusion | **0.416** | **7.572** | **0.007** | **7.542** |  
| Real Data | 1.693 | 6.695 | 0.004 | 6.250 |  

, where we design a toy experiment and find: 1)when given more prior layout information, such as in the C+ S → P setting, the model is less affected by domain confusion as the model tends to “resist” such domain confusion, i.e., The performance of LayoutNUWA is much worse when domain confusion setting is applied compared to when it is not used; 2) However, when there is a lack of layout prior conditions, such as in the C → S + P or Completion setting, the model is greatly affected by the domain confusion as such issue simply causes minor disturbances to the results, i.e., the model “yields to” the domain confusion under such circumstance. Overall, under the DA setting, the model indeed experiences Domain Confusion problems (mainly depending on the amount of given prior knowledge), which is enlightening when using LLM for DA Layout modeling. For more details, please refer to Appendix F of this updated paper.