Paper: CityGPT: Generative Transformer for City Layout of Arbitrary Building Shape (ICLR-2024)

Reviewer comment (Novelty):
Recent studies, such as InfiniCity and CityDreamer, have focused on creating city layouts, incorporating both roads and buildings. However, this particular work only generates buildings without roads, which may limit its practical applicability in real-world scenarios.

Author response:
**Answer:** We acknowledge your concern about the applicability of our model compared to existing models like InfiniCity or CityDreamer. However, it's important to note that we address a different problem compared to InfiniCity and CityDreamer. Both InfiniCity and CityDreamer first generate a 2D semantic map or depth image of the layout, then extend it to 3D voxel rendering, focusing on rasterized data. Our approach, on the other hand, focuses on vectorized data generation. In contrast to the image-represented layout generated by InfiniCity and CityDreamer, our layout comprises a set of polygons represented by arrays. In our task field, 3D buildings are also represented as arrays with additional height value, or in a more sophisticated manner, using wireframe representation. To the best of our knowledge, we believe that the work most closely related to our problem is Globalmapper [1], which also focuses on generating vectorized city layouts. It's important to note that vectorized representations are more concise than rasterized data. By tackling this challenge, we believe our approach offers a valuable exploration in the field.

[1] He, Liu, and Daniel Aliaga. "GlobalMapper: Arbitrary-Shaped Urban Layout Generation." ICCV 2023.

Reviewer comment (Writing):
The paper is not clearly written, missing too many details in Sections 3.2 and 3.3. After reading the two sections, it is still unclear how to convert the "Predicted Position Map" to "Reconstructed Building Layout".

Author response:
**Answer:** We apologize for the unclear explanation in our paper writing. We have updated our Appendix A to better demonstrate our model pipeline. Additionally, we have added a detailed explanation of our inference pipeline in Appendix B.

Reviewer comment (Writing):
What is "in f city layout generation" mentioned in the third contribution?

Author response:
**Answer:** We apologize for the typo in our writing. The correct version is "in city layout generation." We have rectified this error with blue text.

Reviewer comment (Reproducibility):
It is unclear how to generate the height of the buildings. According to the definitions in Section 3.1, the buildings only contains the coordinates of footprints.

Author response:
**Answer:** We apologize for the unclear explanation in our paper writing. We have updated our Appendix A and added some new figures to better demonstrate the further experiments, including the setting that generates a 2.5D city.

Reviewer comment (Evaluation):
The runtime performance analysis should be conducted, including both the 2D and 2.5D generation.

Author response:
Thanks for your constructive comments. We have conducted a runtime performance analysis on our inference stage, including both 2D and 2.5D generation, as detailed in Appendix B. Our method could achieve same level of inference efficiency as baselines.

Reviewer comment (Novelty):
Several layout generation works [1-4] should be cited and discussed in the paper.

Author response:
Thanks for your constructive comments. We have cited and discussed them in the revised Section 2. We have also updated Table 1 to include the discussion of a work from your recommended list.

Reviewer comment (Reproducibility):
Some details in the further experiments are missing. For the classification task, the architecture of the classification model used here is unclear.

Author response:
Thanks for pointing out the weak points in our paper writing. We have added detailed explanations of our CityGPT model structure and the model structure in further experiments in Appendix A. Additionally, we have updated the expression in Section 5 about the road condition generation. We chose the latter approach to generate the layout condition on the road network to enable our model to have the capacity to learn mutual attention between roads and buildings.

Reviewer comment (Experiments):
For the generation based on the road network, why choose the latter approach, rather than concatenating the condition embedding with the mask tokens?

Author response:
Thanks for pointing out the weak points in our paper writing. We have added detailed explanations of our CityGPT model structure and the model structure in further experiments in Appendix A. Additionally, we have updated the expression in Section 5 about the road condition generation. We chose the latter approach to generate the layout condition on the road network to enable our model to have the capacity to learn mutual attention between roads and buildings.

Reviewer comment (Evaluation):
For the user study, why are the results of the proposed model more realistic than the ground-truth layouts?

Author response:
As mentioned in our user study description, we added some small noise to the ground-truth data, which could be a reason for these results. Initially, we added this noise because most of the ground-truth data were perfectly aligned, and we assumed it might provide a simple way to distinguish them. However, we now consider that it may cause confusion for readers. Therefore, we have provided new user study results in Section 4.2 in cases where no noise is added to the ground truth. Additionally, we offer visual comparisons between ground truth with noise and ground truth in Appendix H. Even without nosies, our method can still outperform human baseline.

Reviewer comment (Reproducibility):
Will the dataset be released to facilitate future research in the community?

Author response:
Some of our datasets are open source, and those that we construct ourselves will definitely be released. We will include all of them on our future project page.

Reviewer comment (Novelty):
Not too much novelty: the idea of predicting a probability distribution over building, the idea of autoregressive generation of polygons and the idea of transformer based layout generation can all be traced to prior works. While the domain (city generation) and the combination of techniques are novel (conditioning polygon generation directly on the location distribution), such novelty are likely not directly useful for people who are not interested in this particular problem. Subsequently, results quality would matter much more and

Author response:
We acknowledge your concern that some of our techniques may be traced back to previous works. However, we would like to clarify the novelty of our approach. 

Firstly, in the task setting, we aim to generate city layouts with **arbitrary scales and buildings of arbitrary shapes**, a dimension that has not been thoroughly explored in city layout generation tasks or other more general layout generation tasks. We believe that our exploration in this area, specifically in arbitrary-shaped polygon layout generation, can inspire further research. 

Secondly, our approach possesses its own novelty. While there are existing works that utilize transformers for layout generation, to the best of our knowledge, all these transformer-based models **require predefining the order of the layout**. For example, some works define the order of each box (in our case, polygons) based on the x-coordinate of the center of the box, and in cases where x-coordinates are the same, the y-coordinate is used. In contrast, **our approach eliminates the need for this predefined order** through the introduction of the positional prediction stage. This disordered generation with positional guidance is important in various applications, such as infinite-scale layout generation, city complementation, and more. We believe that our exploration in more general layout generation tasks is valuable.

Reviewer comment (Evaluation):
The evaluation is underwhelming. Most baselines are for general layout problems that has much weaker constraints than the specific problem. The one baseline that addresses this problem (AETree) is a weak one the doesn't even compete with more general methods. Even with this set of baselines, I am not sure whether the proposed method really generates better layout, based on the qualitative results. The quantitative metrics are too generic (FID is too general for evaluating such layout visualizations, WD over edge/area/ratio doesn't really evaluate layout quality), the user/perceptual study is also not well conducted (ground truth shouldn't have minor noises, visualizations of layout should be better so humans can actually judge if the layouts are good i.e. not just with blue boxes/meshlab screenshots).

Author response:
Firstly, we acknowledge your concern about the baseline choice. Originally, we intended to compare our model with Globalmapper[2]. Unfortunately, they released their code close to the submission deadline. Given the non-trivial nature of adapting their method to our task quickly, we couldn't deploy their model as a baseline during the rebuttal time. However, we are open to including a comparison between their model and ours in future work.

Secondly, regarding the metric question, we believe FID and WD are valid metrics, as many related works in the same research field still use these metrics. To better evaluate our model's generation results, we introduced a new metric called CitySim, inspired by READ[1]. Reviewer can find the results in Table 2, and the specific design of these metrics is detailed in Appendix C. Our method still outperforms baselines on this metric.

Thirdly, for addressing reviewer’s concern about the user study, we provide new user study results in Section 4.2, specifically in cases where noise is not added to the ground truth. Additionally, we offer visual comparisons between ground truth with noise and ground truth in Appendix H. The results show that our generation results are most similar to the ground-truth among the baseline.

[1] Patil, Akshay Gadi, et al. "Read: Recursive autoencoders for document layout generation." CVPR 2020. 

[2] He, Liu, and Daniel Aliaga. "GlobalMapper: Arbitrary-Shaped Urban Layout Generation." ICCV 2023.

Reviewer comment (Evaluation):
Lack of evaluation over whether the method can generate novel and diverse layouts that are different from the training set. I am not convinced that the model is not overfitting to some training samples.

Author response:
To address the reviewer's concern about whether our model is overfitting to the training samples, we've included a discussion and relative scores in Appendix D. This aims to clarify that our model is not overfitting to any training samples. Specifically, we add a novelness and uniqueness metrics that verifies our method is able to generate novel and unique layouts instead of overfitting to the training data.

Reviewer comment (Evaluation):
As mentioned in the weakness section, I have concerns over the evaluation protocol, providing more evidence that the model can generate realistic layout (i.e. with more proper metrics and user studies) and is not just overfitting will change my opinion on this paper siginificantly.

Author response:
Firstly, we acknowledge your concern about the baseline choice. Originally, we intended to compare our model with Globalmapper[2]. Unfortunately, they released their code close to the submission deadline. Given the non-trivial nature of adapting their method to our task quickly, we couldn't deploy their model as a baseline during the rebuttal time. However, we are open to including a comparison between their model and ours in future work.

Secondly, regarding the metric question, we believe FID and WD are valid metrics, as many related works in the same research field still use these metrics. To better evaluate our model's generation results, we introduced a new metric called CitySim, inspired by READ[1]. Reviewer can find the results in Table 2, and the specific design of these metrics is detailed in Appendix C. Our method still outperforms baselines on this metric.

Thirdly, for addressing reviewer’s concern about the user study, we provide new user study results in Section 4.2, specifically in cases where noise is not added to the ground truth. Additionally, we offer visual comparisons between ground truth with noise and ground truth in Appendix H. The results show that our generation results are most similar to the ground-truth among the baseline.

[1] Patil, Akshay Gadi, et al. "Read: Recursive autoencoders for document layout generation." CVPR 2020. 

[2] He, Liu, and Daniel Aliaga. "GlobalMapper: Arbitrary-Shaped Urban Layout Generation." ICCV 2023.

Reviewer comment (Evaluation):
The distribution predicted by phase 1 also seems extremely blurry and doesn't really resemble a probability distribution. It almost seems to me that the method is just attempting to memorize the silhouettes of buildings. A bit more analysis would be great here.

Author response:
We apologize for any confusion arising from our explanation of visualization. The visualized positional map is not a normalized distribution across the entire canvas. Instead, it is a 50*50 grid map, with each value falling within the range of [0, 1]. This value represents the probability of the existence of a building at a given position on the map. During our sampling process, we normalize this 50*50 map using softmax and subsequently sample positions from it.

Reviewer comment (Experiments):
In section 4, it mentioned that weights need to be applied to BCE to address class imbalance, this shouldn't happen if the model can actually learn the distribution. It seems that after applying such weights, there isn't really too much difference between high/low likelihood, which is not ideal.

Author response:
We apologize for any confusion in our Stage 1 description. In each grid on the map, we are addressing a binary classification problem. We have applied Binary Cross-Entropy (BCE) loss with a positive weight, considering the much smaller number of buildings in each layout block compared to 50x50. We believe this is a commonly used method in binary classification to mitigate false negative scores. Our results demonstrate that this learned distribution can yield reasonable generation outcomes.