Paper: InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR-2024)

Reviewer comment (Evaluation):
InstructScene is proposed to handle natural language instructions. However, the instruction does not seem to support arbitrary instructions. In qualitative results, the instruction always explicitly mentions one of these relationships.

Author response:
While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability to a broader range of instructions. We use instructions automatically generated in a certain pattern only for evaluation convenience.

The generalizability owes to the pretrained CLIP text encoder, which is adapted to extract semantically meaningful features of text inputs, instead of using a language parser like the early studies of 3D scene synthesis methods [1, 2] that can only handle sentences in limited expressions. It means the text features that we used from conditioning would be close in the feature space if the input texts have similar meanings.

We utilize instructions in various sentence patterns in the stylization task, such as "Let the room be wooden style" and "Make objects in the room black," as illustrated in Appendix Figure 8. We also try some meaningless “arbitrary instructions”, such as “a cat on the ground” and “a dog wearing a santa hat”, and our method can still generate meaningful and aesthetic 3D scenes like those generated unconditionally shown in Appendix Figure 11. When the inputs are some sentences with vague location words, like “Put a chair next to a double bed.”, our method will generate corresponding objects in all possible spatial relations (e.g., “left”, “right”, “front” and “behind”).

Nevertheless, our model still faces limitations in comprehending complex text instructions. For instance, (1) handling instructions with more required triplets, like 4 or 5, poses a challenge. (2) Additionally, identifying the same object within one instruction, such as "Put a table left to a sofa. Then add a chair to the table mentioned before" is also a difficult task. (3) Furthermore, InstructScene struggles with abstract concepts such as artistic style, occupants, and functionalities that do not occur in the curated instructions. Given the rapid development of large language models (LLMs), we believe the integration of LLMs into the proposed pipeline is a promising research topic.

A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset or using crowdsourcing to make the dataset curation pipeline semi-supervised. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.

We will add this discussion to the revision of this work.

Reviewer comment (Evaluation):
It is not clear how iRecall is computed. To the best of my understanding, iRecall computes the recall, where the prediction is generated scene graph triplets and the ground truth is the instruction triplets. However, it is unclear how instruction triplets are obtained. For example, if the instruction is "Add a corner side table with a round top to the left of a black and silver pendant lamp with lights", it looks non-trivial to extract triplets. It is uncertain whether relationship keywords are used to extract them.

Author response:
It is right that “iRecall is the proportion of the required triplets in the generated scenes and the ground truth in instruction triplets”.

However, the triplets are not extracted from instructions. Instead, instructions in our dataset are derived from 1~2 randomly selected triplets. Thus, we have the ground-truth triplets of corresponding instructions for iRecall evaluation. More details about the dataset are provided in Appendix A.

Reviewer comment (Theory):
Some claims or arguments about using VQ-VAE are not convincing and require further substantiation. The authors claim that VQ-VAE is used because objects share common characteristics like colors or materials. But I don't see why OpenCLIP features cannot encode these characteristics. And it's not well grounded to say that pre-trained OpenCLIP features are too complicated to model. As the author also mentioned the efficiency of discrete tokens, it's better to write the motivation clearly for using VQ-VAE.

Author response:
**2. Some claims about using VQ-VAE require further substantiation.**

OpenCLIP features indeed encode characteristics like colors and materials. However, these attributes are entangled within a single vector of 1280 dimensions. This high dimensionality makes it challenging to model these features together with other low-dimensional scene attributes, such as transitions and scales, which only have 3 dimensions each.

The use of VQ-VAE is not for efficiency. Instead, it serves to simplify the learning task. Indexing discrete variables from a codebook can significantly reduce the burden of network optimization. Our preliminary experiments also suggest that both ATISS and DiffuScene face difficulties in modeling high-dimensional semantic feature distributions. To ensure fair comparisons in our experiments, we enhance these models to generate quantized features.

Reviewer comment (Writing):
The annotation is overcomplicated and hard to follow. There are new definitions of previous variables throughout the method section, like $c, f$ in the first paragraph of Sec. 4.2, $\mathcal{C}_i$ and $\mathcal{F}_i$ in the first two lines of Sec. 4.2.2. $f$VQ-VAE is not a standard notation. There are also variables with omitted super/sub-scripts that are not explained clearly.

Author response:
**3.2. The annotation is overcomplicated and hard to follow.**

We apologize for any confusion caused by the annotation. We try our best to keep the annotations rigorous and consistent in the manuscript. We use lowercase characters like $c$ and $f$ for attributes of one object, and use calligraphic fronts like $\mathcal{C}$ and $\mathcal{F}$ for attributes of one scene. $\mathcal{C}_i$ and $\mathcal{F}_i$ refers to the specific $i$-th scene. We use superscripts to denote different scenes and subscripts to denote different objects in the same scene. Variables with omitted super-/sub-scripts mean a non-specific sample.

We will try our best to keep the annotation concise in the revision of the manuscript. Thank you for bringing that to our attention.

Reviewer comment (Reproducibility):
What is the CLIP-aligned feature in Fig. 2? How is this feature vector applied to the VQ-VAE encoder?

Author response:
**4.1. What is the CLIP-aligned feature in Fig. 2?**

The “CLIP-aligned feature” in Figure 2 refers to the output vector of OpenShape [3], which is a pretrained RGB point cloud encoder that is contrastively trained to be aligned with the output space of OpenCLIP ViT-bigG/14.

**4.2. How is the CLIP-aligned feature vector applied to the VQ-VAE encoder?**

As described in the first paragraph of Sec. 4.4, the encoder of $f$VQ-VAE is a stack of cross-attention modules. $n_f$ learnable tokens are employed as queries, and the CLIP-aligned feature vector is served as the key and value.

Reviewer comment (Reproducibility):
Though the relations are view-dependent, the generated instructions do not describe the viewpoint of the camera. Is there an assumed viewpoint for all scenes?

Author response:
**5. Is there an assumed viewpoint from all scenes?**

The term "view-dependent" means that spatial relations are respected by the users who provide instructions. These spatial relations will change in accordance with the users' viewpoints. There is no need to specify the camera's perspective.

Reviewer comment (Evaluation):
Minor: in Fig. 5 row 4, the proposed methods generate a single bed while the instruction mentions a double bed.

Author response:
**6. The proposed method generates a single bed while the instruction mentions a double bed.**

Thank you for the careful assessment. Although we have significantly promoted the controllability of instruction-driven 3D scene synthesis, the iRecall scores of our method are still not 100%. Therefore, it is possible to observe incorrect triplets in the generated scenes. We hope follow-up works could further improve the performance.

Reviewer comment (Novelty):
Missing reference: LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Neurips 2023

Author response:
**7. Missing reference of LayoutGPT.**

Thank you for the kind reminder. We will add this reference in the revision of this work.

Reviewer comment (Experiments):
More details about the dataset creation can be provided. It would be better if the author can provide some examples of the training data.

Author response:
The training data of instructions are very similar to those provided in the first column of Appendix Figure 5~7, which are for scenes in the test split, instead of the training split. The details about the data creation are provided in Appendix A. We also provide some examples below.

(0) Object categories are provided by the original dataset: e.g., `{id1_category: "dressing_table"}`. (1) Spatial Relation Extraction: e.g., `(id1, “closely left”, id2)`. (2) Object Caption: e.g., `{id1_caption: "a 3D model of a wooden desk with a mirror on top in a white background"}`. (3) Caption Refinement: e.g., `{id1_refined_caption: "a modern wooden dressing table with a mirror on top"}`. (4) Instruction Generation: e.g., `{“Place”, “Add”, “Put”, …}` +  ` id1_refined_caption` + `{“closely to the left of”, “to the close left of”, “to the near left of”, …}` + `id2_refined_caption.` + `{Then, Next, And, ...}` + ...

Our dataset and scripts for creation will be publicly available after the review period.

Reviewer comment (Reproducibility):
In the semantic graph, relation e_{jk} can be determined by e_{kj}. Are the both directions of the instructions needed to be created? If so, how to determine which direction to use during training?

Author response:
We randomly pick one direction to generate instructions during training, e.g., “a chair is left to a table” and “a table is right of a chair”, where both instructions should correspond to the same relation in a scene. To incorporate this inductive bias, we did some tricks in the code. In particular, the final logits for loss computation are derived by averaging the upper triangular part and the symmetrized lower triangular part of the relation matrix.

Our code will be publicly available after the review period.

Reviewer comment (Evaluation):
What is the runtime of the proposed method compared to baselines?

Author response:
In the default settings (T=100+10), our method takes about 12 seconds to generate a batch of 128 living rooms by our method on a single A40 GPU. In comparison, ATISS (Paschalidou et al., 2021) takes 3 seconds, and DiffuScene (Tang et al., 2023) requires 22 seconds.

It's noteworthy that our method can be significantly accelerated by reducing the number of diffusion time steps. For instance, setting T=20+5 reduces the runtime to 3 seconds without a noticeable decline in performance. The impact of diffusion time steps is investigated in Sec. 5.5.1. We believe with more advanced diffusion techniques and in more complex scenes, diffusion models can be more effective and efficient than autoregressive models.

Reviewer comment (Experiments):
The instructions are generated mainly from rules. Do you think these rules can cover the majority of the instructions compared to a human-annotated dataset? What are the limitations of the instructions in the current dataset?

Author response:
While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability to a broader range of instructions. For example, in the stylization task, we utilize instructions in various sentence patterns such as "Let the room be wooden style" and "Make objects in the room black", as illustrated in Appendix Figure 8. We also experiment with instructions containing vague location words, like "Put a chair next to a double bed," wherein our method generates corresponding objects in all possible spatial relations (e.g., "left," "right," "front," and "behind").

Nevertheless, our model still faces limitations in comprehending complex text instructions. For instance, (1) handling instructions with more required triplets, like 4 or 5, poses a challenge. (2) Additionally, identifying the same object within one instruction, such as "Put a table left to a sofa. Then add a chair to the table mentioned before" is also a difficult task. (3) Furthermore, InstructScene struggles with abstract concepts such as artistic style, occupants, and functionalities that do not occur in the curated instructions.

These limitations are attributed to the CLIP text encoder, which is contrastively trained with image features and tends to capture global semantic information in sentences. Given the rapid development of large language models (LLMs), we believe the integration of LLMs into the proposed pipeline is a promising research topic.

A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset or using crowdsourcing to make the dataset curation pipeline semi-supervised. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.

We will add this discussion to the revision of this work.

Reviewer comment (Novelty):
More of a system paper: combines many ideas from previous works on 3D scene generation and works on diffusion-based generation model. Seems that there is not too much truly new ideas here (and clarifications on where these happens would be great). I'm fine with it, to be honest, because this paper ticks my checkboxes for what a good system paper should be: clever/novel combination of ideas/components, solid execution, exceptional results. However, this is technically still a "weakness" I guess, as it limits the further applicability of the proposed method in other domains.

Author response:
**1. Applicability of the proposed method in other domains.**

The basic idea behind the proposed method is to introduce structural priors for complex language-driven content generation. The proposed semantic graph prior is a general concept and can be directly applied to other tasks, such as 2D poster design. We hope the exploration of 3D scenes could inspire more research on other domains.

Reviewer comment (Experiments):
Questionable choice of baselines. I get that previous graph-based/text-based works are not trained on 3D-FRONT, but it should be possible to compare with them in some capacities, or at least discussing the issues with them in the evaluation section.

Author response:
**2. Discuss other previous graph-/text-based works that are not trained on 3D-FRONT.**

Thank you for the kind reminder. There are plenty of great works on both graph-based and text-based 3D scene generation. We try our best to include them in the related work section (Sec. 2) and discuss the drawbacks of existing works.

Two methods, namely ATISS (an autoregressive model) by Paschalidou et al. (2021) and DiffuScene (a diffusion model) by Tang et al. (2023), are selected as baselines in our work. That is because both of them are recently proposed (many papers in 2023 are still using the same architecture as ATISS [1, 2]), representative (autoregressive and diffusion models currently stand as two of the most popular generative models), and showcase state-of-the-art performance in various metrics. Other remarkable works, such as FastSynth [3], SceneFormer [4], DepthGAN [5] and Sync2Gen [6], despite their significance, demonstrated notably inferior performance compared to ATISS and DiffuScene, and most of them are unconditional generative methods, so they were excluded from our evaluation.

We acknowledge that we may overlook some important works and would appreciate any suggestions. We will add them to the revision of this work.

Reviewer comment (Experiments):
Despite the good discussions of issues of the way prior works handle latent object features (my impression was that they don't help much), I am left unconvinced that the proposed method actually make use of such features e.g. 1. would manually changing semantic feature f of an object result in notable difference in the final scene? 2. Can the model pick up stylistic information from the training data? Do objects that frequently occur together in the training data behave similarly in the generated scenes? 3. Is there any noticeable degrade in scene layout quality if semantic features are not encoded (i.e. just sample CAD models by category and size, which is a common practice for prior works) or if they are encoded with a smaller feature space?

Author response:
**3.1. Would manually changing semantic feature of an object result in notable differences in the final scene? Can the model pick up stylistic information from the training data? Do objects that frequently occur together in the training data behave similarly in the generated scenes?**

We conduct two additional experiments to further validate our method: (1) masking the semantic feature of one object and utilizing a pretrained semantic graph prior for recovery; (2) masking semantic features of all objects except one and again using the pretrained semantic graph prior for recovery. Instructions for both experiments are set to none. Visualization results are presented in the supplementary material submitted during the rebuttal period.

These results indicate the diversity of our method and highlight that semantic graph prior could effectively capture stylistic information and object co-occurrences from the training data. Our method trends to generate style consistent and thematic harmonious scenes, e.g., chairs and nightstands in a suit, and matched color palettes and cohesive artistic style. Other visualizations in the appendix also support this claim, especially with Figure 8 showcasing the generation of all semantic features based on both instructions and other scene attributes.

**3.2. Is there any noticeable degradation in scene layout quality if semantic features are not encoded or if they are encoded with a smaller feature space?**

We observed a significant decline in the appearance controllability and style consistency of generated scenes when semantic features were omitted. We include these degraded visualization results in the supplementary material.

It arises from the fact that, without semantic features, the generative models solely focus on modeling the distributions of layout attributes, i.e., categories, translations, rotations, and scales. This exclusion of semantic features results in generated objects whose occurrences and combinations lack awareness of object style and appearance, which are crucial elements in scene design.

We will add this discussion to the revision of this work.

Reviewer comment (Evaluation):
More discussions on how the qualitatively the proposed method works better than prior works would be very helpful. I can clearly that this is the case from the supplemental figures, and it would be great to highlight this in the main paper as well with a paragraph or so, as the proposed method clearly avoids many issues with prior works, especially on larger rooms. The set of quantitative metrics, while comprehensive, are known to be not the most sensitive metrics when it comes to evaluating indoor scenes, especially when it comes to layout/appearance details. More in depth analysis of the qualitative examples would make it a much stronger case that the proposed method is clearly superior.

Author response:
**5. Discussions on how qualitatively the proposed method works better than prior works.**

Thank you for this suggestion. In Appendix C.2, we discuss how our proposed method qualitatively outperforms prior works. We find that the autoregressive model ATISS tends to generate redundant objects, resulting in chaotic synthesized scenes. DiffuScene encounters challenges in accurately modeling object semantic features, often yielding objects that lack coherence in terms of style or pairing, thereby diminishing the aesthetic appeal of the synthesized scenes. Moreover, both of these baseline models frequently struggle to follow the provided instructions during conditional generation. In contrast, our approach demonstrates a notable capability to generate highly realistic 3D scenes that concurrently adhere to the provided instructions.

We will add this discussion to the main paper in the revision of this work.