Paper: 3D-GPT: Procedural 3D Modeling with Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
Though it is a nice application to use LLM with procedural 3D modeling tools for text-guided 3D generation, I think the overall contribution of this paper is not enough to be considered for publication on ICLR. The community has had many similar discoveries on the ChatGPT/ GPT-4's ability on generating parameters for images [1] or 3D objects [2] in a zero-shot or in-context learning way. Therefore, there is no surprise that combining LLM with a better parametric 3D modeling tool, e.g., InfiniGen, could produce better visual results. I think all the three potential directions mentioned in the last section are promising and valuable challenges to solve, which would bring more contribution to the community.

Author response:
1. Similar discoveries on ChatGPT/ GPT-4's ability on generating parameters for images [1] or 3D objects [2].  No surprise that combining LLM with a better parametric 3D modeling tool InfiniGen could produce better visual results.

We appreciate the reviewer's comments regarding the perceived limitations of our contribution. However, we respectfully disagree with the assertion that our work lacks novelty. It is essential to differentiate between image generation [1] and 3D modeling, as these areas pose distinct challenges. 

Moreover, the 3D examples provided by [2] do not reflect the complexities we address. Such a way to generate Blender Python code lacks a comprehensive understanding of API functions and their alignment with natural language. The generated Python codes would have high error rates. Yet, it lacks mechanisms for verifying code executability and correctness. Furthermore, directly generating Python codes does not effectively manage dependencies between 3D contents. For instance, when generating trees, the spatial relationship between the leaf and tree branches is not explicitly modeled. As a result, this method of applying LLMs to Blender code could produce inadequate 3D models, especially in the context of complex large-scale scenes.

Also, our research demonstrates that a simple combination of LLMs with the procedural generation library InfiniGen leads to subpar results.
Note that, while the idea of combining LLMs with procedural 3D modeling may now appear straightforward in hindsight, creating an effective workflow is far from simple. Our solution involves three distinct agents: the Task Dispatch Agent interprets and chooses the right API calls, the Conceptualization Agent enhances the scene description with adequate details, and the Modeling Agent comprehends how textual descriptions translate into specific API parameters. This multi-agent approach ensures effective and detailed 3D model generation from textual inputs. Please refer to the results presented in Section 4.2 (Table 1, Figure 6) of our paper, as well as the **updated** supplementary materials (Section 6.2, Table 2, Table 3). These results clearly illustrate the significant roles played by each component within our framework in enhancing the final output.

Reviewer comment (Experiments):
The evaluation is limited. I must admit that for a new task of text-guided 3D scene generation, it is hard to construct baseline methods. But at least, there are similar efforts in the community of text-guided 3D object generation that worth being compared with. For example, score-distillation-based methods, e.g., DreamFusion, can be used as baselines for the "Single Class Control" experiments.

Author response:
2. Comparison with Previous Text-to-3D Methods:

In response to the reviewer's request for a comparison with previous text-to-3D methods, we have included an additional comparison with Dreamfusion [a]  in the **supplementary materials (Section 6.3, Figure 7)**. We show that while Dreamfusion generates reasonable 3D for single objects, it struggles to generate large-scale natural scenes, as demonstrated in our **supplementary materials (Section 6.3, Figure 8)**. In comparison, our method is effective in both scenarios, showing the advantages of our approach in handling more complex 3D modeling tasks. Furthermore,  our framework supports sequential scene editing without necessitating re-training.

Furthermore, it's worth noting that previous text-to-3D methods encountered challenges when generating large-scale natural scenes, as demonstrated in our **supplementary materials (Section 6.3, Figure 8)**.  These observations emphasize the strengths and advantages of our approach in handling more complex 3D modeling tasks.

[a] DreamFusion: Text-to-3D using 2D Diffusion. In ICLR 2023

Reviewer comment (Experiments):
The paper only shows examples of 3D plants and forests. Can 3D-GPT work on other objects or scenes, such as human and street? Can 3D-GPT generalize well to more complex scenarios?

Author response:
1. Expanding the Scope of 3D-GPT Beyond Plants and Forests:

Our method, which leverages the procedural algorithm library Infinigen, was originally designed for natural scene generation. However, its versatility extends beyond this specific application. Its key strength lies in its ability to proficiently handle various 3D modeling tasks in diverse contexts without necessitating fine-tuning. Our system operates in a zero-shot manner, independently processing each modeling function by the conceptualization and modeling agents. This independence ensures that our method can readily adapt to new functions, provided the requisite information. We will provide more complex object modeling in the supplementary before the rebuttal deadline.

Reviewer comment (Experiments):
The paper adopts ChatGPT as LLM. How about other open-source LLMs, such as Alpaca, LLaMA-Adapter, or Vicuna?

Author response:
2. Evaluation with Alternative Open-Source LLMs:

We have also conducted evaluations of our approach using Llama2/GPT4, as outlined in the **supplementary materials (Section 6.2, Table 3)**. Our results demonstrate that the performance of our method remains consistent when applied across different LLMs, underscoring its robustness and broad applicability.

Reviewer comment (Reproducibility):
Lack of details about the model and experiment setting and how the results are affected. For example, which LLM is used? What is the size of the function set F? Does the size of F affect the quality? How many examples are provided? Do the example related to the prompt L? Does zero-shot/few-shot make any difference?

Author response:
**Lack of Detail:**
1. Which Large Language Model (LLM) is used?

We utilized GPT-3.5 for all experiments.

2. What is the size of the function set F?

The function set F comprises all functions and subfunctions detailed in this script (https://github.com/princeton-vl/infinigen/blob/main/worldgen/generate.py), except for the 'creatures' class. These functions are integral to our scene creation process.

3. Does the size of F affect the quality of generated scenes?

Yes, the size of F directly impacts scene quality. Infinigen's scene generation employs a fixed function set. Removing a function, such as 'flower_generation,' eliminates the ability to create corresponding elements (e.g., flowers) in the scene. Similarly, omitting a sub-function like 'control_flower_petal' restricts control over specific aspects (e.g., flower petal movement), impacting scene controllability and diversity.

4. How many examples are provided for each function?

We provide one example per function.

5. Is the provided example related to the user's prompt L?

No, the prompt L is user-supplied. Our examples, one per function, might occasionally resemble some user inputs L, but they are not directly related.

Reviewer comment (Experiments):
Evaluation can be improved. It would be great to do ablation studies on D, C, I, E, and answer the questions mentioned above.

Author response:
**Evaluation Improvement**

1. Ablation Studies on D, C, I, E: Impact of Zero-shot/Few-shot Learning.

Thank you for your insightful comments. In response, we have updated our submission with comprehensive ablation studies in the supplementary material (Section 6.2) . Table 1 elucidates the roles of D, C, I, and E components. Absence of function documentation (D) impairs our method's ability to interpret function input parameters, thereby reducing the CLIP Score and elevating the Failure Rate. Without code (C), while the system grasps basic function and parameter knowledge, it experiences a decline in CLIP Score and an increased Failure Rate. The lack of information (I) hinders the conceptualization agent's capacity to enrich text with vital details, leading to a significant drop in CLIP Score. Excluding examples (E), as shown in Table 4, markedly diminishes the CLIP Score and heightens the Failure Rate. We noted no notable difference in performance between one-shot, two-shot, and three-shot prompts, though introducing more examples tends to reduce parameter diversity.

2. Impact of Scene Complexity and Parameter Count on Results.

The outcomes are indeed influenced by scene complexity and parameter count. The definition of scene complexity matters: if the initial text input features complex objects beyond our procedural algorithm's scope, the quality of generated content is limited. Conversely, detailed initial text descriptions enhance the conceptualization agent's understanding, yielding results more aligned with the input. The number of parameters is critical; removing functions or sub-functions (e.g., 'flower_generation' or 'control_flower_petal') limits modeling abilities, affecting controllability and diversity. Assessing the impact of function set size and parameter count is complex due to varying function significance. For example, removing 'control_flower_petal()' may not markedly affect the CLIP score, but eliminating 'terrain_generation()' substantially reduces it, as terrain modeling becomes impossible. Consequently, we opted not to include evaluations of this aspect, but we will update some visual example in supplementary before rebuttal deadline.

Reviewer comment (Experiments):
It would be great to further explore the limitations and failure cases. For example, does the complexity of the scene affect the results? Does the number of parameters or the design space (e.g., parameter ranges) affect the results?

Author response:
**Evaluation Improvement**

1. Ablation Studies on D, C, I, E: Impact of Zero-shot/Few-shot Learning.

Thank you for your insightful comments. In response, we have updated our submission with comprehensive ablation studies in the supplementary material (Section 6.2) . Table 1 elucidates the roles of D, C, I, and E components. Absence of function documentation (D) impairs our method's ability to interpret function input parameters, thereby reducing the CLIP Score and elevating the Failure Rate. Without code (C), while the system grasps basic function and parameter knowledge, it experiences a decline in CLIP Score and an increased Failure Rate. The lack of information (I) hinders the conceptualization agent's capacity to enrich text with vital details, leading to a significant drop in CLIP Score. Excluding examples (E), as shown in Table 4, markedly diminishes the CLIP Score and heightens the Failure Rate. We noted no notable difference in performance between one-shot, two-shot, and three-shot prompts, though introducing more examples tends to reduce parameter diversity.

2. Impact of Scene Complexity and Parameter Count on Results.

The outcomes are indeed influenced by scene complexity and parameter count. The definition of scene complexity matters: if the initial text input features complex objects beyond our procedural algorithm's scope, the quality of generated content is limited. Conversely, detailed initial text descriptions enhance the conceptualization agent's understanding, yielding results more aligned with the input. The number of parameters is critical; removing functions or sub-functions (e.g., 'flower_generation' or 'control_flower_petal') limits modeling abilities, affecting controllability and diversity. Assessing the impact of function set size and parameter count is complex due to varying function significance. For example, removing 'control_flower_petal()' may not markedly affect the CLIP score, but eliminating 'terrain_generation()' substantially reduces it, as terrain modeling becomes impossible. Consequently, we opted not to include evaluations of this aspect, but we will update some visual example in supplementary before rebuttal deadline.

Reviewer comment (Novelty):
The proposed method demonstrates that the LLM can convert the input prompts to python codes that controls the functions and parameters. However, since the 3D modeling capability comes from the procedural models, not from the LLM, it seems the task is closer to scene composition or object inference instead of modeling. It will be more interesting to see if LLMs can generate 3D modeling commands or procedural modeling sequences/rules.

Author response:
**Others: Suggestions and Queries**

1. LLMs in 3D Modeling and Procedural Sequences:

We value your suggestion about leveraging LLMs for procedural rules in 3D modeling, a topic we've earmarked for future exploration in our paper. Our current research primarily focuses on utilizing LLMs for efficient function control and parameter inference in 3D modeling. This approach is pivotal to our overarching aim of reducing human intervention in the 3D modeling process through advanced LLM applications.

Reviewer comment (Experiments):
There are many procedural models available for Blender. It will be interesting to see if this workflow will work on any arbitrary procedural models given the same amount of information, i.e., D, C, I , and E, such as house, car, airplanes instead of focusing on scenes in InfiniGen.

Author response:
**Others: Suggestions and Queries**

4. Adaptability to Arbitrary Procedural Models:

Our system functions in a zero-shot manner, with each function processed independently by the conceptualization and modeling agents. This independence ensures that our method can adapt to new functions as long as the necessary information is supplied.
As shown in our paper PAGE 5 bottom block, with provided code C, example E, and function document D, our method can understand the function and infer the required parameters.

Reviewer comment (Experiments):
It will be great to see if the selected functions can formulate some dependencies, for example, to generate 'flowers on the trees', the trees need to be created first and the positions of the flowers are based on the positions of the tree branches.

Author response:
**Others: Suggestions and Queries**

2. Function Dependencies and Complex Tasks:

The functions we have selected do establish dependencies. For example, the generate_trees() function can autonomously design tree branches, calculate surfaces programmatically, and accurately place leaves and flowers on these surfaces. This ensures correct dependencies and spares LLMs from the intricate task of determining precise 3D leaf/flower locations, a challenging feat for pretrained LLMs.

Reviewer comment (Reproducibility):
It will be great to see a full example of the input and output of each agent in the process.

Author response:
**Others: Suggestions and Queries**

3. Availability of Full Examples of Input and Output:

We recognize the value of detailed examples for each agent, yet their inclusion would overly extend our manuscript. To keep our narrative concise, we will showcase these examples through supplementary videos or within our user interface. This approach ensures clarity and accessibility without overburdening the text. We will release the code that contains all examples.

Reviewer comment (Experiments):
Have the authors tried to also provide the functions picked by the TDA to CA?

Author response:
**Others: Suggestions and Queries**

5. Integration of Functions Selected by TDA in CA:

To be clear, in our current pipeline, the conceptualization agent processes only those functions that are selected by the task dispatch agent.

Reviewer comment (Experiments):
It is obvious that this method will help users who know nothing about 3D modeling, but I am curious whether the sequential editing task in Fig 4 will be more efficient to professional Blender users (e.g., game developers) to achieve a desired outcome, compared to tweaking the parameters by themselves.

Author response:
**Others: Suggestions and Queries**

6. Efficiency for Professional Blender Users:

For Blender experts that not familiar with procedural generation, in the scenario depicted in Figure 4, transforming flat terrain into mountainous landscapes would require manually rebuilding terrain and repositioning each element, which is time-intensive. 
For Blender experts that are familiar with infinigen libraries might find this task straightforward. However, for each new procedural library, the experts are required to understand every new function, it also poses a significant challenge for professional Blender users to quickly grasp and utilize every new procedural library and its functions. Our approach aims to simplify and expedite this process.