Paper: Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts (ICLR-2024)

Reviewer comment (Evaluation):
The local region for each edit must be manually entered by the user. This slightly limits the intuitive, easy-to-use nature of this work as compared to most other purely text-driven approaches.

Author response:
We underline that our Progressive3D framework supports **different region definitions** including 3D forms (3D boxes, custom meshes) and 2D forms (2D segmentation and attention maps) since the corresponding 2D region mask of each view can be obtained.
(See in **Sec.3.2** and **Appx.A.3**). We adopt 3D boxes as region definitions in **Sec.3** for brevity.
Differing from existing 3D editing methods, our $\mathcal{L}_{consist}$ is imposed on 2D rendered images and is effective for various 3D representations and region definitions (See discussion in **Appx.A.5**). 
We have provided editing samples in **Fig.11** in the revised paper that demonstrates Progressive3D is available with region defined by multiple 3D boxes, custom meshes, and 2D segmentation queried by the provided keyword (**text-driven region selection**).

Noticing that user-provided masks and automatically calculated regions of interest are both widely used in 2D repainting/editing methods, we believe that defining a specific region instead of leveraging the inaccurate attention maps is necessary in many scenarios, and our framework also supports attention maps as the region definition.

Reviewer comment (Theory):
If I am understanding correctly (see question for more details), the edit region can only be defined as an axis-aligned bounding box. This seems like it could be problematic for edits that do not fit nicely into an axis-aligned box.

Author response:
As described in **Sec.3.3**, the 3D bounding box can be rotated arbitrarily with the corresponding rotate matrix $R$, and the 3D bounding box and its rotation matrix can be obtained conveniently through a 3D GUI.

Reviewer comment (Experiments):
Limited comparisons to relevant existing work. There exist methods for focusing on different parts of the text prompt (Attend and Excite [1]) for 2D image generation and editing. These methods have been shown to address issues with attribute binding and “catastrophic neglect.” It would be helpful to see how a baseline performs using these approaches as the 2D model used for distillation. Additionally, DreamEditor [2] enables local editing on NeRFs using an explicit edit region. The region is inferred from the text description using attention maps from the diffusion model. A good baseline would be to use DreamEditor iteratively on each progressive edit as this still gives an explicit edit region, but does not require a user input bounding box.

Author response:
We have provided the qualitative and quantitative comparison with DreamTime-based baselines including DreamTime+CEBM and DreamTime+A&E on CSP-100 in **Fig.6** and **Tab.1** in the revised paper.
Since composing T2I methods including CEBM [1] and A&E [2] are not designed for composing 3D content generation, combining composing T2I methods with DreamTime brings limited performance improvements (even regression), while DreamTime+Progressive3D is superior to all DreamTime-based baselines.
We analyze that although compositional T2I models produce more desirable results with complex prompts than normal T2I methods in each view, the inconsistency among views is still significant, leading to low-quality results.
However, Progressive3D reduces the generation difficulty since the current 3D content has been achieved ideally with the user-provided region definitions and proposed OSCS technique in each editing step.

Compared to DreamEditor, our Progressive3D theoretically performs better in the following aspects: **(1) Generalization:** Their method is carefully designed on mesh-based neural fields and costs extra time and quality loss for distilling other 3D representations to mesh-based neural fields. However, our Progressive3D is general to various 3D representations and region definitions if the corresponding projected depth and opacity can be obtained by rasterization or volume rendering.
**(2) Geometry variation:** Since their editable regions are defined on existing mesh, DreamEditor may suffer challenges when creating additional objects interacting with existing objects with rapid geometry variation, *e.g.*, editing *"an astronaut"* to *"an astronaut riding a motorcycle"*.
However, our Progressive3D proposes an initialization constraint $\mathcal{L}_{initial}$ and effectively tackles the desired rapid geometry variation (See in **Fig.3(d)&(f)** in revised paper).
**(3) Semantic Composition:** There are no specially designed modules to tackle the complicated semantics in DreamEditor, and multiple objects with different attributes may cause issues in text-driven region location and optimization directions.
The comparison with DreamEditor is not implemented due to the time limits, and we will add more comparison results and discussions in the final version.

Reviewer comment (Experiments):
Since the user can only input the box center and the lengths of the box along each axis, it seems that the box will always be axis aligned. Is this not problematic for certain edits that do not line up well with the axes?

Author response:
As described in **Sec.3.3**, the 3D bounding box can be rotated arbitrarily with the corresponding rotate matrix $R$, and the 3D bounding box and its rotation matrix can be obtained conveniently through a 3D GUI.

Reviewer comment (Novelty):
It would be helpful to clarify more how the semantic delta loss differs from (Armandpour et al.) [4]

Author response:
The commonality of OSCS and [3] is we both leverage the perpendicular component between different semantic components as the overlapped semantics. However, the design goal of OSCS and [3] is different, leading to differences in expression forms.
Concretely, we re-formulate their $\hat{\epsilon}^{Prep}\_\theta$ as $\hat{\epsilon}^{Prep}\_\theta(x_t, y_s, y_t, t)=\epsilon\_\theta(x_t, t) + \omega(\Delta\epsilon^s\_\theta+w\Delta\epsilon^{prep}\_\theta)$, and our $\hat{\epsilon}^{OSCS}\_\theta$ as $\hat{\epsilon}^{OSCS}\_\theta(x_t, y_s, y_t, t)=\epsilon_\theta(x_t, t) + \omega(\frac{1}{W}\Delta\epsilon^{proj}\_\theta+\Delta\epsilon^{prep}\_\theta).$
Therefore, their component of $y_s$ is the main optimization orientation and the perpendicular component is leveraged to supply auxiliary information, such as directions (*e.g.*, front, side, back).
However, we mainly focus on the perpendicular component for paying more attention to generating newly mentioned objects or attributes, and the projection component is weighted for harmonious interactives between target objects and generated objects.

Reviewer comment (Experiments):
Further comparisons with other methods for achieving complex semantic prompt-driven 3D content generation are lacking.

Author response:
We have provided the qualitative and quantitative comparison with DreamTime-based baselines including DreamTime+CEBM and DreamTime+A&E on CSP-100 in **Fig.6** and **Tab.1** in the revised paper.
Since composing T2I methods including CEBM [1] and A&E [2] are not designed for composing 3D content generation, combining composing T2I methods with DreamTime brings limited performance improvements (even regression), while DreamTime+Progressive3D is superior to all DreamTime-based baselines.
We analyze that although compositional T2I models produce more desirable results with complex prompts than normal T2I methods in each view, the inconsistency among views is still significant, leading to low-quality results.
However, Progressive3D reduces the generation difficulty since the current 3D content has been achieved ideally with the user-provided region definitions and proposed OSCS technique in each editing step.

Reviewer comment (Experiments):
The overall pipeline is relatively straightforward and simplified. Independently generating each local content and optimizing it after the combining each local 3D content with 3D bounding boxes may yield improved results, particularly for the prompt like some object is on a tabletop.

Author response:
We consider that our Progressive3D is simplified but effective.
Concretely, our Content Consistency Constraint is designed for the generalization on various text-to-3D methods and region definitions, our Content Initialization Constraint is proposed to reduce the generation difficulty when the geometry is changed rapidly, and our Overlapped Semantic Component Suppression is the core for progressive additional editing especially when the source prompt is already complicated.

We recognize that there is an alternative solution *i.e.,* separately generates objects and places them with some region definitions when the mentioned objects simply interact. However, Progressive3D can generate objects with complicated interacts, as shown in **Fig.7&10&11** in revised paper, which cannot be achieved by placing individual objects according to 3D boxes directly.

Reviewer comment (Experiments):
Is the progressive3D approach the only way to generate intricate 3D content? I want to see more comparative experiments to achieve a fairer comparison. For instance, one might first employ a complex semantic prompt to produce consistent images, then use these images to generate the corresponding 3D content. Alternatively, one could generate each object individually and subsequently merge them based on their bounding boxes, then the aggregated 3D content could be optimized to achieve a harmonious result. I am interested in understanding how to demonstrate that the progressive approach is a crucial and effective method for generating complex 3D content.

Author response:
We give the discussion of our Progressive3D and the paths you mentioned (**1**.Generating consistent multi-view images with complex prompts and **2**.generating coarse objects first and fine-tuning for harmony results) here.

For path **1**, the requirement of generating images with complex prompts conflicts with the multi-view consistency since the images with complex content tend to be inconsistent, and the prediction of geometry information including depths and normals is more difficult when the image is complicated.
Furthermore, we highlight that our Progressive3D is **orthogonal** to the improvement of multi-view images generator since MVDream (Submitted to ICLR2024) increases its 3D content generation capacity by leveraging a powerful multi-view images generator and we verified Progressive3D is available with MVDream and achieves impressive results in revised paper.

In addition, we consider that Progressive3D and path **2** can be separately seen as the 3D lifting of 2D progressive editing and 2D layout generation, where progressive editing is more controllable and layout generation is more time efficient.
As discussed in **Appx.A.4**, we have noticed such potential direction (3D layout generation) and will further explore it in future works.

Reviewer comment (Experiments):
How to resolve conflicts between subsequent generated results and earlier ones, such as an astronaut sitting on a red chair, when the first step generates a standing astronaut? Can the model optimize the transition from a standing astronaut to a sitting one?

Author response:
Generating harmonious 3D content with geometry variation (resolving conflicts between prompts) is a significant superiority compared to existing 3D editing methods including DreamEditor and FocalDreamer. 
We achieve such a goal by introducing $\mathcal{L}_{initial}$ and OSCS and the editing results with geometry variation (turn the standing pose into riding or sitting) are shown in **Fig.7&10&11** in revised paper.

Reviewer comment (Experiments):
In the context of complex semantic prompts, does the order of prompt input have an impact on the generated results, and is it necessary to engage in simple semantic prompt planning?

Author response:
We have provided discussion and visual results in **Fig.10** in revised paper.
We find that different object generating orders in Progressive3D typically result in correct 3D content consistent with the complex prompts.
However, the content details of the final content are impacted by created objects since Progressive3D is a local editing chain started from the source content.
With different generating orders, Progressive3D creates 3D content with different details while they are both consistent with the prompt *''*An astronaut sitting on a wooden chair''*, as shown in **Fig.10**.

Reviewer comment (Reproducibility):
The main concern is the heavy reliance on human involvement throughout the pipeline. Users are required to provide prompt divisions and bounding boxes, and the process seems user-unfriendly, as users have to wait for the previous generation to finish before providing the next bounding box prompt.

Author response:
Although we present many long-chain progressive editing processes to demonstrate our Progressive3D supports multi-step complex editing, users might edit the source 3D content only once or twice to achieve desired 3D content in practice.
As discussed in **Appx.A.1**, we consider Progressive3D accords with user realistic usage pipeline, *i.e.,* creating a primary object first, then adjusting its attribute or adding more related objects, and Progressive3D in practice is **flexible** for different user goals.
For instance in **Fig.9** in revised paper, we desire to create the 3D content consistent with the prompt *''An astronaut wearing a green top hat and riding a red horse''*.
We find that MVDream (Submitted to ICLR2024) fails to create the precise result while generating *''An astronaut riding a red horse''* correctly.
Thus the desired content can be achieved by editing *''An astronaut riding a red horse''* within one-step editing, instead of starting from *''an astronaut''*.

Furthermore, we underline that our Progressive3D framework supports **different region definitions** including 3D forms such as 3D boxes, custom meshes and 2D forms such as 2D segmentation and attention maps since the corresponding 2D region mask of each view can be obtained.
(See in **Sec.3.2** and **Appx.A.3**). 
We have provided editing samples in **Fig.11** in the revised paper that demonstrate Progressive3D is available with region defined by multiple 3D boxes, custom meshes, and 2D segmentation queried by the provided keyword, which can be convenient for region selection by users.

Reviewer comment (Novelty):
The paper mentions that current T2I diffusion models often struggle with complex prompts. However, there are existing methods [1,2,3] that address this problem. It would be beneficial to discuss why the authors did not directly utilize these methods, as it seems more straightforward and would save human labor. This discussion is currently missing from the paper.

Author response:
We have provided the qualitative and quantitative comparison with DreamTime-based baselines including DreamTime+CEBM and DreamTime+A&E on CSP-100 in **Fig.6** and **Tab.1** in the revised paper.
Since composing T2I methods including CEBM [1] and A&E [2] are not designed for composing 3D content generation, combining composing T2I methods with DreamTime brings limited performance improvements (even regression), while DreamTime+Progressive3D is superior to all DreamTime-based baselines.
We analyze that although compositional T2I models produce more desirable results with complex prompts than normal T2I methods in each view, the inconsistency among views is still significant, leading to low-quality results.
However, Progressive3D reduces the generation difficulty since the current 3D content has been achieved ideally with the user-provided region definitions and proposed OSCS technique in each editing step.

Reviewer comment (Evaluation):
Figures 7 and 10 show inconsistencies with the claim that undesired regions remain unchanged. The leg of the astronaut turns green, and the foot is missing after adding the prompt "and riding a red motorcycle."

Author response:
We claim that the changes of astronaut legs & feet are allowable since we omit the 3D bounding box in **Fig.7&10** in the original paper for brevity.
To achieve reasonable geometry variation with *'riding a motorcycle''*, *i.e.,* changing the astronaut from standing to riding, we have to select the legs & feet in the 3D box for editing.
The quality reduction is mainly caused by the poor generation capacity of TextMesh.
We further provided similar samples (turn the standing pose into riding or sitting) in **Fig.7&10&11** in revised paper and demonstrate that Progressive3D can produce harmonious geometry and appearance variations by combining stronger text-to-3D generation methods such as MVDream.

Reviewer comment (Experiments):
It would be more convincing if the paper showcased additional results (quantitative and qualitative) based on Fantasia3D. Given the low resolution of image space supervision in DreamTime and DreamFusion, the generated 3D assets appear blurry. Demonstrating the significant improvements offered by this method in more sophisticated Text-to-3D approaches would reinforce the paper's claims.

Author response:
We highlight that our Progressive3D is a **general framework** for progressive local editing for various text-to-3D methods driven by different 3D neural representations.
Thus the quality of our generated 3D content can be improved with the development of text-to-3D area.

**Fig.1&5** in revised paper demonstrate that our editing can produce high-quality 3D content based on Fantasia3D, and our method is also proved effective on other high-quality text-to-3D method like ProlificDreamer[3].
However, 3D results created by high-quality methods, especially ProlificDreamer severely suffered from **geometric inaccuracy problems** including multi-faces and floating noisys, even though their textures are impressive in a single view.
To generate stable results for quantitative comparison, we have to choose DreamTime as the baseline since DreamTime is relatively robust for different prompts and seeds.

We are glad to notice that concurrent works named MVDream (Submitted to ICLR2024) achieve impressive 3D generation performance, and we have verified that Progressive3D works well on MVDream, which further demonstrates the generalization of our Progressive3D. Please check our revised paper for more visual results.

Reviewer comment (Experiments):
Including an ablation study on the last term in the consistency loss (the one that imposes the empty region to be blank) would strengthen the paper's arguments.

Author response:
We have provided a discussion and visual ablation in **Appx.C.1** and **Fig.15** in revised paper.
We highlight that we divide the $\mathcal{L}_{consist}$ into a content term and an empty term to avoid mistakenly treating backgrounds as a part of foreground objects. 
The visual results demonstrate that mistakenly treating backgrounds as a part of foreground objects leads to significant floating.

Reviewer comment (Evaluation):
The quality of generated 3D objects is poor.

Author response:
**A1:** 
We highlight that our Progressive3D is a **general framework** for progressive local editing for various text-to-3D methods driven by different 3D neural  representations.
Thus the quality of our generated 3D content can be improved with the development of text-to-3D area.

**Fig.1&5** in revised paper demonstrate that our editing can produce high-quality 3D content based on Fantasia3D, and our method is also proved effective on other high-quality text-to-3D method like ProlificDreamer[1].
However, 3D results created by high-quality methods especially ProlificDreamer are severe suffered from **geometric inaccuracy problems** including multi-faces and floating noisys, even though their textures are impressive in a single view.
To generate stably results for quantitative comparison, we have to choose DreamTime as the baseline since DreamTime is relatively robust for different prompts and seeds.

We glad to notice that concurrent works named MVDream (Submitted to ICLR2024) achieves impressive 3D generation performance, and we have verfied that Progressive3D works well on MVDream, which further demonstrates the generalization of our Progressive3D. Please check our revised paper for more visual results.

Reviewer comment (Experiments):
The proposed method requires the 3D bounding box as the input, which is inflexible.

Author response:
**A2:**
We underline that our Progressive3D framework supports **different region definitions** including 3D forms (3D boxes, custom meshes) and 2D forms (2D segmentation and attention maps) since the corresponding 2D region mask of each view can be obtained.
(See in **Sec.3.2** and **Appx.A.3**). We adopt 3D boxes as region definitions in **Sec.3** for brevity, and the 3D bounding box can be obtained conveniently through a 3D GUI.
Differing from existing 3D editing methods, our $\mathcal{L}_{consist}$ is imposed on 2D rendered images and is effective for various 3D representations and region definitions (See discussion in **Appx.A.5**). 
We have provided editing samples in **Fig.11** in the revised paper that demonstrates Progressive3D is available with regions defined by multiple 3D boxes, custom meshes, and 2D segmentation queried by the provided keyword.

Noticing that user-provided masks and automatically calculated regions of interest are both widely used in 2D repainting/editing methods, we believe that defining a specific region instead of leveraging the inaccurate attention maps is necessary for many scenarios, and our framework also supports attention maps as the region defination.

Reviewer comment (Experiments):
It is difficult for Progressive3D to change the attribute of the generated 3D objects, such as changing red to blue or metal to wood. If we want to edit the attribute, we might need to train the model case by case.

Author response:
**A3:**
We emphasize that each step in Progressive3D is a **local editing step** and Progressive3D supports both **modifying attributes of existing objects** and **creating additional objects with attributes not mentioned in source prompts from scratch** in user-selected regions.
We have provided attribute editing results in **Fig.12** in revised paper.
Noticing that creating additional objects with attributes not mentioned in source prompts from scratch is more difficult than editing the attributes of existing objects.
Therefore, attribute editing costs significantly less time than additional object generation.

Reviewer comment (Experiments):
Only one dataset was used in the experiments.

Author response:
**A4:**
Considering the 3D editing task with complicated prompts has not been broadly studied, each work has to design specific settings and datasets to demonstrate the effectiveness of the proposed method.
We underline that our proposed CSP-100 is the first dataset for complex text-to-3D content generation.
For more convincing, we have provided more results on MVDream in the revised paper.

Reviewer comment (Reproducibility):
Given a complex prompt, how to decompose the complex text, automatically or manually? And how many steps are required?

Author response:
**A6:**
As discussed in **Appx.A.1**, the Progressive3D editing in practice is **flexible** for different user goals.
For instance in **Fig.9** in revised paper, we desire to create the 3D content consistent with the prompt *''An astronaut wearing a green top hat and riding a red horse''*.
We find that MVDream fails to create the precise result while generating *''An astronaut riding a red horse''* correctly.
Thus the desired content can be achieved by editing *''An astronaut riding a red horse''* within one-step editing, instead of starting from *''An astronaut''*.
Furthermore, as discussed in **Appx.A.2**, different object generating orders in Progressive3D typically result in correct 3D content consistent with the complex prompts (See in **Fig.10** in revised paper).
We provide a general prompt decompose criterion, *i.e.,* we first generate the primary object which is desired to occupy most of the space and interact with other additional objects.
In addition, the prompt decomposition can be achieved through LLMs in a similar way with BLIP-VQA (See in **Appx.B.2**).
In our experiments, we generate all results step-by-step manually and add one additional object in one step for quantitative evaluation.

Reviewer comment (Reproducibility):
How can we provide the 3D bounding box prompt, I think it is difficult. In my opinion, I think the 3D bounding box limits the application of the proposed method.

Author response:
**A2:**
We underline that our Progressive3D framework supports **different region definitions** including 3D forms (3D boxes, custom meshes) and 2D forms (2D segmentation and attention maps) since the corresponding 2D region mask of each view can be obtained.
(See in **Sec.3.2** and **Appx.A.3**). We adopt 3D boxes as region definitions in **Sec.3** for brevity, and the 3D bounding box can be obtained conveniently through a 3D GUI.
Differing from existing 3D editing methods, our $\mathcal{L}_{consist}$ is imposed on 2D rendered images and is effective for various 3D representations and region definitions (See discussion in **Appx.A.5**). 
We have provided editing samples in **Fig.11** in the revised paper that demonstrates Progressive3D is available with regions defined by multiple 3D boxes, custom meshes, and 2D segmentation queried by the provided keyword.

Noticing that user-provided masks and automatically calculated regions of interest are both widely used in 2D repainting/editing methods, we believe that defining a specific region instead of leveraging the inaccurate attention maps is necessary for many scenarios, and our framework also supports attention maps as the region defination.

Reviewer comment (Evaluation):
Can you report the CLIP-Score?

Author response:
**A7:**
We underline that CLIP is verified that fail to measure the fine-grained correspondences between objects and binding attributes.
For instance, we report the CLIP and our leveraged BLIP-VQA of the baseline result and our result with a specific prompt in Tab.1.

**Table 1:** Quantitative comparison on sample with prompt *"An astronaut holding a red rifle and riding a green origami motorcycle and wearing a cyan chef’s hat."* shown in **Fig.1** in revised paper.
| Method | CLIP| BLIP-VQA|
| -------| ---------|---|
| DreamTime | 0.340 | 0.062 |
| + Progressive3D | 0.312 | 0.556 |
||

We find that CLIP believes that the baseline result is more similar to the given prompts. 
However, 3D content generated by Progressive3D is significantly more accurate according to the visualization in **Fig.1** in revised paper, and BLIP-VQA shows consistent result with the visualization.
We report the CLIP result over CSP-100 in Tab.2.

**Table 2:** Quantitative comparison on metrics and user studies over CSP-100 
| Method | CLIP| 
| -------| --------------------- |
| DreamTime | 0.289 |
| + Progressive3D | 0.292 |
||