Paper: DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (ICLR-2024)

Reviewer comment (Experiments):
I am curious about the setup of the stage 3. If we already have a mesh with coarse texture, we can optimize it with differentiable mesh rendering and SDS loss, as Fantasia3d did in the appearance modeling stage. What is the advantage of the proposed refinement compared to this?

Author response:
**Q2: What is the advantage of the proposed refinement stage compared to Fantasia3D?**

The key advantage of our refinement stage lies in its efficiency.

Unlike other methods, which typically start without initial textures and require considerable time for generation, our approach builds on an existing mesh with coarse textures. For example, the texturing stage in Fantasia3D can take up to 20 minutes using 8 GPUs, whereas our refinement stage enhances the coarse texture in just 1 minute on a single GPU.

Reviewer comment (Experiments):
Another concurrent ICLR submission ( https://openreview.net/forum?id=pnwh3JspxT ) optimizes SDS much longer (1 hour and 40 minutes) than this paper. However, this paper claims that longer training does not give better results. Can the authors clarify the differences?

Author response:
**Q3: Differences compared to GSGEN which optimizes for longer time.**

There are several different designs that lead to the different conclusions about optimization time:

(1) GSGEN prioritizes high-quality text-to-3D generation, which is more challenging than image-to-3D. To address the Janus problem, they also generate a point cloud using Point-E [2] to initialize the Gaussians.

(2) They introduce a novel compactness-based densification strategy, which is more suitable for longer optimization. 

(3) In addition to the 2D SDS loss using Stable-diffusion, GSGEN applies a 3D SDS loss with Point-E [2], potentially prolonging each optimization step.

In contrast, our method emphasizes efficiency, trading off some level of detail. We also observe that text-to-3D typically requires longer optimization to achieve finer details, especially with an appropriate densification strategy.

Reviewer comment (Reproducibility):
Missing bits of context information - How much does it cost? While indicative timings and implementation details (covering experimental setup) are provided, information regarding the resource usage, model size and complexity are currently underdescribed. A comparative disclosure of such information covering the main experimental baselines that are considered would help the reader better assess its relative positioning throughout the typical criteria. Mentioning where the computation bottlenecks lie in terms of pipeline components would also be valuable in order to fully assess the practical usefulness of the proposed sequential pipeline, beyond rough timings.

Author response:
**Q1: Missing details about the model size, complexity, computation bottlenecks.**

Thanks for reminding us! 

The size of the generated 3D models varies depending on the input and training schedule. Typically, image-to-3D results involve around $10^4$ 3D Gaussians, while text-to-3D can reach up to $10^5$ Gaussians due to a more aggressive densification strategy.

For the mesh extraction, we apply remeshing and decimation as post-processing so the size is controlled. Specifically, we perform isotropic remeshing to an average edge length of $0.015$, and then decimate the number of faces to $10^5$.

The primary computational bottleneck in our pipeline remains the forwarding of the 2D diffusion model. We have broken down the time consumption for each operation per iteration as follows:

| Operation | Render 3D Gaussians | Run diffusion model | Loss backward |
| --------- | ------------------- | ------------------- | ------------- |
| Time (ms) | 3.8                 | 58.4                | 26.4          |

Our method primarily reduces generation time by requiring fewer iterations to converge (from several thousand to 500 steps). However, each iteration is still constrained by the 2D diffusion model.

For the mesh extraction, the time consumption for each operation is detailed as:

| Operation | Extract geometry | Unwarp UV | Extract texture |
| --------- | ---------------- | --------- | --------------- |
| Time (s)  | 7.9              | 4.6       | 4.0             |

Reviewer comment (Experiments):
Challenging the ad hoc meshing post processing. As it currently stands, the mesh extraction technique relies on many subsequent post-processing steps, including mesh decimation and remeshing (end of page 5 in the main paper). I believe the explanations around eq. (4) (before and after) could be further improved and detailed. My current intuition is that the mesh complexity at least could be controlled jointly during the density query step. Also, given the lack of statistics given regarding each step (Weakness 1 above), the relative need and ROI to fuse these steps is also hard to assess. The current procedure also produces non-manifold and non-watertight meshes with arbitrary complexity.

Author response:
**Q2: How to control the mesh complexity? How to solve non-watertight/manifold meshes?**

As mentioned in response to Q1, we control mesh complexity through post-processing, limiting the number of faces to $10^5$ through decimation. These post-processing details have been added to the implementation details.

We acknowledge the potential for non-manifold or non-watertight meshes in cases of poor Gaussian convergence, but we consider repairing it is less related to the goal of this paper and leave it for future work.

Reviewer comment (Evaluation):
Evaluation. The user study (i.e., subjective quality analysis) that is presented in the main paper and further detailed in the appendix is a good idea and often overlooked in the field. Nevertheless, its size and statistical informative validity are rather limited as they are based on a "cohort" of 40 users and 15 input samples to assess from.

Author response:
**Q3: The size of user study is limited.**

Thanks for the advice! We have conducted additional surveys with 20 more participants, bringing the total to 60 users. The updated results are included in the paper. Due to time constraints in the rebuttal period, we plan to release the data from our experiments for public comparison to facilitate broader assessment and validation.

Reviewer comment (Writing):
There are numerous minor language issues such as grammar mistakes and sometimes unnecessarily complicated sentence structure. This can easily be solved by a few rounds of careful proofreading.

Author response:
Thanks for your advice! The paper has been revised to enhance the writing quality. Additionally, we have included a preliminary section in the appendix to introduce relevant background details including SDS loss and mesh UV mapping.

Reviewer comment (Writing):
The work reads more like a paper written for a computer vision conference without providing sufficient background to a broader audience at the ICLR community. While this style is not unprecedented in ML, it makes the work much harder to access and misses an opportunity.

Author response:
Thanks for your advice! The paper has been revised to enhance the writing quality. Additionally, we have included a preliminary section in the appendix to introduce relevant background details including SDS loss and mesh UV mapping.

Reviewer comment (Writing):
The paper assumes the reader is familiar with all mentioned vision / graphics concepts such as UV space, without providing a proper background section.

Author response:
Thanks for your advice! The paper has been revised to enhance the writing quality. Additionally, we have included a preliminary section in the appendix to introduce relevant background details including SDS loss and mesh UV mapping.

Reviewer comment (Writing):
The decision to provide some background (e.g., on SDS loss) but only to an extent that is meaningful for people already familiar with DreamFusion limits accessibility; while in vision many of these things can be assumed known, it would be useful to the learning community to provide information here.

Author response:
Thanks for your advice! The paper has been revised to enhance the writing quality. Additionally, we have included a preliminary section in the appendix to introduce relevant background details including SDS loss and mesh UV mapping.

Reviewer comment (Writing):
When writing "we decrease the timestep t linearly, which is used to weight the random noise ϵ added to the rendered RGB image", it is not fully clear how this is performed.

Author response:
As answered in Q2, we linearly decrease $t$ from $0.98$ to $0.02$ with iteration step $i$ during training. 

This can be viewed a simplification of the TP-SDS algorithm proposed in Dreamtime [2]. Their findings suggest that larger values of $t$ are crucial for the global structure, whereas smaller values enhance local details. Hence, a decreasing schedule aligns the SDS noise level with NeRF optimization, facilitating a coarse-to-fine generation process.

Reviewer comment (Writing):
The SDS loss formulation is not clear without knowing the SDS loss; for example, the expectation is taken, among other variables, over p and t. The distribution of p and t is not specified.

Author response:
$p$ represents the camera pose to render the 3D Gaussians, which is uniformly sampled to orbit the object center. 

For example, for image-to-3D, the camera is sampled with a radius of $2$, a y-axis FOV of $46$ degree, with the azimuth in $[-180, 180] $ degree and the elevation in $[-30, 30]$ degree.

$t$ represents the timestep in denoising diffusion probabilistic model.

Early works like Dreamfusion [1] samples $t$ from $[0.02, 0.98]$ uniformly during training. We follow Dreamtime [2] to perform an annealing of  $t$ from $0.98$ to $0.02$ during training, which leads to faster convergence as shown in the ablation study (Figure 7).