Paper: DreamFlow: High-quality text-to-3D generation by Approximating Probability Flow (ICLR-2024)

Reviewer comment (Novelty):
Although the authors are developing an optimization loss from a different angle, the algorithm they ultimately present is actually the same as Variational Score Distillation (VSD). Similarly, the results obtained by the authors are similar to those of VSD. From this perspective, it doesn't seem sufficiently novel.

Author response:
**[W1, Q2, Q4] Technical Contribution: Difference vs. VSD**

To clarify the difference between APFO and VSD, we provide a detailed explanation in Common Response 1. We also show how APFO differs from VSD with a time-step annealing schedule, either using a linearly decreasing schedule or using weighting priors such as DreamTime [1].

Reviewer comment (Novelty):
The author mentions that score distillation uses randomly drawn time steps at each update. However, there have already been many works that use a linear time schedule, which decreases from large t to small t during optimization. Additionally, there is a specialized work (DreamTime) that specifically studies this point. From this perspective, the technique does not appear to be original or novel.

Author response:
**[W2, Q1] Motivation and Technical Details**

We provide additional information on how the loss and gradient norms are implied in Common Response 2. At a high level, the convergence of the loss implies the convergence of the distribution to the data distribution, and the low-variance gradient norm implies the stability of the 3D optimization. We also show why high-variance loss and gradient norm in VSD could be problematic in Common Response 2, as this can lead to inconsistent 3D optimization and content shifting during optimization.

Reviewer comment (Novelty):
How is the time schedule related to your modeling? Is your model only applicable to a linear schedule? What distinguishes your method from the timestep annealing mentioned in works like DreamTime?

Author response:
**[W1, Q2, Q4] Technical Contribution: Difference vs. VSD**

To clarify the difference between APFO and VSD, we provide a detailed explanation in Common Response 1. We also show how APFO differs from VSD with a time-step annealing schedule, either using a linearly decreasing schedule or using weighting priors such as DreamTime [1].

Reviewer comment (Novelty):
Does your modeling approach have any broader applications? For instance, are there applications that could not be achieved in previous score distillation methods but might be feasible with your model?

Author response:
**[Q3] Broader application**

We demonstrate the advantage of our method that it achieves faster 3D optimization when using text-to-image diffusion priors. Our approach could be further applied to various applications in using the diffusion model priors to optimize visual generators (e.g., neural fields), such as text-to-video generation by optimizing video neural fields with video diffusion models, or text-to-4D generation using image and video diffusion models. We believe that exploring APFO for these tasks are interesting directions that we leave for future work. 

Furthermore, since our approach makes use of probability flow ODE, it could be more advantageous when using fast-sampling diffusion models such as distilled diffusion models (e.g., guided distillation [2] or consistency models [3]). On the other hand, the current score distillation method cannot achieve full advantage when used with these models.

Reviewer comment (Novelty):
Please elaborate on the differences and similarities between your method and VSD, as the final formulations appear to be nearly identical.

Author response:
**[W1, Q2, Q4] Technical Contribution: Difference vs. VSD**

To clarify the difference between APFO and VSD, we provide a detailed explanation in Common Response 1. We also show how APFO differs from VSD with a time-step annealing schedule, either using a linearly decreasing schedule or using weighting priors such as DreamTime [1].

Reviewer comment (Experiments):
Please elaborate in detail on where the improvements in experimental results come from. For example, why is it faster, why do you think the results are better than VSD, and have you conducted independent ablation studies? (The current ablation study seems to be of limited significance, as adding each stage naturally leads to better results, but this is not the focus or contribution of the paper).

Author response:
**[Q5] Why is the proposed method superior to VSD?**

As we discussed in Common Response 1, our approach allows for faster optimization as we approximate the probability flow ODE, which is a straight path towards data distribution, while VSD utilizes random timesteps in optimization. Also, in Common Response 2, we show that APFO achieves convergence to the data distribution with more stable 3D optimization, while VSD does not. Therefore, those explanations demonstrate how APFO presents faster and higher-quality text-to-3D optimization. Those claims are empirically supported in Tables 2 and 3 of our manuscript, where we provide experiments to compare APFO and VSD under the same setup and show that our approach presents better quality in the CLIP precision metric.

Reviewer comment (Novelty):
My main concern is that the proposed APFO is too similar with VSD. Maybe a more detailed explanation should be given to demonstrate the difference between VSD and APFO.

Author response:
**[W1] Comparison to VSD**

We clarify the difference between APFO and VSD in Common Response 1. We remark that the APFO leverages the time-derivative of noise scale derived from probability flow ODE in leveraging the diffusion model prior, as opposed to the weighting function in VSD. This leads to faster convergence and better fidelity as shown in 2D experiments.

Reviewer comment (Presentation):
The color of some results are kind of "green" or "brown" (including the cactus in Fig. 4, the corgi in Fig. 7, etc.), can you explain the reason?

Author response:
**[W2] Rationale for Color Bias**

Note that the color bias does not exist in our method, and different colors or textures can be generated due to the diversity of generation. However, as we have shown in the limitation part in Section 6, some undesirable results could be obtained due to the use of pre-trained text-to-image diffusion models.

Reviewer comment (Reproducibility):
What is the CFG scale used in 1/2/3 stages?

Author response:
**[Q1] CFG Scale**

For your information, we use a classifier-free guidance scale of 7.5 for each stage.

Reviewer comment (Reproducibility):
How is the GPU memory consumation of finetuning the LoRA of SDXL in stage3?

Author response:
**[Q2] Memory consumption**

For your information, our approach requires >30GB for stage 1 NeRF optimization, and less than 20GB for stages 2 and 3 3D mesh tuning. Note that since we use instant NGP for stage 1 NeRF optimization, the GPU memory may vary due to sparsity during optimization. We tested with an A100 40GB GPU.

Reviewer comment (Experiments):
Can you provide the 2D experiment of APFO (using APFO to optimize a 2D image)? A 2D experiment will demonstrate the effectiveness of APFO better.

Author response:
**[Q3] 2D experiments**

Following your suggestion, we provide 2D experiments to compare APFO with VSD and VSD with timestep annealing methods (see Common Response 1 and Appendix B of revised manuscripts for details). Note that APFO can produce high-fidelity 2D images with fewer steps than VSD without blurry artifacts.

Reviewer comment (Novelty):
While the authors provide insightful interpretations of the SDS method as an SB problem, its implementation is quite similar to the prior work ProlificDreamer. Additionally, timestep scheduling has also been proposed in previous works [1, 2, 3], limiting the impact of the proposed method.

Author response:
**[W1] Difference between VSD and comparison with previous works on decreasing noise schedule**

In Common Response 1, we clarify that APFO and VSD have different optimization objectives, and they are different in implementation as APFO is entailed with a predetermined decreasing timestep schedule. In addition, APFO not only differs from VSD together with a decreasing timestep schedule, but also is a better approach in leveraging the diffusion model prior. We also compare APFO with VSD + DreamTime [1], which uses priors on the weighting function. Our explanation with 2D experiments shows that APFO is not only different from VSD with timestep annealing, but also faster (in terms of optimization speed) and better at generating high-fidelity images.

Reviewer comment (Evaluation):
When viewed as an SB problem, the proposed method using a LoRA-fine-tuned score function seems to have difficulties in generating diverse results. Further explanations are encouraged.

Author response:
**[W2] Regarding the Diversity**

Our method does not suffer from diversity since we use small CFG scales (e.g. $\omega=7.5$) throughout the experiment.

Reviewer comment (Experiments):
The improved numerical scores, as shown in the paper, are modest and are computed based on only 20 3D assets. Furthermore, some results contain weird textures, as observed in the example of "A stack of pancakes covered in maple syrup."

Author response:
**[W3] Experimental Validation**

We conducted a human evaluation with 20 text prompts to compare with other baselines, where 10 pairwise comparisons were made for each prompt, thus 200 comparisons were made for each baseline. Our human evaluation experiment suggests that our method shows better photorealism than previous work. In fact, in Table 2 and Table 3, we perform an additional comparison with 100 text prompts taken from the DreamFusion gallery, which shows that our method outperforms others in the CLIP R-precision metric.

Reviewer comment (Experiments):
The results after the third stage exhibit color bias (Fig.5), and most generated 3D assets do not appear significantly improved after SDXL fine-tuning, except for the cottage example in Fig. 13. To explain this, additional ablation studies are recommended to fully evaluate the effectiveness of SDXL.

Author response:
**[W4] Additional ablation study on the effect of using SDXL**

To further demonstrate the effectiveness of Stage 3 mesh refinement, we provide more qualitative results in Appendix Figure 13 of our revised manuscript. Remark that Stage 3 mesh refinement results in more high-contrast 3D contents. Also, for quantitative evaluation, we conduct an ablation study by computing CLIP aesthetic score [2]. Similar to evaluation of CLIP R-precision, we render views and compute CLIP aesthetic scores and report average scores. Note that the average CLIP aesthetic score has improved after stage 3 mesh refinement from 4.82 to 5.02, which indicates the advantage of stage 3 mesh refinement. 
Nevertheless, as we have shown in the limitation part in Section 6, some undesirable results could be obtained due to the use of pre-trained text-to-image diffusion models.

Reviewer comment (Writing):
There appear to be typos in Eq.13, with "\Phi" possibly intended to be "\phi," and "q" possibly intended to be "p." Otherwise, clarification regarding the transition from Eq.12 to Eq.13 would be appreciated.

Author response:
**[Miscellaneous]**

Thank you for pointing out the typos and missing details. We will address these before the discussion period closes. Regarding the details in Eq. (12) to Eq. (13), it simply computes a gradient $dx/dt$ by inserting discrete time steps, and the loss is given by the difference between the generator $g(\theta,c)$ and the target generated by $x+\Delta t \cdot dx/dt$.

Reviewer comment (Evaluation):
In Fig.6, the stability of the APFO loss is attributed to the non-increasing timestep scheduling, as opposed to the two-stage stratified random scheduling used in the VSD loss from ProlificDreamer. However, the convergence of the loss curves may not precisely reflect the training quality of the 3D representation due to the random timestep sampling in VSD. Could the authors provide further clarification on this point?

Author response:
**[Q1] Gradient variance and the quality of 3D representation**

To clarify our points, in Common Response 2, we provide detailed analysis on how the convergence of loss and gradient norm indicates in 3D optimization. Indeed, the convergence of loss denotes the convergence of 3D views to the data distribution and the low-variance gradient norm indicates the stability of 3D optimization, where both implies superiority of APFO compared to VSD.

Reviewer comment (Experiments):
Based on the above question, what would be the outcome if the VSD loss were applied using the same timestep scheduling as proposed by the authors? Moreover, the predetermined timestep selection process remains unclear, particularly regarding the decreasing ratio over iterations.

Author response:
**[Q2-1] Comparison with VSD + timestep annealing**

Please refer to [W1] and Common Response 1 for detailed comparison on APFO with VSD and its variants with timestep annealing methods including DreamTime [1]. 

**[Q2-2] Timestep selection**
To clarify, we choose the time steps $t_1,\cdots,t_N$ such that they have decreasing variance $\sigma(t_1) >\cdots > \sigma(t_N)$. In general, diffusion models are trained with 1000 timesteps, and we denote the timestep as the ratio divided by 1000 (e.g., $t_1=1.0$ denotes the timestep at 1000, and $t_N=0.2$ denotes the timestep at 200). While we use $t_1=1.0$ when optimizing NeRF from scratch (i.e., step 1), we choose a smaller value of $t_1$ for fine-tuning the 3D mesh (i.e., steps 2 and 3). This is because starting from a large time step may alter the textures due to the addition of large amounts of noise and lose fidelity to the original 3D content from the previous stage. Empirically, we have found that $t_1=0.5$ is sufficient for stage 2 mesh refinement and $t_1=0.3$ is sufficient for stage 3 mesh refinement.