Paper: Noise-free Score Distillation (ICLR-2024)

Reviewer comment (Experiments):
How did the authors separate the small and large timestep values based on t=200? Why not t=100, 300, or 400?

Author response:
**How did the authors separate the small and large timestep values based on $t=200$? Why not $t=100, 300$, or $400$?**

We added ablation studies in Appendix A.5, including an ablation on this threshold. As can be seen in the results presented in Figure 12, $t\_s=200$ provides the best tradeoff between fine details and artifacts.

Reviewer comment (Theory):
Is it valid to assume δ_{C=p_neg} ≈ −δ_{D}? Did the choice of the negative prompt affect the performance?

Author response:
**Is it valid to assume $\delta\_{C=p\_\text{neg}} ≈ −\delta\_\text{D}$? Did the choice of the negative prompt affect the performance?**

We added a discussion that motivates this choice in Appendix A.4. As explained there, there is a close relation between this assumption and the common technique of using negative prompts in ancestral sampling to improve image quality.
The negative prompt was kept fixed during all experiments (both images and NeRFs). Our method is not sensitive to the exact choice of words in the negative prompt, and other combinations of negative words do not have much impact.

Reviewer comment (Experiments):
How about changing (6) into just using the second part of (6) for all the time steps (i.e., unconditional term - negative-prompt-induced term, for all the time steps)?

Author response:
**How about changing (6) into just using the second part of (6) for all the time steps (i.e., unconditional term - negative-prompt-induced term, for all the time steps)?**

We added ablation studies in Appendix A.5. In the time threshold ablation presented in Figure 12 we included results for $t\_s=0$, in which the negative prompt term is applied for all timesteps.

Reviewer comment (Evaluation):
I appreciate the effort for many visual results; however, the lack of any quantitative results concerns me a lot. Is it possible to include the comparison using CLIP R-Precision of Table 1 in the DreamFusion paper? Furthermore, leveraging the MS-COCO text-to-image benchmark with FID/IS/CLIP score metrics may be worth trying to justify the results of 2D image generation in Figure 7.

Author response:
**Lack of any quantitative results concerns me a lot. Is it possible to include the comparison using CLIP R-Precision of Table 1 in the DreamFusion paper? Furthermore, leveraging the MS-COCO text-to-image benchmark with FID/IS/CLIP score metrics may be worth trying to justify the results of 2D image generation in Figure**

As mentioned in our conclusion section, quantitatively comparing NeRFs generated from text prompt remains a challenge due to the absence of suitable metrics and benchmarks. Additionally, most works do not provide an official implementation which makes the quantitative evaluation even more tricky. Therefore, we chose to provide a large gallery of results (with prompts *taken* from the baseline works) to allow the reader to get an impression and to directly assess the quality of our results compared to other works.
Yet, we add quantitative evaluation in Appendix A.6, measuring FID on generated images with NFSD and SDS-100, and presenting user study results, following Magic3D and ProlificDreamer.
As can be seen in Table 1, our method achieves lower (better) FID compared to SDS-100, confirming the better quality achieved by our method.
As can be seen in Table 2, our method also achieved the best results in both prompt alignment and image quality in the user study. We suspect that in the prompt alignment question responders were biased by the superior image quality of our results. Note that our results are aligned with the results shown in the user study in ProlificDreamer, in the sense that in both studies ProlificDreamer got better results than Fantasia3D, which, in turn, got better results than Magic3D.

Reviewer comment (Novelty):
Would a better generator g(\theta) be able to achieve the same effect, or train a model to estimate the bias \delta_{N}?

Author response:
We evaluated our approach with two generators $g(\theta)$, one that renders NeRFs, and the other is defined by $g(\theta) = \theta$. We used the same $p\_\text{neg}$ in both cases and observed that it generalizes across them.
It is worth mentioning that we observed that incorporating $\delta\_\text{D}$ for the directly optimized image was more significant compared with adding it to the optimized NeRF. This is related to the NeRF representation which naturally provides regularization on the rendered images. Therefore, it is true that our implementation for $\delta\_\text{D}$ affects differently on different generators $g$.
As mentioned in Section 5, training a model that estimates $\delta\_\text{N}$ is exactly the method introduced in ProlificDreamer. We believe that presenting ProlificDreamer with this formulation helps to gain more intuition over their method, and see this as one of our contributions.

Reviewer comment (Novelty):
While the explanation is intuitively presented, it remains somewhat challenging to discern the fundamental distinction from the negative prompt trick.

Author response:
Stemming from the presented score decomposition, our fundamental distinction is that $\delta\_\text{C}$ and $\delta\_\text{D}$ should be used to optimize the parameters $\theta$. While it is clear how to calculate $\delta\_\text{C}$, separating $\delta\_\text{D}$ from $\delta\_\text{N}$ remains a challenge.
We added a discussion about negative prompts in Appendix A.4, motivating our definition of $\delta\_\text{D}$ by relating it to the common practice of negative prompts.
Please note that we see the exact definition of $\delta\_\text{D}$ as an implementation detail for applying our main observation.

Reviewer comment (Experiments):
In Sec. 5, the paper asserts that NFSD is notably more efficient than VSD, despite sharing a similar working mechanism. Although this claim appears obvious, I would recommend providing quantitative evidence to substantiate this advantage when compared to other baseline methods. It is conceivable that dropping the noise term could even speed up the convergence of ancestral sampling by using fewer optimization steps.

Author response:
For 2D-image generation, trained on a single RTX2080 for 1K iterations, on average:

SDS - 2.9 [min]  NFSD - 4.3 [min] and Prolific - 8 [min]

For NeRF generation, trained on a single A100 for 25K iterations, on average:

SDS - 3.1 [hr], NFSD - 3.5 [hr] and Prolific - 4.5 [hr]

Please note that we did not optimize our implementation.
The noise term is undesirable for score distillation sampling. In contrast, for ancestral sampling, the score (comprising the noise term) is inherent to the step-by-step stochastic progression.

Reviewer comment (Experiments):
Further ablation studies are needed to validate the assertions put forth in this paper. In comparison to SDS, two terms have been omitted according to Eqs. 5 and 7: the noise prediction $\delta_N$ and the noise ground truth $\epsilon$. However, it remains unclear which of these terms plays the most pivotal role in improving the final results.

Author response:
In practice, the difference between SDS and our NFSD is that for $t < t\_s$ we do not subtract the added noise $\epsilon$, where $t\_s=200$, and for $t >= t\_s$ we subtract $\epsilon(z\_t, p\_\text{neg}, t)$ (instead of subtracting $\epsilon$). We added ablation studies in Appendix A.5, where we ablate the following:
* Subtracting $\epsilon(\mathbf{z}\_t, p\_\text{neg}, t)$ in all timesteps (instead of subtracting $\epsilon$ in SDS) ($t\_s=0$ in Figure 12)
* Not subtracting $\epsilon$ from SDS loss in all timesteps ($t\_s=1000$ in Figure 12)
* Change the threshold $t\_s$ (Figure 12)
* Using only $\delta\_\text{C}$ as the loss (leftmost column in Figure 13)

Reviewer comment (Theory):
Furthermore, it is not evident how steering the update of SDS could alter the optimization objective. Providing a more rigorous and formal argument would deepen the contribution of this work.

Author response:
We developed NFSD with the perspective of score function distillation, aiming to distill the score of a diffusion model for optimizing the parameters of rendered images. We observed that the score distilled in SDS includes a noise component, resulting in an averaging effect, leading to a local optimum that is overly smooth and lacks detail. NFSD attempts to avoid this noise component, allowing the optimization process to converge to a better local optimum.
The optimization objective for diffusion time $t < t_s$ is similar to SDS, while for $ t \geq t\_s$ it can be shown that $\nabla\_\theta \mathcal{L}\_{\textrm{NFSD}} =\nabla\_\theta \mathbb{E}\_{t}\left[w(t)\frac{\sigma\_t}{\alpha\_t}\textrm{KL}\left(q(\mathbf{z}\_t|g(\theta);y,t)\parallel \frac{1}{Z}\frac{p\_\phi(\mathbf{z}\_t;y,t)}{p\_\phi(\mathbf{z}\_t;y\_{neg},t)}\right)\right]$, where $Z$ is a normliazation factor. 
However, we find the optimized objective to lack intuitiveness and offer only little additional insight into NFSD. Therefore, we choose to focus on the score decomposition formulation.

Reviewer comment (Theory):
The authors introduce Eq. 6 to estimate $\delta_D$. Can the authors offer a rationale or justification for this approximation? Additionally, including visualizations that align with Fig. 3 would enhance the clarity and understanding of this proposal.

Author response:
Separating $\delta\_\text{D}$ from $\delta\_\text{N}$ is challenging. We found that for small enough timesteps (e.g., $t<200$) $\delta\_\text{N}$ is negligible and therefore $\epsilon(\mathbf{z}\_t, \varnothing, t) = \delta\_\text{N} + \delta\_\text{D}$ provides a good enough approximation for $\delta\_\text{D}$.
For other timesteps, we approximate $\delta\_\text{D}$ by $-\delta\_{C=p\_\text{neg}}$. The discussion we added in Appendix A.4 motivates the use of a negative prompt to define $\delta\_\text{D}$.
Additionally, to show the effect of our defined $\delta\_\text{D}$ in a similar manner to Figure 3, we added Figure 10  where we extract $\delta\_\text{D}$ from the definition in Eq. (6) (two rightmost images on the bottom row).

Reviewer comment (Experiments):
Thorough experiments to validate the effectiveness of NFSD are absent. Although the qualitative results show improved quality of text-to-NeRF than conventional SDS-based approaches, there is no ablation study and quantitative result.

Author response:
**W2. Thorough experiments to validate the effectiveness of NFSD are absent…There is no ablation study and quantitative result.**

We added ablation studies in Appendix A.5 and quantitative evaluation in Appendix A.6.

Reviewer comment (Experiments):
Although the authors discuss the low diversity of NFSD, I wonder the detailed reason why the reduced CFG scale cannot produce diverse visual contents. In addition, can the authors provide the samples with different seeds and the same text prompts to show the diversity of generated contents?

Author response:
**Q1.Why the reduced CFG scale cannot produce diverse visual contents. In addition, can the authors provide the samples with different seeds and the same text prompts to show the diversity of generated contents?**

Our approach actually sheds some light on the lack of diversity - it is not due to the high CFG. We hypothesize that the reason for the low diversity is rooted deeply in the process of score distillation. Unlike ancestral sampling, at each SDS iteration we add random noise to a rendered image, and we update the rendered image (or the generator parameters) independently of previous noises . Therefore, for a specific noised rendered image we follow some direction to optimize $p\_t(\mathbf{x}\_t)$. At the next step we again add (different) random noise and optimize $p\_t’(\mathbf{x}\_t’)$, which can yield a completely different direction. This results in an averaging effect (unlike ancestral sampling where each step corrects the previous as it is conditioned on it).
Moreover, in our work, as in most score sampling methods, even the same diffusion time (i.e. similar noise level) can be used with different noises during the optimization process, yielding a different direction each time. 
We added a discussion alongside results obtained from different seeds in Appendix A.3.

Reviewer comment (Experiments):
In Figure 3, what is the diffusion timestep? In addition, I think that the authors should show the results of 
$x_{\text{OOD}} + \delta_D + \delta_N^{\text{OOD}}$
, where $\delta_N^{\text{OOD}}$ is the denoising score of $x_\text{OOD}$, not $x_\text{ID}$. I also suggest clarifying the notation of $\delta_N$ and $\delta_D$ in Figure 3, since the two scores are from different samples.

Author response:
**Q2. In Figure 3, what is the diffusion timestep? ... show the results of $x\_{OOD}+\delta\_D + \delta\_N^{OOD}$ where $ֿ\delta\_\text{N}^{\text{OOD}}$  is the denoising score of $x\_{\text{OOD}}$ not $x\_{\text{ID}}$. I also suggest clarifying the notation of $\delta\_\text{D}$ and $\delta\_\text{N}$  in Figure 3, since the two scores are from different samples**

The diffusion timestep used is $t=400$. Indeed, this should be specified in the paper.
In Figure 3 we propose to visualize $\delta\_\text{N}$ and $\delta\_\text{D}$ by an engineered pair of $\mathbf{x}\_\text{ID}$ and $\mathbf{x}\_\text{OOD}$ images. We set $\delta\_\text{N}$ to be the denoising score of $\mathbf{x}\_\text{ID}$ as the image is in-domain and the prediction mostly just removes noise. We assume that $ֿ\delta\_\text{N}$ is shared between the in-domain and out-of-domain predictions, thus $\delta\_\text{D}$ is obtained as the difference between predictions of $\mathbf{x}\_\text{OOD}$ and $\mathbf{x}\_\text{ID}$. We explain this more clearly in the revised version. 
Regarding the term $\mathbf{x}\_\text{OOD}+\delta\_\text{D} + \delta\_\text{N}$, please note that this includes a noise direction term, thus the resulting image is noisy, as we show in Figure 10 in our revision.

Reviewer comment (Evaluation):
Why do the prediction errors in Figure 4 (the second row) show a less-noisy map at t=1000? I think that the results are unintuitive, since they indicate that the score prediction at t=1 is difficult, while the score prediction at t=1000 is conducted almost perfectly except for the central region.

Author response:
**Q3. Why do the prediction errors in Figure 4 (the second row) show a less-noisy map at t=1000?**

Although the initial impression may appear counterintuitive, it's important to bear in mind that these are unnormalized disparities between the noise $\epsilon$ and the network's prediction $\epsilon\_{\phi}$—meaning, they haven't been adjusted in relation to the diffusion noise scheduler. Predicting the precise added noise to the image becomes notably more challenging when a small amount of noise is introduced. In contrast, at $t=1000$, the noised image predominantly consists of noise ($\epsilon \approx \mathbf{x}\_t$), simplifying the prediction process. as the image is heavily influenced by the noise itself.

Reviewer comment (Theory):
In Section 4, the authors claim that the magnitude of the noise to be removed is monotonically decreased in the backward process. I wonder how we can assume that the scale of the domain score is preserved? Is there any rationale that only $\delta_N$ decreases over the backward process, while $\delta_D$ preserves its scale?

Author response:
**Q4. How can we assume that the scale of the domain score is preserved? Is there any rationale that only  $\delta\_N$ decreases over the backward process, while $\delta\_\text{D}$ preserves its scale?**

You are correct that there is no guarantee that $\delta\_\text{D}$ will preserve its scale. Nonetheless, our main point is that since the noise to be removed is monotonically decreasing in the backward process (according to the noise-scheduler), the noise prediction $\delta\_\text{N}$ part of the score (corresponding to $\epsilon\_{\phi}$) is becoming increasingly negligible as diffusion time becomes smaller. In contrast, the magnitude of the $\delta\_D$ component depends both on the diffusion time and the current stage of the optimization process. Thank you for this important comment, and we have revised our paper accordingly.

Reviewer comment (Experiments):
The authors have discussed that ProlificDreamer’s LoRA adaptation has a similar role with NFSD to exclude the prediction error of the denoising term $\delta_N$. Then, can the LoRA of ProlificDreamer be replaced with NFSD, while variational particle optimization is used? It would be interesting to show the compatibility of NFSD with ProlificDreamer.

Author response:
**Q6. Can the LoRA of ProlificDreamer be replaced with NFSD, while variational particle optimization is used?** 

The variational particle optimization proposed in ProlificDreamer is entangled with the LoRA variational model, as it is designed to model possible 3D representations (rather than one). 
Hence, applying NFSD for multiple particles can not be directly transferred from ProlificDreamer to our method.
A possible approach for multiple particles was proposed by [Kim et al](https://arxiv.org/abs/2307.04787). This is a general multi-particle approach for SDS-based methods, which can be combined with NFSD, but is out-of-the scope of our work.

Reviewer comment (Evaluation):
Since NFSD requires additional inference at each training iteration due to negative prompting, I think that comparing the results of NFSD with those of SDS in terms of the number of function evaluations (NFEs) of diffusion models.

Author response:
**Q7. Comparing the results of NFSD with those of SDS in terms of the number of function evaluations (NFEs) of diffusion models.**

To the best of our knowledge, there isn't a method available for automatically determining the optimal number of iterations for SDS-based methods.
In all our experiments, we use the same fixed number of iterations both for SDS and NFSD, resulting in an equal count of parameter updates for both methodologies.
The term "NFEs" of diffusion models might be misleading here, given that the outcome of the diffusion model functions merely as a guiding factor without involving subsequent backpropagation through it. Theoretically, NFSD necessitates an additional evaluation of the negative prompt (in the worst-case scenario, as the negative prompt isn't utilized for diffusion times below $t\_s = 200$). Consequently, NFEs(NFSD) equate to 1.5 times NFEs(SDS). 
Increasing the iteration count did not yield improvements in the SDS results; On the contrary, it tended to introduce artifacts.

Reviewer comment (Theory):
In Section 4, how can we assume that the score prediction on text conditions is also composed of $\delta_D + \delta_N + \delta_C$, where $\delta_D + \delta_N$ is equal to the unconditional prediction? I think that it is a technical flaw, since Eq.(3) just implies $\epsilon_\phi (z_t ; t) - \epsilon_\phi (z_t  ; y=p_\text{neg}, t) = \delta_{C=p_\text{neg}}$. That is, $\delta_C$ is defined with both conditional and unconditional scores, not solely on the conditional score term.

Author response:
**Q8. In Section 4, how can we assume that the score prediction on text conditions is also composed of  $\delta\_\text{D} + \delta\_\text{N} + \delta\_\text{C}$, where $\delta\_\text{D} + \delta\_\text{N}$ is equal to the unconditional prediction?** 

Thank you for the clarification. Indeed, $\delta\_\text{C}$ is defined as $\epsilon\_\phi(\mathbf{z}\_t;y,t) - \epsilon\_\phi(\mathbf{z}\_t;\varnothing,t)$. Plugging $y=p\_{\text{neg}}$ into the definition directly gives $\epsilon\_\phi(\mathbf{z}\_t;\varnothing,t) - \epsilon\_\phi(\mathbf{z}\_t;y=p\_\text{neg},t) = -\delta\_{c=p\_\text{neg}}$. Therefore, applying the score decomposition on the terms before Equation (6) is unnecessary and we fixed it in the revision.

Reviewer comment (Theory):
How is the negative prompt to estimate the domain term defined? I wonder whether the negative prompt is universal regardless of the image renderer. In addition, it assumes that the domain score can be estimated by the text prompts. However, how can we say that the image is from out-of-distribution, when the image can be estimated by text prompts of text-to-image models?

Author response:
**Q9. How is the negative prompt to estimate the domain term defined? 
How can we say that the image is from out-of-distribution, when the image can be estimated by text prompts of text-to-image models?**

We use the same prompt for all experiments (3D and 2D). The prompt is simply a collection of synonym words for “low quality” image.
Given a specific prompt, e.g. “a photo of a panda”, the score of the diffusion model will be in the direction that maximizes the likelihood of the sample given the prompt, i.e. it will point to a dense region of the conditioned distribution. The term out-of-distribution refers to this conditioned distribution. While low quality images with noticeable artifacts can be generated by a diffusion model (simply input the negative prompt), under a “normal” condition e.g. “a photo of a panda”, this is highly unlikely and thus we term it as out-of-distribution.

Reviewer comment (Theory):
Can be simply using $s \delta_C$ for the score distillation possible without $\delta_D$? That is, using $\delta_D$ is necessary?

Author response:
**Q10. Can be simply using $s\delta\_\text{C}$   for the score distillation possible without $\delta\_\text{D}$? That is, is using  $\delta\_\text{D}$  is necessary?**

We added ablation studies in Appendix A.5, including an experiment for using $s\delta\_\text{C}$ only for 2D-latent in Figure 13. While the generated image is consistent with the prompt it contains artifacts, indicating that $\delta\_\text{D}$ term is necessary for obtaining high-quality results.
We do stress that the effect of $\delta\_\text{D}$ can be achieved by other means of regularization, not only the diffusion model. For example, using additional losses for smoothness regularization can be used. Specifically in NeRF optimization, the parameter space and other losses (e.g. sprasity and opacity losses) can be used to regularize the results, serving as $\delta\_\text{D}$.