Paper: Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping (ICLR-2024)

Reviewer comment (Presentation):
The most important Figure in the paper, i.e. Figure 2, is unclear. Figure 2 is first referenced in Section 3.1, but it is very hard to understand the meaning and experiment setup of this Figure. It is unclear what the three rows respectively represent and what the task of Figure 2 is. Is the mesh fixed while only optimizing the texture? Many important details are missing, making it hard to follow the paper.

Author response:
**Q1: Description for Figure 2 is unclear.**

Thanks for pointing it out. For both Stable Diffusion and SDXL, we present images organized in 3 rows and 6 columns. Let’s explain them in detail one by one.

Row1: The setting is training a 2D image using SDS loss with different gradients.

Row2: The visualized gradients of the optimized image as shown in Row1.

Row3: The setting is training a texture field for a fixed mesh using SDS loss with different gradients.

Column1: The setting is training 2D image/3D texture field in latent space as Latent-NeRF.

Column2-6: training them in image space where the VAE encoder will be incorporated in gradient backward.

Column2: Using original gradient backward through VAE encoder. (VAE gradient)

Column3: Using a linear approximation for VAE encoder.

Column4: Using the pixel-wise normalized VAE gradient.

Column5: Using the pixel-wise clipped VAE gradient based on value.

Column6: Using the pixel-wise clipped VAE gradient based on norm.

We have made the caption of Figure 2 more clear in the revision.

Reviewer comment (Presentation):
Some other figures and descriptions are unclear, and more necessary details should be provided. In Figure 6, it is unclear what the meaning of the 6 images in the right column is — are they gradients? The author could also provide more illustrations to support Section 4.5.

Author response:
**Q2: Figure 6 is unclear.**

Thanks for pointing it out. Since VAE has two parts: encoder and decoder, we fit linear approximation for both parts. The left three columns are the results of decoder approximation where fitted and GT RGB images are presented. 
The right three columns are the results of encoder approximation where fitted and GT latent images are presented. 
We have made a clear description for them in the revision.

**Q3: More illustrations to support Section 4.5 (depth controlnet).**

We have provided two cases to illustrate the problem in Section 1.2 of the **separate supplementary file**. Depth controlnet improves the success rate of generating high quality meshes like the bottom of Figure 1. In our observation, only about 40% cases succeed without depth controlnet and the rate increases to 60%-70% by incorporating the depth controlnet. In the rest cases, while the quality is not as high as in Figure 1, our PGC could still  enhance  over the baseline.

Reviewer comment (Theory):
In Section 4.2, it states "we observe that the difference between x and \hat{x} remains strictly constrained within the interval of (0, 1) due to RGB restrictions". The reviewer believes the range of the difference should be in [-1,1] and asks if they missed something.

Author response:
**Q4: Typo of the range (0,1).**

Yes, it is a typo and should be [-1, 1]. We appreciate your careful reading and have fixed the typo in the revision.

Reviewer comment (Theory):
The reviewer asks if the author can provide insights on why "this constraint plays a crucial role in stabilizing the training process for the 3D model".

Author response:
**Q5: More insights of the constraint.**

Great comment. More specifically, we note that RGB values are normalized to the range of [0, 1] while the absolute value of the image gradients produced by SDS loss can vary from $0$ to $10^{5}$. If we optimize a 2D image directly using these gradients with SGD, the optimized image will quickly explode thereby leading to failure. Although sigmoid function and Adam optimizer are often used to guarantee the RGB constraints and smoother gradients (as all the experiments in the main paper), we think the extremely large and unbalanced gradients still do not make sense.

Reviewer comment (Theory):
The reviewer suggests that another way is to clip the gradient of the entire term, i.e. (eps_phi - eps) * dz/dx, rather than just dz/dx, and asks the author to comment on this alternative method.

Author response:
**Q6: Alternative method by clipping $(\epsilon_{\phi} -\epsilon) \frac{dz}{dx}$ rather than the  $\frac{dz}{dx}$.**

Apologies for this misunderstanding. In our design, we take operations on $w(t)(\epsilon_{\phi} -\epsilon) \frac{\partial z}{\partial x}$ which are the gradients of an image. The term $\frac{\partial z}{\partial x}$  is the Jocobian matrix of size $(128\times 128 \times 4) \times (1024\times 1024 \times 3)$ under SDXL setting. We have revised the relevant expressions in the paper.

Reviewer comment (Experiments):
Insufficient ablation study  
The paper reviews parameter-wise normalized Gradient Descent (NGD) and Gradient Clipping (GC) in Sec. 3.2 & 3.3 and proposes pixel-wise NGD and GC in Sec. 4.3 & 4.4. Pixel gradients are computed from parameter gradients, and bounded parameter gradients may result in bounded pixel gradients too -- an alternative of the proposed Sec. 4.3 & 4.4 to limit pixel gradient magnitudes. Therefore, parameter-wise NGD and GC would have served as good baselines for ablation study. However, the paper only showed results of pixel-wise NGD and GC but not results of parameter-wise NGD and GC or other simple ways to bound pixel gradients.

Author response:
**Q1: More ablations on parameter-wise NGD and GC.**

First we note that when we conduct 2D experiments (i.e. just optimizing a 2D image as the first row of Figure 2), “parameter-wise” design is equal to the “pixel-wise” counterpart because a pixel itself is a parameter to be optimized. For 3D experiments, parameter-wise NGD and GC will allow the extremely large and unbalanced gradients of the rendered image to be back-propagated until the gradients of the earliest parameters of NeRF/DMTet. This causes suboptimal results. 

Importantly, we provide the evidence in the Section 1.2 of the **separate supplementary file** (Figures 4) that parameter-wise NGD and GC cannot produce as detailed results as our PGC under the text-to-texture setting using SDXL.

Reviewer comment (Experiments):
Insufficient experiments to show that the proposed method "benefit existing SDS and LDM-based 3D generative models"  
The paper claims "PGC consistently benefit existing SDS and LDM-based 3D generative models", but experiments only compared with two baselines: Stable-DreamFusion and (SDXL variant of) Fantasia3D, which are not published state-of-the-arts and not sufficient to support the claim. For example, ProlificDreamer [Wang et al. 2023] and improved Fantasia3D (https://github.com/Gorilla-Lab-SCUT/Fantasia3D "Q7") can produce more realistic texture results than the compared baselines. Will the proposed PGC still be effective on top these two methods?

Author response:
**Q2: More comparisons.**

Thanks. We have now provided the results of ProlificDreamer and improved Fantasia3D in the Section 1.3 of the **separate supplementary file** (Figures 5 and 6).

For ProlificDreamer, we use Stable Diffusion as guidance. We observe that our PGC also stabilizes the training. We  cannot test the combination of SDXL and VSD due to the restriction of GPU memory. Overall, our method enables the unleashing of the potentials of the advanced diffusion model successfully.

For improved Fantasia3D, we have already tried the alternative weighting strategy: $w(t)=\frac{1}{\sigma_t^2}$ mentioned in their readme Q7. Since threestudio and stable-dreamfusion have not implemented this strategy, we did not include this for fair comparisons in our submission. In our experiments, we find that our proposed PGC consistently enhances  texture quality.

Reviewer comment (Novelty):
Suggestion to review general gradient-related techniques in Sec. 2  
Since this paper mostly focuses on addressing the gradient issues, I would suggest adding a paragraph in Sec. 2 to review general gradient-related techniques and their connections & differences with the proposed method. Example related papers include:  
[a] Zhang et al. Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. ICLR 2020.  
[b] Zhang et al. Improved analysis of clipping algorithms for non-convex optimization. NeurIPS 2020.  
[c] Brock et al. High-Performance Large-Scale Image Recognition Without Normalization. ICML 2021.  
[d] Koloskova et al. Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. ICML 2023.

Author response:
**Q3: Related work for gradient-related techniques.**

Thanks. We have added extra discussion in the related work following this suggestion. We note an important difference that previous works focus on model parameters while we manipulate the images –  the intermediate variables in the forward/backward process.

Reviewer comment (Writing):
Minor typographical error  
Typo: last paragraph before Sec. 4.3: "xt-1" --> "x_{t-1}"

Author response:
**Q5: Typo.**

We appreciate your careful reading. We have fixed the typo in the revision.

Reviewer comment (Evaluation):
I think it would be better if there is an in-depth analysis regarding the gradient control aspect of the proposed method. It would be beneficial to have more thorough consideration and analysis on how the gradient is influenced by the proposed method compared to the vanilla model. Presenting a more comprehensive investigation and analysis in this regard would enhance the paper.

Author response:
**Q1: In-depth analysis.**

We acknowledge that we find enhancement in the experiments. However, we have tried our best to explain the phenomenon in Section 4 of the main paper. 
First from the failure of the vanilla method and partial success of latent gradients/linear approximated gradients (Columns (i)(ii)(iii) of Figure 2), we find the gradient issue. Then through the numerical value (Table below) and visualization of the gradients (Row (b) of Figure 2), we establish the noise assumption (Section 4.4.1). Based on the assumption and equations 8&10, we can effectively remove the noise through equation 11. Figure 2 also shows how these gradients affect the generation quality.

To gain further insights, we analyze the distribution of per-pixel gradient norms through a histogram on all the images at different steps during one optimization on 2D experiments using SDXL (as the Row (a) of Figure 2). The table below illustrates our findings, revealing that 1.3% of pixel gradient norms surpass 1,000, predominantly falling outside the typical pixel value range of [0, 1]. Additionally, our investigation indicates that 99.8% of images contain at least one pixel with a gradient norm exceeding 10,000.
| >1  | >10  | >100 | >1,000 |
|  ----  | ----  | ----  | ----  |
| 64\% |  17\% |  6\%  |  1.3\% |

Reviewer comment (Evaluation):
It is disappointing that the stability of the gradient through PGC and PNGD is only demonstrated through the results without further analysis.

Author response:
**Q1: In-depth analysis.**

We acknowledge that we find enhancement in the experiments. However, we have tried our best to explain the phenomenon in Section 4 of the main paper. 
First from the failure of the vanilla method and partial success of latent gradients/linear approximated gradients (Columns (i)(ii)(iii) of Figure 2), we find the gradient issue. Then through the numerical value (Table below) and visualization of the gradients (Row (b) of Figure 2), we establish the noise assumption (Section 4.4.1). Based on the assumption and equations 8&10, we can effectively remove the noise through equation 11. Figure 2 also shows how these gradients affect the generation quality.

To gain further insights, we analyze the distribution of per-pixel gradient norms through a histogram on all the images at different steps during one optimization on 2D experiments using SDXL (as the Row (a) of Figure 2). The table below illustrates our findings, revealing that 1.3% of pixel gradient norms surpass 1,000, predominantly falling outside the typical pixel value range of [0, 1]. Additionally, our investigation indicates that 99.8% of images contain at least one pixel with a gradient norm exceeding 10,000.
| >1  | >10  | >100 | >1,000 |
|  ----  | ----  | ----  | ----  |
| 64\% |  17\% |  6\%  |  1.3\% |

Reviewer comment (Evaluation):
I am curious if there were any side effects that arose from applying the proposed method to 3D object generation.

Author response:
**Q2: Any side effects?**

In the mesh optimization where an initial mesh is provided, we observe a consistent improvement using our method. We did find side effects in NeRF optimization from scratch: sometimes the shape may vanish at the beginning when we use SDS or VSD loss with PGC. However, Stable-DreamFusion and Fantasia3D use (normal+mask) as latent for SDS in the early training. This provides a more stable shape initialization. For other pipelines (e.g. ProlificDreamer), we can also skip the early training stage and directly apply our proposed PGC in the later training.

Reviewer comment (Experiments):
I am curious if there are any experimental results for more unusual exceptional samples that were tested for comparison evaluation.

Author response:
**Q3: More results?**

Yes. Thanks. We have now provided more results in the **separate supplementary file** (Figures 7, 8, 9).