Paper: Multi-Scale Generative Modeling in Wavelet Domain (ICLR-2024)

Reviewer comment (Theory):
The Gaussian noise assumption is not crucial for working diffusion models [1]. It would have been fair to mention as much.

Author response:
Thank you for sharing the interesting related work [1]. The work explores the possibility of using deterministic transformations instead of random Gaussian noise in diffusion models. We have added the work in the related work section in the revised version for a broader audience.

Reviewer comment (Theory):
Motivation for the proposed method. The authors claim repeatedly that low-frequency wavelet coefficients are better conditioned and 'more Gaussian,' and that high-frequency wavelet coefficients are sparse. To support the first claim, the authors show in Figure 5 that KL divergence between a unit gaussian $\mathcal{N}_0$, and a Gaussian with mean and covariance matched to the data $\mathcal{N}_1$, is decreasing as the scale increases. Why does $\text{KL}(\mathcal{N}_0 \mid \mathcal{N}_1)$ have anything to do with well-conditioning of covariance and/or Gaussianity of the data distribution? The discussion in A.4 is vague and extremely non-rigorous. The second claim, about sparsity of the high frequency coefficients, is discussed alongside some supporting citations, but it would be helpful to choose a subset of plots in Figure 6 to show in the body to at least demonstrate these claims empirically. Proposition 1 in A.5 only shows that high average sparsity of $x^k_H$ can imply high average sparsity of $x^k_H$ conditional on $x^k_S$, but I don't understand why this is relevant.

Author response:
**2.Motivation for the proposed method.**

Firstly, we would like to point out a potential misunderstanding here and emphasize the $KL$ divergence and Fig. 5 (Page 15) experimentally support our statements in Sec 3.2.2 (Page 5) that wavelet decomposition has the whitening effect on low-frequency coefficients. Moreover, we would also like to point out that $N_1$ is the Gaussian of normalized samples drawn from image datasets like CelebA-HQ. The normalization process is detailed in Appendix 10 (Page 23), in order to remove the effects of average pixel intensity and image dynamic range. As shown by Equation (78) (Page 23), the $KL$ divergence measures the similarity between the standard Gaussian $N_0$ and the normalized sample distribution $N_1$. Thus, a higher $KL$ divergence indicates higher non-Gaussianity of the sample distribution, and vice versa. We plotted the $KL$ divergence to support the claim of the whitening effect of multi-scale wavelet transform. As shown in Fig. 5, the LL subband at a coarser scale will have better Gaussianity.

Secondly, we would like to emphasize that Proposition 1 (Page 16) in Appendix A.5 (Page 16, 17) justifies the choice of GANO. In Eq. 13 (Page 6), we factorize the sampling process from true distribution p in the multi-scale wavelet domain, and the term $p(x_H^k|x_L^k)$ is highly non-Gaussian due to the property of high-frequency coefficients. Therefore we chose using a multi-scale GANO instead of other generative learning models such as diffusion models based on the non-Gaussianity of high-frequency coefficients. Combining the fact that $x_H^k$ is sparse, as shown in Fig. 6 (Page 17) and Proposition 1, one can easily draw the conclusion that $x_H^k|x_L^k$ has a high expected sparsity, and therefore supporting its high non-Gaussianity.

Reviewer comment (Experiments):
Experimental results. The comparison to Song et al. 2020 is unfair in light of much followup work on tuning score-based samplers (for example: Score SDE, Song et al. 2021). This method should also be compared to existing work on accelerating diffusion sampling with feed-forward nets, such as Consistency Models (Song et al. 2023) and SBGM in Latent Space (Vahdat et al. 2021). Also, in Figure 4, it's unclear whether the GAN upsampling step of the proposed method is counted as a sampling step. If the comparison is made between FIDs of SGM at 16 steps, WSGM at 16 steps, and 16 steps of WSGM for down-sampled images + GAN upsampling, then it is an unfair comparison in which the proposed method will obviously win. Ideally, the authors should compare to other methods that combine score-based sampling and feedforward nets, and they should demonstrate that wavelet-based architectures can augment this approach.

Author response:
**3.Experimental results.**

We would like to express our gratitude to the reviewer for bringing attention to the influential literature in the field. It is crucial to clarify that our approach fundamentally diverges from methods that expedite diffusion sampling through feed-forward networks. Instead of training a network to approximate the reverse diffusion process with larger temporal intervals, as most related works do, our methodology involves the refactorization of the image generation process in the multi-scale wavelet domain. Specifically, we have developed the SGM and GANO to handle the coarsest level of low-frequency and high-frequency coefficients, respectively. We have also undertaken a comprehensive theoretical analysis of coefficient distributions, providing a rationale and motivation for including GANO. It is essential to emphasize that GANO is an intrinsic and indispensable component of our approach for generating non-Gaussian high-frequency coefficients, rather than just a substitute for diffusion models.

Regarding the concern about "unfair comparison" you raised, our ‘sampling steps’ refers to the total diffusion steps calculated. For instance, in the case of WSGM, which consists of three SGM models, 16 sampling steps represent the total number of steps across these three SGM models. Similarly, we set the diffusion steps in our model to 16, ensuring a relatively fair comparison. For not including the sampling steps of the GANO model. The reason is two-fold: Firstly, the steps in a GANO model cannot be directly compared with those in a diffusion model due to fundamental differences in their theory and model structure. Secondly, compared to diffusion models, GANOs typically require less time, fewer resources, and have lower model complexity. In the Table 1 of our paper, our model's parameters amount to 75M (Diffusion model) + 14M (GANO), with sampling times of 3823ms (Diffusion model) + 352ms (GANO). Both in terms of parameter count and sampling time, our model is significantly more efficient than the other two models under consideration. Therefore, our method of comparison is relatively rigorous and aligns with the goal of our experiment: fast sampling.