Paper: Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian distributions (ICLR-2024)

Reviewer comment (Theory):
Why asking -- in constants in proofs there are terms that have said Baron constant in the exponent. So if for densities that are highly non-convex -- we should expect curse of dimensionality to re-appear?

Author response:
We are very grateful to the reviewer for their encouraging feedback and insightful questions. We address your remarks as follows. 

- Note that the only exponential dependence in our constants is on the $L^{\infty}$ norm of $f$, not the Barron norm of $f$. The only dependence on the Barron norm of $f$ in our main result (Theorem 2) is **linear**.

Reviewer comment (Theory):
What is connection of densities that are $-||x||^2$+Baron class perturbation to $(\alpha, \beta)-$dissipative distributions? (convex outside of region)? It seems that this is sub-class of such, and if so, can authors elaborate what obstacles there is to generalise results on such class? (that are common in say analysing Langevin Dynamics generalization).

Author response:
We are very grateful to the reviewer for their encouraging feedback and insightful questions. We address your remarks as follows. 

- As we understand it, a measure $p = \frac{1}{Z} e^{-V(x)}$ is considered $(\alpha,\beta)$ dissipative if $\langle \nabla V(x), x \rangle \geq a\|x\|^{\alpha} - b$ for some $a,b \geq 0$, and $\nabla V$ is $\beta$-Holder continuous. In the case that the potential $V$ satisfies $V(x) = \|x\|^2/2 + f(x)$ where $f$ is a spectral Barron function, you are correct that the density is $(\alpha,\beta)$-dissipative: since $$|\nabla f|_{\infty} < \infty$$ 

for Barron functions, we find that
    $$\langle \nabla V(x), x \rangle \geq \frac{1}{2}\|x\|^2 - \frac{\|\nabla f\|_{\infty}}{2}$$

and similarly $V$ is Lipschitz with Lipschitz constant at most $(1+\|\nabla f\|_{\infty})$. Hence $\frac{1}{Z}e^{-\|x\|^2/2+f(x)}$ is $(\alpha,\beta)$-dissipative with $\alpha = 2$ and $\beta = 1$. However, our main assumption on the distribution is a low-complexity assumption, which general $(\alpha,\beta)$-dissipative distributions need not satisfy. Those SGMs for learning $(\alpha,\beta)$-dissipative distributions would in general suffer from curse of dimensionality without further low-dimension or low-complexity assumption.

Reviewer comment (Theory):
Baron class assumption needed only for universal approximation results about shallow network, right?

Author response:
We are very grateful to the reviewer for their encouraging feedback and insightful questions. We address your remarks as follows. 

- Yes, the Barron assumption is needed for the approximation properties of neural networks.

Reviewer comment (Theory):
The results have a very big limitation, which is the exponential dependence of the error bound on the $L^\infty$ norm of the perturbation $f$. To avoid the curse of dimensionality, this norm thus needs to be $O(1)$ in the dimension. However, in this setting, diffusion models are unnecessary, as much simpler classical techniques are sufficient. Indeed, the Holley-Stroock perturbation lemma [1] implies that the distribution $p(x)$ satisfies a logarithmic Sobolev inequality with a constant that is $\Omega(1)$ in the dimension. The score matching loss at time $t=0$ then controls the error in KL divergence [2], and knowing the score at time $t=0$ is sufficient to obtain sampling guarantees with Langevin/MALA algorithms [3,4]. Therefore, it is not necessary to approximate the scores at all times with a neural network.

Author response:
We do recognize that the $L^{\infty}$ norm of $f$ may depend on the dimension in many contexts. However, as we mentioned in our response to reviewer 1, such exponential dependence on dimension in the prefactors is generally unavoidable.
It is a sharp observation that, since the density we study satisfies LSI, it would be enough to learn the score at time 0 and run a Langevin process with the empirical score. To show that distributions which do not satisfy functional inequalities can be learned by SGMs without the curse of dimensionality is therefore an interesting open problem. However, we feel that our bounds are still meaningful because, in practice, one cannot use such structural information about the data to design the generative model, as it is unknown to the learner.

Reviewer comment (Theory):
After a quick assessment of the proof technique, it seems hard to remove this exponential dependence in this norm as the variance of the Monte-Carlo approximation of the integrals depends exponentially on the $L^\infty$ norm of $f$.

Author response:
We agree with the reviewer that, if the target distribution satisfies a LSI (with a constant which may be dimension-free), then Langevin dynamics with the true score converge exponentially fast, with a rate depending only on the LSI constant. However, in the generative modeling set-up, there is additional error incurred from estimating the score function, and under the LSI alone, this source of error may suffer from the curse of dimensionality. A consequence of our result is that this estimation error has a dimension-independent rate with respect to the number of samples when the density with respect to the Gaussian measure has low-complexity structure.

We have added a paragraph after Assumption 1 to explain this and provide further context for our results. 

Also, the typo in the added text is fixed. Thank you for pointing that out.

Reviewer comment (Theory):
Since $f$ is defined up to a constant, the $L^\infty$ norm of $f$ can be replaced with the "oscillation" of $f$ $(\sup f - \inf f)/2$, which makes the invariance of $p(x)$ on addition of constants to $f$ explicit.

Author response:
We have added a remark after the statement of Theorem 2 to mention that, without loss of generality, the $L^{\infty}$ norm of $f$ can be replaced with oscillation of $f$, i.e., that the $L^{\infty}$ norm of $f$ can be assumed minimal over all translates. Thank you very much for pointing this out to us.

Reviewer comment (Writing):
The specification of the function class for neural networks makes no mention of the intermediate widths. In general, I was initially confused whether they were assumed to be one or not, as the previous paragraph talks about real-valued shallow networks.

Author response:
About intermediate widths, the proof of Theorem 3 shows that for approximation, it is enough to assume the intermediate widths are all 1, since we're really just approximating a bunch of single-variable functions. However, this is obviously not done in practice, and it does not hurt our generalization argument to assume that intermediate widths are arbitrarily large, as long as the path norm at each layer is bounded independently of the width. We have added a statement about this in the paper (in order to keep under the page limit, we have made this comment in the appendix, specifically at the end of Appendix C).

Reviewer comment (Theory):
The equation (9) seems related to the Miyasawa/Tweedie relationship for the structure of the score (see e.g. equation (4) in [5]), as is commonly used in denoising score matching [6]. It would make the paper clearer to state that the network has to approximate the conditional expectation $\mathbb E[x_0 | x_t]$, which is essentially the ratio of the functions $F_t$ and $G_t$.

Author response:
We have stated in Section 3 that the network is estimating the conditional expectation $\mathbb{E}[x_0|x_t]$, which is basically just $F_t/G_t$ under our assumptions. We have also made clearer in Section 4.2 that the intermediate function approximation is only done locally, i.e., that the metric is the $L^{\infty}$ norm restricted to a ball.

Reviewer comment (Theory):
In section 4.2, it would help the reader to state that the approximations of the intermediate functions (product, exponential, etc) are in the $L^\infty$ norm restricted to balls.

Author response:
We have also made clearer in Section 4.2 that the intermediate function approximation is only done locally, i.e., that the metric is the $L^{\infty}$ norm restricted to a ball.

Reviewer comment (Writing):
Typos: Lemma 4, "K0"; Lemma 12, extra $|\ell_{R,S}$.

Author response:
Thank you for pointing out the typos; we have corrected them.

Reviewer comment (Theory):
It would help motivate the setting better if examples of non-trivial target distributions that satisfy the assumptions were provided. What are some well-known distributions which are Gaussians exponentially tilted by a “low-complexity” function from Barron space?

Author response:
We are very grateful for the positive feedback and helpful suggestions from the reviewer. We have addressed the remarks below:
-  We would first like to mention that, upon further inspection of the proofs, the approximation and generalization proofs can be adapted to the case that the function $f$ is allowed to grow linearly at infinity, say, if $|f(x)| \leq C_f \|x\|$ whenever $\|x\|$ is sufficiently large. Under this change, the main results can be extended to a larger class of distributions, where the function $f$ need not be globally bounded, at the expense of a slightly slower estimation rate. We will update the changes in the final revised version of the paper accordingly.

A motivating example that falls into the linear growth framework for $f$ is the case where the initial density is a Gaussian mixture. Suppose that
    $$ p(x) = \frac{1}{Z} \left( e^{-\|x-x_1\|^2/2} + e^{-\|x-x_2\|^2/2} \right), x_1, x_2 \in \mathbb{R}^d,.
    $$
    Then the log-relative density with respect to the standard normal distribution is
    $$f(x) = \log(1/Z) + \log \left( e^{-\|x-x_1\|^2/2} + e^{-\|x-x_2\|^2/2} \right) + \frac{\|x\|^2}{2}.$$

Reviewer comment (Theory):
The sudden appearance of L=8 in Theorem 2 is unnecessarily surprising and mysterious. It would be helpful if this was explained prior to Theorem 2. For example, one can mention that the score function is a composition of the tilting function, which can be approximated by a shallow network, and basic function/arithmetic primitives like exponentiation and multiplication, which can be implemented by stacking more layers.

Author response:
- The depth of the network constructed in Theorem 2 actually should be $L = 7$, and it is explained further in the proof of Theorem 3 in the paper; in essence, it arises from the compositional structure of the score function under our assumption, and the fact that every elementary function that we approximate (e.g., the exponential or product function) corresponds to a single layer in the final network. We are glad that you made us aware that this sudden introduction of depth appears unnatural to the reader, so we have added a brief explanation of where it comes from after the statement of the theorem.

Reviewer comment (Writing):
Typo in p.9: “allowed to be grow”

Author response:
- Thank you for pointing out the typo, and for suggesting the inclusion of a table of contents for the appendix in place of an extra section. We have edited the paper accordingly.

Reviewer comment (Presentation):
I would suggest simply adding a table of contents to the Appendix (there is a latex package for this) instead of Section B.

Author response:
- Thank you for pointing out the typo, and for suggesting the inclusion of a table of contents for the appendix in place of an extra section. We have edited the paper accordingly.

Reviewer comment (Theory):
Aside from the Barron space assumption, what conditions must the tilting function satisfy to ensure that corresponding function is a probability density? An immediate one seems to be subquadratic growth, but are there other conditions that one needs to or would want to assume?

Author response:
- As you point out, for $\frac{1}{Z}e^{-\|x\|^2/2 + f(x)}$ to be a density, we really just need $\|x\|^2/2 - f(x)$ to grow fast enough that $e^{-\|x\|^2/2 + f(x)}$ is integrable. The Barron assumption on $f$ immediately implies that $f$ is bounded hence leading to the function $e^{-|x|^2/2+f}$ being integrable.

Reviewer comment (Theory):
What are some well-known distributions which are Gaussians exponentially tilted by a “low-complexity” function from Barron space?

Author response:
We are very grateful for the positive feedback and helpful suggestions from the reviewer. We have addressed the remarks below:
-  We would first like to mention that, upon further inspection of the proofs, the approximation and generalization proofs can be adapted to the case that the function $f$ is allowed to grow linearly at infinity, say, if $|f(x)| \leq C_f \|x\|$ whenever $\|x\|$ is sufficiently large. Under this change, the main results can be extended to a larger class of distributions, where the function $f$ need not be globally bounded, at the expense of a slightly slower estimation rate. We will update the changes in the final revised version of the paper accordingly.

A motivating example that falls into the linear growth framework for $f$ is the case where the initial density is a Gaussian mixture. Suppose that
    $$ p(x) = \frac{1}{Z} \left( e^{-\|x-x_1\|^2/2} + e^{-\|x-x_2\|^2/2} \right), x_1, x_2 \in \mathbb{R}^d,.
    $$
    Then the log-relative density with respect to the standard normal distribution is
    $$f(x) = \log(1/Z) + \log \left( e^{-\|x-x_1\|^2/2} + e^{-\|x-x_2\|^2/2} \right) + \frac{\|x\|^2}{2}.$$

Reviewer comment (Theory):
Exponential dependence on $\\|f\\|\_{\\infty}$. The assumption that $\\|f\\|\_\\infty < \\infty$ seems extremely strong and the constants in this bound depend exponentially on $\\|f\\|\_{\\infty}$. As a thought experiment, suppose $p_0(x) = \\prod_{i=1}^d p_0^{(i)}(x_i)$ is the tensor product of 1-d distributions $p_0^{(i)}(x_i) \\propto e^{g(x_i)} \\gamma_1(dx_i)$. Then it is reasonable to expect $f(x) = \\sum_{i=1}^d g(x_i)$ can have $\\|f\\|\_{\\infty} = \\Omega(d) \\|g\\|_{\\infty}$ if the support is not restricted to a low-dimensional subspace. Because of this, it's not clear to me that the stated bounds can actually 'escape the curse of dimensionalty' or be 'dimension free' in non-trivial cases.

Author response:
It is true that the $L^{\infty}$ norm of $f$ may depend on the dimension for certain distributions. However, since the function $f$ is only defined up to an additive constant, we can replace the dependence on $\|f\|_{\infty}$ with the quantity $\frac{1}{2} \textrm{osc}(f)$, where $\textrm{osc}(f) := \sup f - \inf f$. 
- In addition, it is generally difficult to obtain sample complexity bounds where the prefactors do not depend exponentially on the dimension of the problem. In fact, in the classical nonparametric estimation of Sobolev densities, the $L^2$ error scales like $$O(\|p\|_{H^s}) n^{-\frac{s}{2s+d}}$$

where the Sobolev norm can grow exponentially in $d$. For example, in the example pointed out by the reviewer where $p_0(x) = \Pi_{i=1}^d p_0^{(1)}(x_i)$,  the $H^s$  norm of $p_0$ on the unit cube $[0,1]^d$ would be $O(C^d)$ where $C \sim \|p_0^{(1)}\|_{H^s([0,1])}$, leading to an exponential dependence of $d$ when $C>1$.

We stress here that our goal is to prove that, in the presence of low-complexity structure of the function $f$ defined in our paper, the **rate of convergence with respect to the growing sample size** is independent of the dimension $d$. Nonetheless, it is worth discussing the potential limitation of the constant factor $e^{\|f\|_{\infty}}$ in our bound and its comparison to classical density estimation, and we have added a remark to this end in the conclusion paper.

Reviewer comment (Theory):
No lower bound. It would also be helpful to prove (or at least discuss) lower bounds for this estimation problem. For example, do you know a minimax lower bound for estimating a generic function in $\\mathcal{F}\_{\\text{score}}(\\{K_i\\}_{i=1}^L)$? How does it compare to the generalization error of $\\hat{s}$? It is difficult make any takeaways from this work given that the upper bound is likely suboptimal and there is no lower bound to compare against.

Author response:
We thank the reviewer for pointing out the very interesting question on the minimax lower bound. First we would like to clarify that $\mathcal F_{score}(\{K_i\}_{i=1}^{L})$  is class of estimators, not the target function class of the score function. Right now, we do not have a minimax lower bound for estimating densities of the form $e^{-\|x\|^2/2 + f(x)}$ with $f$ Barron, nor do we have a lower bound for estimating score functions of such distributions. Our focus here is to derive generalization error bound for the sample quality of score-based generative models instead of the estimation error of the density function itself. The main takeaway message is that the rate of the generalization error between the generated distribution and the target data distribution is dimension-free when the target density has certain low-complexity structure whereas the estimation rate of density estimation under the standard Sobolev and Holder assumptions suffers from the curse of dimensionality.

We admit that the problem of obtaining a minimax lower bound of the generalization error for SGMs under our low-complexity assumption is very important and that understanding it would make the upper bound more meaningful, but it is highly non-trivial and requires deeper understanding of the approximation of the relevant multi-layer function space for the score function. We think investigation of this question is beyond the scope of the paper. We have noted this question as an interesting future direction in updated the conclusion section.

Reviewer comment (Theory):
Statement of Theorem 2: please define $\hat{p}_t$. The theorem statement includes $\text{TV}(\hat{p}\_0, p\_{t\_0})$, but based on the proof (pages 31, 32) it seems like it should be $\text{TV}(\hat{p}\_{t\_0}, p\_{0})$.

Author response:
We thank the reviewer for catching several typos and, in response to the helpful comments on notation, we decided to begin the proof of Theorem 2 with a list of all relevant distributions and their explicit definitions. Right now, this addition is in the proof of Theorem 2 in the appendix, not the main text, due to the page limit.

Reviewer comment (Theory):
Page 7, display following "Girsanov's theorem, combined with the data processing inequality" shouldn't this be an inequality instead of an equality? Also, there seems to be inconsistent notation regarding $\hat{p}\_{t_0}$, $\hat{p}\_{t\_0, T}$, etc -- please make the notation consistent and define it.

Author response:
We thank the reviewer for catching several typos and, in response to the helpful comments on notation, we decided to begin the proof of Theorem 2 with a list of all relevant distributions and their explicit definitions. Right now, this addition is in the proof of Theorem 2 in the appendix, not the main text, due to the page limit.