Learning (Very) Simple Generative Models Is Hard
Abstract
Motivated by the recent empirical successes of deep generative models, we study the computational complexity of the following unsupervised learning problem.
For an unknown neural network F : R d → R d ′ , let D be the distribution over R d ′ given by pushing the standard Gaussian N (0 , Id d ) through F .
Given i.i.d.
samples from D , the goal is to output any distribution close to D in statistical distance.
We show under the statistical query (SQ) model that no polynomial-time algorithm can solve this problem even when the output coordinates of F are one-hidden-layer
ReLU networks with log( d ) neurons.
Previously, the best lower bounds for this problem simply followed from lower bounds for supervised learning and required at least two hidden layers and poly( d ) neurons [CGKM22, DV21].
The key ingredient in our proof is an ODE-based construction of a compactly supported, piecewise-linear function f with polynomially-bounded slopes such that the pushforward of N (0 , 1) under f matches all low-degree moments of N (0 , 1) .
Introduction
In recent years, deep generative models such as variational autoencoders, generative adversarial networks and normalizing flows [GPAM + 14, KW13, RM15] have seen incredible success in modeling real world data.
These work by learning a parametric transformation (e g. a neural network) of a simple distribution, usually a standard normal random variable, into a complex and high-dimensional one.
The learned distributions have been shown to be shockingly effective at modeling real world data.
The success of these generative models begs the following question: when is it possible to learn such a distribution?
Not only is this a very natural question from a learning-theoretic perspective, but understanding this may also lead to more direct methods to learn generative models for real data.
More formally, we consider the following problem.
Let D be the unknown pushforward distribution over R d given by f ( g ) , where g ∼ N (0 , Id ) is a standard normal Gaussian, and f is an unknown feedforward neural network with non-linear (typically ReLU) activations.
Such distributions naturally arise as the output of many common deep generative models in practice.
The learner is given n samples from D , and their goal is to output the description of some distribution which is close to D .
When f is a one-layer network (i.e. of the form f ( g )
=
ReLU( Wg ) ), there are efficient algorithms for learning the distribution [WDS19, LLDD20].
However, this setting is unsatisfactory in many ways, as one-layer networks lack much of the complex structure that makes these generative models so appealing in practice.
Indeed, when the neural network only has a single layer, the resulting distribution is similar to a truncated Gaussian, and one can leverage techniques developed for learning from truncated samples.
Notably, this structure disappears even with two-layer neural networks.
Even in the two-layer case, despite significant interest, very little is known about how to learn D efficiently.
In fact, a recent line of work suggests that learning neural network pushforwards of Gaussians may be an inherently difficult computational task.
Recent results of [DV21, CGKM22] show hardness of supervised learning from labeled Gaussian examples under cryptographic asssumptions, and the latter also demonstrates hardness for all statistical query (SQ) algorithms (see Section 1.3 for a more 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
detailed description of related work).
These naturally imply hardness in the unsupervised setting (see supplement).
However, these lower bound constructions still have their downsides.
For one, all of these constructions require at least three layers (i.e. two hidden layers), and so leave open the possibility that efficient learning is possible when the neural network only has one hidden layer.
Additionally, the resulting neural networks in these constructions are quite complicated.
In particular, the size of the neural networks in these hard instances, that is, the number of hidden nonlinear activations for any output coordinate, must be polynomially large.
This begs the natural question:
Can we learn pushforwards of Gaussians under one-hidden-layer neural networks of small size?
1.1 Our Results
We demonstrate strong evidence that despite the simplicity of the setting, this learning task is already computationally intractable.
We show there is no polynomial-time statistical query (SQ) algorithm which can learn the distribution of f ( g ) , when g ∼ N (0 , Id ) and each output coordinate of f is a one-hidden-layer neural network of logarithmic hidden size .
We formally define the SQ model in Definition 2; we note that it is well-known to capture almost all popular learning algorithms [FGR + 17].
Theorem 1.1 (informal, see Theorem 3.1) .
For any d > 0 , and any C ≥ 1 , there exists a family of one-hidden-layer neural networks F from R d to R d C so that the following properties hold.
For any f ∈ F , let D f denote the distribution of f ( g ) , for g ∼ N (0 , Id ) .
Then, we have that:
• For all f ∈ F , d TV ( N (0 , Id ) , D f ) = Ω(1) ,• Every output coordinate of f is a sum of O (log d/ log log d ) ReLUs, with poly( d ) -bounded weights.
• Any SQ algorithm which can distinguish between D f and N (0 , Id ) with high probability for all f ∈ F requires d ω (1) time and/or samples.
In other words, there is a family of one-hidden-layer ReLU networks of logarithmic size whose corresponding pushforwards are statistically very far from Gaussian, yet no efficient SQ algorithm can distinguish them from a Gaussian.
Note this implies hardness even of improperly learning such pushforwards: not only is it hard to recover the parameters of the network or output a network close to the underlying distribution D f , but it is hard to learn any distribution close to D f .
In contrast, if one ignores issues of computational efficiency, one can easily learn networks in this family with polynomial sample complexity via hypothesis selection, as there are only poly( d ) many parameters specifying any network in this family.
Since such networks are arguably some of the simplest neural networks with more than one layer, this suggests that learning even the most basic deep generative models may already be a very difficult task, at least without additional assumptions.
Still, this is by no means the last word in this direction.
Given the real world success of deep generative models, a natural and important direction is to identify natural conditions under which we can efficiently learn.
We view our results as a first step towards understanding the computational landscape of this important learning problem, and our result provides evidence that (strong) assumptions need to be made on f for the pushforward f ( g ) to be efficiently learnable, even in very simple two-layer cases .
1.2 Our Techniques
Like many recent SQ lower bounds, ours follows the general framework which was introduced in [DKS17] and builds on [FGR + 17].
Here one considers the following “non-Gaussian component analysis” task.
We will consider a family of distributions, parametrized by a “hidden” direction v ∈ S d − 1 , which are Gaussian in every direction orthogonal to the hidden direction but which are very non-Gaussian along the hidden direction.
To formalize this, let D be a distribution D over R which is known to the learner and far in statistical distance from Gaussian.
Given a unit vector v in d dimensions, let P Dv denote the distribution over R d whose projection along v is given by D and whose projection in all directions orthogonal to v is standard Gaussian.
Given samples from some unknown distribution over R d , the goal is to decide whether the unknown distribution is N (0 , Id d ) or P Dv for some v .
[DKS17] showed that if
D ’s moments match those of N (0 , 1) up to some degree m , then under mild conditions, any SQ algorithm for this task requires at least d Ω( m ) queries (Lemma 3.3).
Suppose one could exhibit a one-hidden-layer ReLU network f : R ℓ → R such that the pushforward
D = f
( N (0 , Id )) satisfied such properties.
Then we can realize P Dv as a pushforward as follows.
Let U be a rotation mapping the first standard basis vector in R d to v .
Then consider the function
F : R ℓ + d − 1 → R d mapping z to U · ( f ( z 1 , . . . , z ℓ ) , z ℓ +1 , . .
, z ℓ + d − 1 ) .
One can check that every output coordinate of F is computed by a one-hidden-layer ReLU network with size essentially equal to that of f .
By [DKS17], we would immediately get the desired SQ lower bound.
The main challenge is thus to construct such a network whose pushforward matches the low-degree moments of N (0 , 1) .
It is not hard to ensure the existence of such f with essentially infinite weights (Corollary 4.2 and Lemma 4.4).
It is much less clear whether this is possible with polynomially bounded weights, and this is our primary technical contribution.
We design and analyze a certain
ODE which defines a one-parameter family of perturbations to f , such that the low-degree moments of the corresponding pushforwards remain unchanged over time.
By evolving along this family over an inverse-polynomial time scale, we obtain a network with polynomially bounded weights whose pushforward matches the low-degree moments of N (0 , 1) .
We defer the details to Section 4.
1.3 Related Work
A full literature review on the theory of learning deep generative models is beyond the scope of this paper (see e.g. the survey of [GSW + 21]).
For conciseness we cover only the most relevant papers.
Upper bounds.
In terms of upper bounds, much of the literature has focused on a different setting, where the goal is to understand when first order dynamics can learn toy generative models [FFGT17, DISZ17, GHP + 19, LLDD20, AZL21, JMGL22], which are much simpler than the ones we consider here.
For learning pushforwards of Gaussians under neural networks with ReLU activations, algorithms with provable guarantees are only known when the network has no hidden layers [WDS19, LLDD20].
This is in contrast to the supervised setting, where fixed parameter tractable algorithms are known for learning ReLU networks of arbitrary depth [CKM22].
A different line of work seeks to find efficient learning algorithms when the activations are given by low degree polynomials [FFGT17, LD20, CLLZ22].
Arguably the closest to our work is [CLLZ22], which gives polynomial-time algorithms for learning low-degree polynomial transformations of Gaussians, in a smoothed setting.
It is a very interesting open question if similar smoothed assumptions can be leveraged to circumvent our lower bound when the activations are ReLU.
Unfortunately, these papers heavily leverage the nice moment structure of low-degree Gaussian polynomials, and it is unclear how their techniques can generalize to different activations.
Lower bounds.
Much of the literature on lower bounds for learning neural networks has focused on the supervised setting, where a learner is given labeled examples ( x, f ( x )) , and the goal is to output a good predictor.
There are many lower bounds known in the distribution-free setting [BR92, Vu98,
KS09, LSSS14, DV20], however, these do not transfer over to our (unsupervised) setting.
When x is
Gaussian, the aforementioned work of [CGKM22] derives hardness for learning two-hidden-layer networks with polynomial size for all SQ algorithms, as well as under cryptographic assumptions (see also [DV21]).
It is not hard to show (see supplement) that this lower bound immediately implies a lower bound for the unsupervised problem.
In the supervised setting, lower bounds are also known against restricted families of SQ [VW19, GGJ + 20, DKKZ20, SVWX17], when there are adversarially noisy labels [KK14, DKZ20, GGK20, SZB21], and in discrete settings [Val84, Kha95,
AK95, Fel09, CGV15, DGKP20, AAK21], but to our knowledge, these results do not transfer to our setting.
The literature on lower bounds for the unsupervised problem we consider here is much sparser.
Besides [DV21, CGKM22], we also mention the recent work of [CLLM22] that studies whether achieving small Wasserstein GAN loss implies distribution learning.
A corollary of their results is cryptographic hardness for learning pushforwards of Gaussians under networks with constant depth and polynomial size, but only when the learner is given by a Lipschitz ReLU network discriminator.
However, this does not rule out efficient algorithms which do not output such Lipschitz discriminators.
Finally, we remark that the family of hard distributions we construct can be thought of as a close cousin of the “parallel pancakes” construction of [DKS17].
This and slight modifications thereof are mixtures of Gaussians which are known to be computationally hard both in the SQ model [DKS17, BLPR19] and under cryptographic assumptions [BRST21, GVV22].
SQ lower bounds via ODEs.
We remark that in a very different context, [DKZ20] also used an ODE to design a moment-matching construction.
While our approach draws inspiration from theirs, an important difference is that they use their ODE as a “size reduction” trick to construct a step function f : R → {± 1 } with a small number of linear pieces such that E g ∼N (0 , 1) [ f ( g ) g k ] = 0 for all small k , while we use our ODE as a “weight reduction” trick to construct a continuous neural network f : R → R with bounded weights such that E g ∼N (0 , 1) [ f ( g ) k ] =
E g ∼N (0 , 1 [ g k ] .
The form of the moments E g ∼N (0 , 1) [ f ( g ) g
k ] they consider is simpler than in our setting, and while they essentially run their ODE to singularity and use non-quantitative facts like the invertibility of a certain Jacobian, we only run our ODE for a finite horizon and need to carefully control the condition number of the
Jacobian arising in our setting over this horizon (e.g. Lemma 4.8).
Technical Preliminaries
We freely abuse notation and use the same symbols to denote probability distributions, their laws, and their density functions.
Given a distribution A over a domain Ω and a function f : Ω → Ω ′ , we let f ( A ) denote the pushforward of A through f , that is, the distribution A ′ of the random variable f ( z ) for z ∼ A .
Given distributions p, q for which p is absolutely continuous with respect to q , let χ 2 ( p, q ) denote their chi-squared divergence.
Let p ⋆ q denote the convolution of p and q .
Also, we use ∥·∥ p to denote ℓ p norm, omitting the subscript when p = 2 .
σ min ( · ) denotes minimum singular value.
Given an invertible matrix M , we use M −⊤ to denote the matrix ( M ⊤ ) − 1 =
( M − 1 ) ⊤ .
Henceforth E g [ · ] will always denote E g ∼N (0 , Id ) [ · ] , where the dimensionality is implicit from context.
Let γ σ 2 ( x ) ≜
[Equation]
Given S ⊂ R , we use γ σ 2 ( S ) to denote (cid:82)
[Equation]
When σ = 1 , we omit the subscript σ 2 .
We will also use γ ( d ) ( x ) to denote the density of N (0 , Id d ) .
Definition 1 (One-hidden-layer ReLU networks) .
We say that g : R d → R is a one-hidden-layer
ReLU network with size S and W -bounded weights if there exist w 1 , . .
w S ∈ R d , b 1 , . .
, b S ∈ R , and s 1 , . . .
, s S ∈ {± 1 } for which g ( x ) =
(cid:80) Si
=1 s i ReLU( ⟨ w i , x ⟩ + b i ) and ∥ w i ∥ , | b i | ≤ W for all i ∈ S .
2 Here ReLU( z ) ≜ max(0 , z ) .
Given f : R d → R d ′ , together with a distribution A over R d , we say that f ( A ) is a one-hidden-layer ReLU network pushforward of A with size S and W -bounded weights if each coordinate of f is of this form.
Fact 2.1 ([GMSR20]) .
If V ∈ R n × n is a Vandermonde matrix with nodes z 1 , . .
, z n , that is,
V i,j = z i − 1 j , and { z i } are ζ -separated, then σ min ( V ) ≥ 1 n · Ω( ζ ) n − 1 .
Theorem 2.2 (Peano’s existence theorem, see e.g. Theorem 2.1 from [Har02]) .
For T, r > 0 and y 0 ∈ R n , let B ⊂ R × R n be the parallelepiped consisting of ( t, y ) for which 0 ≤ t ≤ T and ∥ y − y
∥ ∞ ≤ r .
If f : B → R is continuous and satisfies | f ( t, y ) | ≤ M for all ( t, y ) ∈ B , then the initial value problem { y ′ ( t ) =
f ( t, y ) and y (0) =
y 0 } has a solution over t ∈ [0 , min( T, r/M )] .
Definition 2 (Statistical queries) .
Given distribution D over R d and parameters τ, t > 0 , a STAT( τ ) oracle takes in any query of the form f : R d → [ − 1 , 1] and outputs a value from [ E x ∼ D [ f ( x )] − τ, E x ∼ D [ f ( x )]
+ τ ] , while a VSTAT( t ) oracle takes in any query of the form f : R d → [0 , 1] and outputs a value from [ E x ∼ D [ f ( x )] − τ, E x ∼ D
[ f ( x )]
+ τ ] for τ = max(1 /t, (cid:112) V x ∼ D
[ f ( x )] /t ) .
Statistical Query Lower Bound
Our main result is the following lower bound in this model for learning ReLU network pushforwards:
Theorem 3.1.
Let d ∈ N be sufficiently large.
Any SQ algorithm which, given SQ access to an arbitrary one-hidden-layer ReLU network pushforward of N (0 , Id d ) of size O (log d/ log log d ) with poly( d ) -bounded weights, outputs a distribution which is O (1) -close in d TV ( · ) must make at least d Ω(log d/ loglog d ) queries to either STAT( τ ) or VSTAT(1 /τ 2 ) for τ
=
d − Ω(log d/ loglog d ) .
Our proof will invoke the following key technical result whose proof we defer to Section 4.
It exhibits a one-hidden-layer ReLU network f : R 2 → R with bounded weights under which the pushforward of N (0 , 1) matches the low-degree moments of N (0 , 1) to arbitrary precision, in addition to some other technical conditions that we need to formally establish our statistical query lower bound:
Theorem 3.2.
Fix any odd m and ν, σ < 1 .
There is a one-hidden-layer ReLU network f ∗ : R 2 → R of size O ( m ) with weights at most m O ( m ) for which the pushforward D ≜ f ∗ ( N (0 , Id )) satisfies
| E x ∼ D [ x k ] − E g ∼N (0 , 1) [ g k ] | < ν for all k = 1 , . . .
, m 2.
χ 2 ( D, N (0 , 1)) ≤ exp( O ( m )) /σ 3.
d TV ( P Dv , P Dv ′ ) ≥ 1 − 2 σ log(1 /σ )
− m − Ω( m ) for any v, v ′ ∈ S
d − 1 satisfying |⟨ v, v ′ ⟩| ≥ 1 / 2 .
The rest of the proof of our lower bound will then follow the framework introduced in [DKS17] and subsequently generalized in [DK20].
First, given a distribution D over R and v ∈ S d − 1 , let P Dv denote the distribution over R d with density P Dv ( x ) =
D ( ⟨ v, x ⟩ ) · γ ( d − 1) ( x − ⟨ v, x ⟩ v ) , that is the distribution which is given by D in the direction v and is given by N (0 , Id − vv ⊤ ) orthogonal to v .
We use the following generic statistical query lower bound about learning such distributions, a proof of which we include in the supplement for completeness.
Lemma 3.3.
Let m ∈ N and 0 < C < 1 / 2 .
Let D be a distribution such that 1) χ 2 ( D, N (0 , 1)) is finite, and 2)
| E x ∼ D [ x k ] − E g ∼N (0 , 1) [ g k ] | ≤ Ω( d ) − ( m +1)(1 / 4 − C/ 2) (cid:112) χ 2 ( D, N (0 , 1)) for all k = 1 , . . .
, m .
Consider the set of distributions { P Dv } v ∈ S d − 1 for d ≥ m Ω(1 /C ) .
If there is some ϵ > 0 for which d TV ( P Dv , P Dv ′ ) > 2 ϵ whenever |⟨ v, v ′ ⟩| ≤ 1 / 2 , then any SQ algorithm which, given SQ access to
P Dv for an unknown v ∈ S d − 1 , outputs a hypothesis Q with d TV ( Q, P Dv ) ≤
ϵ needs at least d m + queries to STAT( τ ) or to VSTAT(1 /τ 2 ) for τ ≜ O ( d ) − ( m +1)(1 / 4 − C/ 2) · (cid:112) χ 2 ( D, N (0 , 1)) .
Proof of Theorem 3.1.
By Theorem 3.2 applied with sufficiently large odd m and sufficiently small σ , there exists a distribution D = f ∗ ( N (0 , Id 2 )) over R for f ∗ : R 2 → R of size O ( m ) with m O ( m ) - bounded weights satisfying the hypotheses of Lemma 3.3 for ϵ = 0 . 49 , and χ 2 ( D, N (0 , 1))
≤ exp( O ( m ))
(note that we can absorb the 1 /σ factor in Theorem 3.2 into exp( O ( m )) because we can take σ = exp( − Θ( m )) ).
As long as m ≤ d O ( C ) , we conclude that an SQ algorithm for learning any distribution from { P Dv } v ∈ S
d − 1 to total variation distance 1 / 4 must make at least d m + queries to STAT( τ ) or VSTAT(1 /τ 2 ) for τ ≜ O ( d ) − ( m +1)(1 / 4 − C/ 2) · exp( O ( m )) .
By taking m = Θ(log d/ log log d ) , we ensure that m O ( m ) ≤ poly( d ) .
We’re done by taking C in Lemma 3.
to be C = 1 / 4 .
The proof of the theorem is complete upon noting that any distribution P Dv can be implemented as a pushforward of N (0 , Id d +1 ) under a one-hidden-layer ReLU network F v : R d +1 → R d of size O (log d/ log log d ) with poly( d ) -bounded weights.
Let U ∈
O
( d ) be a rotation mapping the first standard basis vector in R d to v .
Then for F v ( z 1 , . .
. , z d +1 ) ≜
U ( f ∗ ( z 1 , z 2 ) , z 3 , . . . ,
z d +1 )
we have that F v ( N (0 , Id d +1 ))
=
P D v as desired.
Furthermore, note that every output coordinate of
F v ( z 1 , . .
. , z d +1 ) is a one-hidden-layer ReLU network of the form αf ∗ ( z 1 , z 2 )+ ⟨ u, ( z 3 , . .
. , z d +1 ) ⟩ for some vector ( α, u ) ∈ R d .
Note that the size of this network is two plus that of f ∗ , and its weights are also upper bounded by poly( d ) , so F v ’s output coordinates are of size O (log d/ log log d ) .
Remark 3.4 .
Theorem 1.1 was stated with output dimension polynomially bigger than input dimension, whereas in our construction, the output dimension ( d ) is less than the input dimension ( d + 1 ).
One can get the former by a padding argument (i.e. by duplicating output coordinates) to give a generator with arbitrarily large polynomial stretch and such that the d log d/ loglog d lower bound still applies.
Moment-Matching Construction
In this section we prove Theorem 3.2, the main technical ingredient in the proof of Theorem 3.1.
4.1 Moment-Matching With Unbounded Weights
We begin by making the simple observation that for one-hidden-layer networks with unbounded weights , it is easy to construct networks such that the pushforward of N (0 , 1) under these networks matches the moments of N (0 , 1) .
The starting point for this is the following standard construction.
Roughly speaking, it gives a set of well-separated points on the real line and a distribution over them which matches low-order moments with N (0 , 1) .
Lemma 4.1 (Lemma 4.3 from [DKS17]) .
For any m ∈ N , there exist weights λ 1 , . . .
, λ m ≥ 0 and points h 1 , . . .
, h m ∈ R for which
(Moments match) (cid:80) mi =
λ
i h
ki =
E g
[ g k ] for all k = 0 , . . .
, 2 m − 1 (note that the special case of k = 0 implies (cid:80) mi =
λ i = 0 ).
(Points symmetric about origin) h 1 ≤ · · · ≤ h m
and h i = − h m −
i +1 for all 1 ≤ i ≤ m .
(Weights symmetric) λ 1 ≤ · · · ≤ λ ⌈ m/ 2 ⌉ and λ i
= λ m − i +1 .
(Points bounded and separated) Ω(1 / √ m ) ≤ | h i | ≤ O ( √ m ) for all 1 ≤ i ≤ m and { h i } are Ω(1 / √ m ) -separated.
(Weights not too small) min
i
λ
i ≥ e − cm for an absolute constant c > 0 .
(Central point and weight)
If m is odd, then h ( m +1) / 2 = 0 and λ ( m +1) / 2 = Θ(1 / √ m ) .
This immediately implies that there exists a discontinuous piecewise linear function f : R → R for which the pushforward f
( N (0 , 1)) matches the low-degree moments of N (0 , 1) .
The reason is that we can take f to be a step function which takes on values given by the positions of the points h 1 , . . .
, h m in Lemma 4.1.
We can then take the steps to have lengths such that the probability that a standard Gaussian input to f falls under the step given by some h i is precisely λ
i (see Figure 1).
Corollary 4.2.
For any m ∈ N , there is a partition of R into disjoint intervals I 1 , . .
, I m , along with a choice of scalars h 1 , . .
, h m , such that the step function f
: R → R given by f ( z ) = (cid:80) mi =1 h i · 1 [ z ∈ I i ] satisfies E x ∼ f ( N (0 , 1))
[ x k ] =
E g ∼N (0 , 1)
[ g k ] for all k = 0 , . . .
, 2 m − 1 .
Proof.
Let { λ i } , { h i } be as in Lemma 4.1.
As (cid:80) i λ i = 1 , there are intervals R
=
I 1 ⊔ · · · ⊔ I m for which γ ( I i ) = λ i .
As E x ∼ f ( N (0 , 1))
[ x k ] = (cid:80) i λ i h ki , the claim follows by Part 1 of Lemma 4.1.
By infinitesimally perturbing f in Corollary 4.2, we can ensure f ( N (0 , 1)) still approximately matches the low-degree moments of N (0 , 1) to arbitrary precision and that the linear pieces of f have finite slopes, though some slopes will be arbitrarily large.
The new function can thus be represented as a one-hidden-layer ReLU network, but unfortunately its weights will be arbitrarily large.
The key challenge in the sequel is to design a better perturbation so the resulting f has polynomially bounded slopes yet is such that f ( N (0 , 1)) matches the low-degree moments of N (0 , 1) .
4.2 Bump Construction
Before we describe our perturbation scheme, we slightly modify the construction in Corollary 4.2.
In place of a step function, we will consider a certain sum of bump functions , illustrated in Figure 2.
Definition 3 (Bump functions) .
Given w, ϵ > 0 and h, c ∈ R , define T w,h,ϵ c : R → R by
[Equation]
otherwise .
Because these are piecewise linear functions, we can implement them as one-hidden-layer ReLU networks:
Fact 4.3.
Given w, ϵ > 0 and h, c ∈ R , T w,h,ϵc can be implemented as a one-hidden-layer ReLU network with size 4 and W -bounded weights for W ≤ hϵ max(1 , | c | + ϵ + w ) .
In the next lemma, we show that we can replace the construction in Corollary 4.2, given by a step function, with a sum of well-separated bumps with infinitesimally small ϵ parameter.
This will be the instance on which we will apply our perturbation scheme.
Lemma 4.4.
For any odd m and ν < 1 , there are centers c 1 ≤ · · · ≤ c m − 1 , widths w 1 , . . .
, w m − 1 > 0 , heights h 1 ≤ · · · ≤ h m − 1 ∈ R , and parameter ϵ > 0 for which the following holds.
Define the function f : R → R (see Figure 1) by f ( z ) ≜ (cid:80) m − 1 i =1 T w i ,h
i ,ϵ c i ( z ) for any 0 ≤ ϵ < ϵ .
Then f satisfies
Proof sketch, see supplement.
The idea is that when m is odd one of the points h
i from Lemma 4. is zero and does not contribute to the moments.
We want to design f for which f ( N (0 , 1)) is the mixture of point masses given by removing this point and its corresponding weight.
We take f to have one linear piece for every one of the remaining point masses.
By Part 6 of Lemma 4.1, these points have total weight 1 − Θ(1 / √ m ) , so by anticoncentration we can arrange these linear pieces to be ∼ m − 3 / 2 -separated, satisfying Part 1.
The remaining parts follow easily from Lemma 4.1.
Unfortunately the slopes in the function constructed in Lemma 4.4 are arbitrarily large if ν is arbitrarily small.
The issue still remains of how to get a continuous piecewise-linear function whose slopes are polynomially bounded so that the corresponding ReLU network has polynomially bounded weights.
As we illustrate next however, the Ω( m − 3 / 2 ) spacing between the bumps in the definition of f in Lemma 4.4 gives us sufficient “room” to carefully perturb f to achieve this goal.
4.2.1 Some Estimates for Bump Moments
For convenience, define M w,h,ϵc,k ≜ E g (cid:2) T w,h,ϵc ( g ) k (cid:3) .
We conclude this subsection by collecting some useful bounds for this quantity.
As we verify in the supplement, M w,h,ϵc,k is continuously differentiable with respect to ϵ when ϵ > 0 .
Additionally, we have the following (see supplement):
Lemma
[Equation]
Lemma 4.6.
For any w ≥ 0 , c, h, h ′ ∈ R , ϵ ′ ≥ ϵ ≥ 0 , and even k ∈ N , we have the bound | M w,h ′ ,ϵ ′ c,k − M w,h,ϵc,k
[Equation] [Equation]
−
[Equation]
|
[Equation]
ϵ
[Equation]
ϵ
[Equation]
In particular, this implies that (cid:12) (cid:12)(cid:12)
[Equation]
≤ h k .
4.3 ODE-Driven Perturbation
Denote the parameters of the function constructed in Lemma 4.4 by { ( h i (0) , w i , c i ) } 1 ≤ i<m .
We also define ϵ (0) to be some arbitrarily small positive quantity satisfying ϵ (0) ≤ ϵ for ϵ from Lemma 4.4.
We will design an ODE whose solution specifies a one-parameter family of functions
[Equation]
(1) that arise from gradually perturbing the function from Lemma 4.4.
Roughly speaking, starting at h i (0) and ϵ (0) for all 1 ≤ i < m , perturbing the function along this one-parameter family will correspond to keeping the widths w i and centers
c i of the bumps fixed, increasing the ϵ parameter of every bump at unit speed, and evolving the heights h i ( t ) in such a way that the moments of the pushforward of N (0 , 1) under f t remain constant in t for all 0 ≤ t ≤ T .
We illustrate this evolution in Figure 2.
Here T is some horizon which is at least inverse-polynomially large but smaller than m − 3 / 2 so that the “edges” c i ± ( ϵ ( t ) + w i ) of the bumps don’t collide with each other (this is where we make crucial use of Part 1 of Lemma 4.4).
At the end of this horizon, we want to show that the heights will not have changed too much, whereas the bumps now have ϵ parameter given by inverse-polynomially large T .
This will imply that f T has polynomially bounded slopes as desired.
As the odd moments of f 0 ( N (0 , 1)) vanish and the parameters { ( h i (0) , w i , c i ) } 1 ≤ i<m satisfy the symmetry properties from Part 3 of Lemma 4.4, it is easy to ensure that the odd moments of f t ( N (0 , 1)) also vanish: simply take h i ( t )
=
− h m − i ( t ) for all 1 ≤ i ≤ m − 1 .
We thus focus on evolving ( h 1 ( t ) , . .
, h ( m − 1) / 2 ( t )) .
For convenience denote this by h ( t ) .
Define the moment vector µ : R ( m − 1) / 2 × R → R m − 1 by
[Equation]
For any fixed ( h , ϵ ) in a small neighborhood of ( h (0) , ϵ (0)) , we want to show there is a direction v ∈ R ( m − 1) / 2 such that the directional derivative of µ in the direction w ≜ ( v 1 , . .
, v ( m − 1) / 2 , 1) is zero.
The constraint that the directional derivative ∇ w µ ( h , ϵ ) vanishes specifies a linear system in v :
[Equation]
Recalling Lemma 4.5, we can rewrite this as
[Equation]
∂ϵ
[Equation]
To express this more compactly, define b ( ϵ ) ∈ R ( m − 1) / 2 and Z ( h , ϵ ) by b
[Equation]
(cid:88) i = ∂M
w i ,h i ,ϵ c
i , 2 ℓ ∂ϵ and
Z ( h , ϵ ) i,ℓ
≜ M w i ,h i ,ϵ i c i , 2 ℓ .
Also define the matrices A ( h ) ≜ diag(1 /h 1 , . .
, 1 /h ( m − 1) / 2 ) and B ≜ diag(2 , 4 , . . .
, m − 1) .
Then (2) is equivalent to v ⊤ · A ( h ) Z ( h , ϵ ) B = b ( ϵ ) ⊤ .
Provided A ( h ) Z ( h , ϵ ) B is invertible, the natural choice for v would thus be v =
B − 1 Z ( h , ϵ ) −⊤
A ( h ) − 1 · b ( ϵ ) .
Therefore, defining w ( t, h ) ≜ (cid:0) B − 1 Z ( h , ϵ (0) + t ) −⊤
A ( h ) − 1 · b ( ϵ (0) + t ) , 1 (cid:1) , (4) we consider the following initial value problem
[Equation]
and
[Equation] [Equation]
Note that if we had a solution h ( t ) to (5) for t ∈ [0 , T ] for some horizon T , then we would have
[Equation]
implying that the low-degree moments of f t defined in (1) are constant in t as desired.
4.4 Existence and Boundedness of h ( t )
To carry out the strategy in Section 4.3, we must establish that (1) a solution to the initial value problem (5) exists over a non-negligible horizon T , and (2) the entries of h ( t ) do not explode in t .
For both of these, we need to show that the matrix Z ( h , ϵ ) is invertible or, more specifically, wellconditioned for ( h , ϵ ) in a neighborhood of ( h (0) , ϵ (0)) .
We first establish this at time t = 0 by relating Z ( h (0) , ϵ (0)) to a certain Vandermonde matrix and appealing to Fact 2.1, see supplement:
Lemma 4.7.
σ min ( Z ( h (0) , ϵ (0))) ≥ m − Cm for an absolute constant C > 0 .
Thus, for ( h , ϵ ) in a neighborhood of ( h (0) , ϵ (0)) , Z ( h , ϵ ) is well-conditioned (see supplement).
Lemma 4.8.
Let C > 0 be from Lemma 4.7.
For any ( h , ϵ ) satisfying ∥
h − h (0) ∥ ∞
≤ m − C ′ m and 0 ≤ ϵ − ϵ (0) ≤ m − C ′ m for sufficiently large absolute constant C ′ > 0 , σ min ( Z ( h , ϵ )) ≥ m − Cm / 2 .
To establish Property (1), we must first verify that w ( t, h ) is continuous (see supplement for proof):
Lemma 4.9.
Let C ′ > 0 be the absolute constant from Lemma 4.8.
Then the function w ( t, h ) defined in (4) is continuous with respect to both t and h for t ≤
m − C ′ m and ∥ h − h (0) ∥ ∞ ≤ m − C ′ m .
Lastly, we must show that under the hypotheses of Lemma 4.9, ∥ w ( t, h ) ∥ ∞ is not too large.
This implies Property (1) by Theorem 2.2 and Property (2) because ∥ h ′ ( t ) ∥ ∞ =
∥ w ( t, h ( t ))
∥ ∞ :
Lemma 4.10.
Let C ′ > 0 be the absolute constant from Lemma 4.8.
If t ≤ m − C ′ m and ∥ h − h (0) ∥ ∞ ≤ m − C ′ m , then ∥ w ( t, h ) ∥ ∞ ≤ m C ′′ m for some absolute constant C ′′ > 0 .
Proof.
By the second part of Lemma 4.6, every entry of b ( ϵ (0)+ t ) is at most m − 1 2 · (max i h i ) m − 1 ≤ m − 1 2 · ( O ( √ m )+ m − C ′ m ) m − 1 ≤ m O ( m ) , where in the penultimate step we used Part 4 of Lemma 4.
and our hypothesis on h .
Note that σ min ( Z ( h , ϵ (0) + t )) ≥ m − Cm / 2 by Lemma 4.8, σ min ( B ) ≥ 2 , and σ min ( A ( h )) ≥ min i 1 /h i ≥ Ω( √ m ) by Part 4 of Lemma 4.4 and our hypothesis on h .
We conclude that B − 1 Z ( h , ϵ (0) + t ) −⊤
A ( h ) − 1 · b ( ϵ (0) + t ) has L ∞ norm at most m C ′′ m for some absolute constant C ′′ > 0 , so ∥ w ( t, h ) ∥ ∞ ≤ m C ′′ m as claimed.
We are now ready to put these ingredients together to prove the key lemma for showing Theorem 3.2:
Lemma 4.11.
Fix any odd m ∈ N and any 0 < ν < 1 .
There is a one-hidden-layer ReLU network f : R → R of size O ( m ) with weights at most m O ( m ) for which the pushforward D = f
( N (0 , 1)) satisfies | E x ∼ D
[ x k ] − E g ∼N (0 , 1) [ g k ] | < ν for all k = 1 , . . .
, m .
Proof.
Define the parallelepiped B of pairs ( t, h ) for which 0 ≤ t ≤ m − C ′ m and ∥ h − h (0) ∥ ∞ ≤ m − C ′ m .
By Lemma 4.9, w is continuous over B .
By Lemma 4.10, ∥ w ( t, h ) ∥ ∞ ≤ m C ′′ m .
By Theorem 2.2, the initial value problem in (5) has a solution h ( t ) over t ∈ [0 , T ] for T = m − ( C ′ + C ′′ ) m .
Furthermore, because ∥ h ′ ( t ) ∥ ∞
=
∥ w ( t, h ( t ))
∥ ∞ ≤ m C ′′ m , we conclude that 1 T ∥
h ( T ) ∥ ∞ ≤ m C ′′ m .
The slopes of the bumps T w i ,h i ( T ) ,ϵ (0)+ T c i are therefore bounded by m C ′′ m .
For ( m − 1) / 2 < i ≤ m − 1 , define h
i ( T ) = − h
m
−
i
( T ) and consider the one-parameter family of functions f t ≜ (cid:80) m − 1 i =1 T w i ,h
i ( t ) ,ϵ (0)+ t c i .
Because c i =
c m − i and w i = w m − i for all 1 ≤ i < m by Part 3 of Lemma 4.4, we conclude that the odd moments of f T ( N (0 , 1)) all vanish.
As for the even moments, because ∂∂t µ ( h ( t ) , t ) = 0 by (6), we conclude that the even moments of f T ( N (0 , 1)) up to degree m − 1 agree with those of f 0 ( N (0 , 1)) .
So because D 0 = f 0 ( N (0 , 1)) satisfies | E x ∼ D
[ x k ] − E g ∼N (0 , 1) [ g k ] | < ν for 1 ≤ k ≤ m by Part 2 of Lemma 4.4, the same holds for f T .
As the endpoints of the intervals supporting the bumps are bounded by O (log m ) , Fact 4.3 implies f T is a one-hidden-layer network with size O ( m ) and m O ( m ) -bounded weights.
4.5 Proof of Theorem 3.
Having designed a pushforward given by a one-hidden-layer network with m O ( m ) -bounded weights satisfying the first part of Theorem 3.2, we now modify this to also satisfy the remaining two parts.
To do this, we convolve a suitable scaling of the pushforward by a thin Gaussian.
The following lemmas show that the moments continue to match, the chi-squared divergence between the result and
N (0 , 1) is bounded, and the third part of Theorem 3.2 is satisfied (see supplement for proof):
Lemma 4.12.
Let D be any symmetric distribution for which | E x ∼ D [ x k ] − E g [ g k ] | < ν for all 1 ≤ k < m .
For any c ∈ R let c · D denote the distribution obtained by rescaling D by a factor of c .
Then D ′ ≜ √ 1 − σ 2 · D ⋆ N (0 , σ 2 ) satisfies | E x ∼ D ′
[ x k ] − E g [ g k ] | <
ν for all 1 ≤ k < m .
Lemma 4.13.
For any density A on [ − R, R ] and σ ≤ 1 / 2 , χ 2 ( A ⋆ N (0 , σ 2 ) , N (0 , 1)) ≤ e O ( R 2 ) /σ .
Lemma 4.14.
Let D = f ( N (0 , 1)) be from Lemma 4.11, and define D ′ ≜ √ 1 − σ 2 · D ⋆ N (0 , σ 2 ) .
Then for any v, v ′ ∈ S
d − 1 satisfying |⟨ v, v ′ ⟩| ≤ 1 / 2 , d TV ( P D ′ v , P D ′ v ′ ) ≥ 1 − 2 σ log(1 /σ )
− m − Ω( m ) .
Proof of Theorem 3.2.
Let f be the function constructed in Lemma 4.11 and define f ∗ : R 2 → R by f ∗ ( z 1 , z 2 ) = √ 1 − σ 2 f ( z 1 ) + σz 2 .
Note that f ∗ ( N (0 , Id )) is exactly √ 1 − σ 2 f ( N (0 , 1))
⋆
N (0 , σ 2 ) , so the three parts of Theorem 3.2 follow immediately from Lemmas 4.12, 4.13, and 4.
respectively.
Conclusion.
In this work we established that learning one-hidden-layer ReLU network pushforwards of logarithmic size is hard for efficient SQ algorithms.
Given that SQ algorithms capture a large family of efficient algorithms, this suggests that this basic learning problem, despite its simple description, is computationally intractable.
Of course, our lower bound construction is worst case, and so it is unlikely to actually arise in practice.
A natural open question, as mentioned previously, is to come up with natural assumptions (for instance, smoothed analysis settings like in [CLLZ22]) that allow us to circumvent this lower bound, and to achieve efficient learning algorithms.
[-| ≤ |⟨ v, v ′ ⟩| m +
-]
[-(Sum of weights bounded away from 1) (cid:80) m − 1 i =
-]
[-(Moments match) | (cid:80) m − 1 i =
-]
[-The fact that (cid:80) m − 1 i =
-]