On the non-universality of deep learning: quantifying the cost of symmetry
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recent work [AS20] has shown that deep learning is a universal paradigm: it canemulate any efﬁcient learning algorithm.
However, this result relies on contrivedneural network architectures that do not reﬂect architectures used in practice. Inthis paper, we prove that if one trains “regular” architectures and initializations,where the training algorithm satisﬁes a certain symmetry (a.k.a. equivariance), thendeep learning is no longer universal. As our ﬁrst main contribution, we prove ageneral computational lower bound for G -equivariant gradient descent in termsof the “ G -orbit-alignment” of the data.
We apply this bound in conjunction withrepresentation theory tools to obtain two new results: (i) we characterize whichfunctions fully-connected networks can weakly-learn on the binary hypercube andon the unit sphere, which demonstrates that as far as weak-learning and fullyconnected networks are concerned, depth-2 is as powerful as any other depth;(ii) we take a further step towards characterizing which functions with latentlow-dimensional structure neural networks can learn by strengthening the mergedstaircase property necessity result of [ABM22] beyond the mean-ﬁeld regime, toany architecture such that gradient descent is permutation-equivariant.
As oursecond main contribution, we extend the ideas to stochastic gradient descent,proving hardness results for learning on fully-connected networks based on the Learning Parities with Noise cryptographic assumption.
Introduction
Over the last decade, deep learning has made impressive advances in a diverse range of areas,including image classiﬁcation [KSH12], classical board games [SHS + 18], language translation[BCB14], physics [RPK19] and, most recently, programming [LCC + 22].
Deep learning trained withsimple gradient-based optimizers has surpassed previous classical methods for these tasks, raisingthe question of how much further these advances can be pushed. Can we hope for deep learningmethods to eventually replace all other learning algorithms?
In other words, is deep learning auniversal learning paradigm?
Recently, [AS20] proved that in a certain sense the answer is yes:any PAC-learning algorithm
[Val84] can be efﬁciently implemented by a neural network trained bystochastic gradient descent; analogously, any Statistical Query algorithm
[Kea98] can be efﬁcientlyimplemented by a neural network trained by noisy gradient descent.
In short, for any learningalgorithm one can construct a deep learning architecture capable of efﬁciently replacing it.
However, there is a catch: the result of [AS20] relies on a very carefully choice of neural networkarchitecture, which is capable of emulating arbitrary learning algorithms using handcrafted memoryand computation modules.
This is far from the architectures which have been shown to be successfulin practice.
Indeed, although neural networks in practice do incorporate some domain knowledge(e.g., convolutional networks and transformers for image recognition and language processing [LB + 95, LKF10], graph neural networks for analyzing graph data [GMS05, BZSL13, VCC + 17],and equivariant neural networks [CW16] such as Lorentz-equivariant networks for particle physics[BAO + 20]), they have more “regularity” than the architectures of [AS20], in the sense that they donot rely on heterogeneous and carefully assigned initial weights.
We therefore reﬁne our question:
Is deep learning with “regular” architectures and initializations a universal learning paradigm? If not, can we quantify its limitations?
The above question is broad, and we would like an answer that is applicable to the incredible diversityof possible architectures.
In order to formalize the problem and develop a general theory, we take anapproach pioneered by [Ng04] and developed by [LZA21] of understanding deep learning on a certainarchitecture and initialization through the symmetry group G (otherwise known as equivariance group)of the learning algorithm.
Namely, an algorithm A , such as gradient descent training, that takes in adata distribution D 2 P ( X ⇥ R ) 1 and outputs a function A ( D ) :
X !
R is said to be G -equivariant 48 if for each g
[Equation]
[Equation] Here g acts on the data space X , and so is viewed as a function g : X !
X , and g ( D ) is thedistribution of ( g ( x ) , y ) , where ( x , y )
⇠ D .
The equivariance group depends on the particularoptimizer used, the architecture, and the initialization [Ng04, LZA21].
We note that the equivariancegroup of a training algorithm is distinct from, and should not be confused with, the equivariancegroup of an architecture in the context of geometric deep learning [BBCV21]. We think of a deep learning algorithm that has a large equivariance group as being “regular”,in opposition to the emulation architectures of [AS20], for which training has only the trivialequivariance.
Studying the equivariance group of a training algorithm is a useful perspective sinceit is nontrivial in most practical cases: for example, we have orthogonal equivariance for gradientdescent training of fully-connected networks with Gaussian initialization [Ng04], and permutationequivariance for gradient descent training of i.i.d. initialized ResNets and certain transformers.
Furthermore, [LZA21, Theorem C.1] provides general conditions under which a deep learningtraining algorithm is G -equivariant.
In this work, we prove computational limitations on the performance of deep learning when the G -equivariance group of noisy gradient descent (GD) or stochastic gradient descent (SGD) training isnot the trivial group.
Our results show that deep learning with regular architectures and initializationsis not universal , implying a separation with the carefully-crafted initializations and architecturesconsidered in [AS20].
Speciﬁcally, the ﬁrst contribution of this work is to develop a general theoryfor computational lower bounds for G -equivariant GD.
This allows us (a) to fully characterize whichfunctions can be weakly learned fully-connected networks on both the hypercube and the unit sphere,showing that, for fully-connected networks and weak learning, two-layer networks are as powerfulas networks of arbitrary depth; and (b) to take a step towards characterizing which functions withlatent low-dimensional structure can be learned, by extending the necessity result of [ABM22] forstaircases beyond the two-layer mean-ﬁeld regime.
The second main contribution of our paper is todemonstrate nontrivial computational hardness results for fully-connected with Gaussian initializationtrained by SGD, implying that these cannot learn universally, and providing a separation with [AS20].
An informal overview of these contributions is given in Sections 1.2 and 1.3, but ﬁrst we cover priorwork in more detail in Section 1.1.
1.1 Related work
The key challenge to proving limitations of deep learning is its nonlinear and nonconvex nature. Dueto this, one direction of prior work has focused on proving limitations of linear methods such as kernelmethods.
Dimension lower bound arguments originating with [Bar93] have recently been reﬁned andapplied to show failure of general kernel methods on certain problems [WLLM19, AZL19, KMS20, AZL20, Hsu, HSSVG21, ABM22], which implies failure of neural networks in the Neural Tangent
Kernel (NTK) regime
[JGH18], but does not apply to general nonlinear training.
Furthermore, forspeciﬁc architectures such as fully-connected architectures [GMMM21, Mis22] and convolutionalarchitectures [MM21], the kernels and random features models at random initialization are now wellunderstood, yielding bounds for training in the NTK regime with those architectures.
For nonlinear training, considerably less is known.
The idea of studying equivariance groups oftraining was introduced in [Ng04], which used it to show a task with a ⌦ ( d ) versus O (1) samplecomplexity separation for learning a certain distribution on d -dimensional space with an SGD-trained, Gaussian-initialized fully-connected architecture versus an arbitrary algorithm.
More recently,[LZA21] built on [Ng04] to show a O (1) versus ⌦ ( d 2 ) sample-complexity separation betweenconvolutional and fully-connected architectures that are initialized i.i.d.
Gaussian and trained by SGD.
In this paper, we also analyze the equivariance group of training, but with the different focusof proving computational lower bounds.
For nonlinear training, there are known lower bounds fornonlinear stochastic gradient descent (SGD) training of two-layer mean-ﬁeld networks [ABM22],but these only apply when SGD converges to the mean-ﬁeld limit, and so do not apply to generalarchitectures.
It is also known that networks trained with noisy 2 gradient descent (GD) fall under the Statistical Query (SQ) framework [Kea98], which allows to show lower bounds for GD training basedon SQ lower bounds.
This has been combined in [SSSS17, MSS20, ACHM22] with the permutationsymmetry of GD-training of i.i.d.
fully-connected networks to prove lower bounds for learninghigh-degree parities and polynomials.
Our work reveals that their arguments can be fruitfully viewedin the broader context of more general group symmetries.
We also note the sample-complexity beneﬁts of invariant and equivariant architectures have beenstudied for linear methods in [MMM21, EZ21, Ele21, BVB21], and recently for non-linear PAClearning by [Ele22].
However, we remind the reader that equivariant training algorithms andequivariant architectures are distinct, and the former is the focus of this work.
1.2 Contribution 1: Lower bounds for noisy gradient descent (GD
We consider the supervised learning setup where we train a neural network f NN ( · ; ✓ ) :
X !
R 109 parametrized by ✓ 2 R p to minimize the mean-squared error on a data distribution D 2 P ( X ⇥ R ) , [Equation] D [Equation]
[Equation]
✓ [Equation] x , y )] , where ` ( ✓
[Equation]
x
[Equation]
y ) [Equation] ( y  f NN ( x
[Equation]
✓ )
[Equation]
[Equation]
The noisy Gradient Descent (GD) training algorithm randomly initializes ✓ 0 ⇠ µ ✓ for some initialization distribution µ ✓ 2 P ( R p ) , and then iteratively updates the parameters with step size ⌘ > 0 inthe direction of the population loss gradient, plus Gaussian noise ⇠ k ⇠ N (0 , ⌧ 2 I ) :[-k z k 2  R } ⇢ R p , 
-]
[Equation]
Roughly speaking, GD training is considered efﬁcient if the network is Lipschitz, the step size isbounded, and the noise magnitude ⌧ is lower-bounded, since then it can be efﬁciently implementedwith noisy minibatch SGD.
Assuming that the GD training algorithm is G -equivariant as in (1) forsome nontrivial group G , then we show a general result that certain data distributions cannot beefﬁciently learned.
For our result, we introduce a quantity called the G -orbit-alignment:
Deﬁnition 1.1 ( G -orbit-alignment) .
Let G be a compact group, let µ X 2 P ( X ) be a distribution 119 over data points, and let f : X ! R be a labeling function.
The G -orbit-alignment of ( µ X , f ) is:
[Equation]
where µ G is the Haar measure of G and the supremum is over h 2 L 2 ( µ X )
such that k h k 2 = 1 .
In our applications, we use tools from representation theory (see e.g., [Kna96]) to evaluate the
G -orbit-alignment.
Using the G -orbit-alignment, we can prove a master theorem for lower bounds,stated here informally:
Theorem 1.2 (GD lower bound, informal statement of Theorem 3.1) .
Let D f 2 P
( X ⇥ R ) be the 125 distribution of ( x , f ( x )) for x ⇠ µ X .
If µ X is G -invariant, and the G -orbit-alignment of ( µ X , f ) is 126 small, then networks cannot efﬁciently learn f when trained by (GD) on data distribution D = D f if 127 the GD algorithm is G -equivariant.
The ﬁrst insight behind this theorem can be traced back at least to [Ng04]: if a G -equivariantalgorithm can learn the function f by training on the distribution D f , then for any group elementg 2 G , it can learn f  g by training on the distribution D f  g .
In other words, the algorithm can learn the class of functions F = { f  g : g 2 g } , which can potentially be much larger than just the single 132 function { f } .
Therefore, if we could prove that the class of functions F cannot be efﬁciently learned 133 by any network with GD, then this would show that f cannot be efﬁciently learned by any networkwhere GD is G -equivariant.
The second insight of the theorem is to show that if the G -orbit-alignment of ( µ X , f ) is small, then F cannot be efﬁciently learned by GD on any network.
Why does the G -orbit-alignment arise?
Theintuition is that the G -orbit-alignment measures the diversity of the functions in F : if the G -orbitalignment of f is large, then there must be some function h that has high correlation with most ofthe functions in F , and therefore the functions in F are not very diverse.
On the other hand, if the 140 G -orbit-alignment is small, then this must mean that there is no h that correlates with most of them,and so F is very diverse, which can be used to show that it is hard to learn by gradient descent.
Thistype of argument appears in [AS20, ACHM22] in the speciﬁc case of Boolean functions and forpermutation equivariance; our proof both applies to a more general setting (beyond Boolean functionsand permutations) and sharpens the bounds in these papers.
While the idea behind Theorem 1.2 is intuitively simple, the theorem is very broadly applicable, toarbitrary groups and beyond functions deﬁned on the Boolean hypercube.
We demonstrate its powerand ease-of-use by deriving two novel applications that are interesting in their own right, and thatshowcase the limitations of fully-connected networks with standard initializations.
We expect that Theorem 1.2 will ﬁnd further applications in future work.
Application: Characterization of weak-learnability by fully-connected networks In our ﬁrstapplication, we consider weak-learnability: i.e., when can a function be learned non-negligibly betterthan just outputting the estimate f NN ⌘ 0 ?
Using Theorem 1.2, we characterize which functionsover the binary hypercube f : { +1 ,  1 } d ! R and over the sphere f : S
d  1 !
R are efﬁciently 154 weakly-learnable by GD-trained fully-connected networks with i.i.d.
symmetric and i.i.d.
Gaussianinitialization, respectively.
The takeaway is that a function f : { +1 ,  1 } d !
R is weakly-learnableif and only if it has a nonnegligible Fourier coefﬁcient of order O (1) or d  O (1) .
Similarly, afunction f : S d  1 !
R is weakly-learnable if and only if it has nonnegligible projection onto thedegree- O (1) spherical harmonics.
Such functions can be weakly-learned by 2-layer fully-connectednetworks, which shows that adding more depth does not help for weak learning.
This application ispresented in Section 3.1.
Application: Evidence for the staircase property In our second application, we consider learninga target function f : { +1 ,  1 } d !
R that only depends on the ﬁrst P coordinates, f ( x ) = 163 h ( x 1 , . . .
, x P ) .
Our regime of interest here is when the function h : { +1 ,  1 } P !
R remains ﬁxedand the dimension d grows, since this models the situation where a latent low-dimensional spacedetermines the labels in a high-dimensional dataset.
Recently, [ABM22] studied SGD-training ofmean-ﬁeld two-layer networks, and gave a near-characterization of which functions can be learnedto arbitrary accuracy ✏ in O h, ✏ ( d ) samples, in terms of the merged-staircase property (MSP).
Using Theorem 1.2, we prove that the MSP is necessary for GD-learnability by any permutation-equivariantarchitecture, well beyond the 2-layer mean-ﬁeld regime, and also generalize it beyond leaps of size 1.
This application is presented in Section 3.2.
1.3 Contribution 2: Hardness for stochastic gradient descent (SGD)
Our second main contribution concerns Stochastic Gradient Descent (SGD) training, which randomlyinitializes the weights ✓ {+
+} ⇠ µ ✓ , and then iteratively trains the parameters with the following updaterule to try to minimize the loss (2):
[Equation]
where ( y k +1 , x k +1 )
⇠ D is a fresh sample on each iteration, and ⌘ > 0 is the learning rate.
Proving lower bounds for SGD is a notoriously difﬁcult problem [AKM + 21], exacerbated by the factthat for general architectures SGD can be used to simulate any polynomial-time learning algorithm
[AS20].
However, we demonstrate that one can prove hardness results for SGD training based off ofcryptographic assumptions when the training has large equivariance groups.
This demonstrates thenon-universality of SGD in certain practical architectures.
Theorem 1.3 (Hardness for SGD, informal statement of Theorem 4.4) .
Under the assumption that the
Learning Parities with Noise (LPN) problem 5 is hard, fully-connected neural networks with Gaussian initialization trained by SGD cannot learn f mod8 : { +1 ,  1 } d ! { 0 , . .
, 7 } , f mod8 ( x ) ⌘ d
X i =
x i (mod 8) , in polynomial time from noisy samples ( x , f mod8 ( x ) + ⇠ ) where x ⇠ { +1 ,  1 } d and ⇠ ⇠ N (0 , 1) .
This result is the ﬁrst of its kind, in the sense that it proves nontrivial limitations of SGD trainingbased on an average-case reduction from a cryptographic problem.
The architecture and initializationplay a crucial role: a network whose ﬁrst layer is initialized and ﬁxed to all 1 ’s, and whose secondlayer is trained, would be able to easily learn f mod8 , or indeed any other bounded function dependingonly on
P i x i .
On the other hand, SGD on fully-connected, Gaussian-initialized architectures issign-ﬂip equivariant, and we can use this to show that f mod8 is hard to learn.
Preliminaries
Notation The law of a random variable X is denoted by L ( X ) .
If S is a ﬁnite set, then X ⇠ S isshorthand for X ⇠ Unif[ S ] .
For any set ⌦ , we let P ( ⌦ ) denote the set of distributions on ⌦ . Let H d = { +1 ,  1 } d denote the binary hypercube, and let S d  1 = { x 2 R d : k x k 2 = 1 } denote the 192 unit sphere.
We use  to denote the elementwise product of vectors.
For any µ X 2 P ( X ) , and anygroup G acting on X , we say µ X is G -invariant if g ( x ) d = x for x ⇠ µ X .
2.1 Equivariance of GD and SGD
We deﬁne GD and SGD equivariance separately.
Deﬁnition 2.1.
Let A GD be the algorithm that takes in data distribution D 2 P
( X ⇥ R )
, runs (GD)on initialization ✓
⇠ µ ✓ for k steps, and outputs the function A GD ( D ) =
f NN ( · ; ✓ k )
We say “ ( f NN , µ ✓ ) -GD is G -equivariant” if A GD is G -equivariant in the sense of (1) .
Deﬁnition 2.2.
Let A SGD be the algorithm that takes in samples { ( x i , y i ) } i 2 [ n ] , runs (SGD) oninitialization ✓ 0 ⇠ µ ✓ for n steps, and outputs A SGD ( { ( x i , y i ) } i 2 [ n ] ) =
f NN ( · ; ✓ k ) .
We say “ ( f NN , µ ✓ ) -SGD is G -equivariant” if A SGD ( { ( g ( x i ) , y i ) } i 2 [ n ] ) d = A SGD ( { ( x i , y i ) } i 2 [ n ] )  gfor any g 2 G , and any samples { ( x i , y i ) } i 2 [ n ] .
2.2 Regularity conditions on networks imply equivariances of GD and SGD
We take the data space X ✓ R d , and consider the following groups that act on R d .
Deﬁnition 2.3.
Deﬁne the following groups and actions:
• Let G perm =
S d denote the group of permutations on [ d ] .
An element  2 G perm acts onx 2 R d in the standard way:  ( x ) =
( x  (1) , . .
, x  ( d ) ) .•
Let G sign,perm denote the group of signed permutations, an element g =
( s ,  ) G sign,perm is given by a sign-ﬂip vector s 2 { +1 ,  1 } d and a permutation  2 G perm .
Itacts on x 2 R d by g ( x ) =
s   ( x ) =
( s 1 x  (1) , . . .
, s d x  ( d ) ) .
Let G rot =
G rot ✓ GL ( d, R ) denote the rotation group.
An element g 2 G rot is a rotationmatrix that acts on x by matrix multiplication. The
[Equation] [Equation] [Equation] Under mild conditions on the neural network architecture and initialization, GD and SGD trainingare known to be G perm -, G sign,perm -, or G rot -equivariant
[Ng04, LZA21].
Assumption 2.4 (Fully-connected i.i.d.
ﬁrst layer and no skip connections from the input) .
We candecompose the parameters as ✓ =
( W , ) , where W 2 R m ⇥ d is the matrix of the ﬁrst-layer weights,and there is a function g NN ( · ; ˜ ✓ ) : R m !
R such that f NN ( x ; ✓ ) =
g NN ( W x ; ) .
Furthermore, theinitialization distribution is µ ✓ = µ W ⇥ µ , where µ W = µ m ⌦ d w for µ w 2 P ( R ) .
Notice that Assumption 2.4 is satisﬁed by fully-connected networks with i.i.d. initialization.
Underassumptions on µ w , we obtain equivariances of GD and SGD (see Appendix E for proofs.)
Proposition 2.5 ([Ng04, LZA21]) .
Under Assumption 2.4, GD and SGD are G perm -equivariant.
If µ w is sign-ﬂip symmetric, then GD and SGD are G sign,perm -equivariant.
If µ w = N (0 ,  2 ) forsome  , then GD and SGD are G rot -equivariant.
Lower bounds for learning with GD
In this section, let D ( f, µ X ) 2 P ( X ⇥ R ) denote the distribution of ( x , f ( x )) where x ⇠ µ X .
We give a master theorem for lower bounds for learning with G -equivariant GD, depending on the
G -orbit-alignment from Deﬁnition 1.1.
Theorem 3.1 (Lower bound for GD based on G -orbit-alignment) .
Let G be a compact group, andlet f NN ( · ; ✓ ) :
X !
R be an architecture and µ ✓ 2 P ( R p ) be an initialization such that GD is 230 G -equivariant
and the architecture has bounded Lipschitz constant L = sup ✓ , x kr ✓ f NN ( x ; ✓ ) k .
Fix any G -invariant distribution µ X 2 P ( X ) , any label function f 2 L 2 ( µ X ) , and any baselinefunction ↵ 2 L 2 ( µ X ) satisfying ↵  g = ↵ for all g 2 G .
Let ✓ k be the random weights afterk time-steps of GD training with noise parameter ⌧ > 0 and step size ⌘ > 0 on the distribution
D = D ( f ⇤ , µ X ) .
Then, for any ✏ > 0 ,
[Equation]
where C = C (( f ⇤  ↵ , µ X ); G ) is the G -orbit-alignment of Deﬁnition 1.1.
As discussed in Section 1.2, the theorem states that if the G -orbit-alignment is very small, then GDtraining cannot efﬁciently improve on the trivial loss.
Appendix A shows a generalization of thetheorem for learning a class of functions F = { f 1 , . . .
, f m } instead of just a single function f ⇤ .
This generalization provides strictly tighter bounds than [AS20], even in the case where G is thetrivial group with one element.
We demonstrate the usefulness of the theorem through two novelapplications in Sections 3.1 and 3.2.
3.1 Application: Characterizing weak-learnability by fully-connected networks
In our ﬁrst application of Theorem 3.1, we consider fully-connected architectures with i.i.d. initialization, and show how to use their training equivariances to characterize what functions theycan weakly-learn: i.e., for what target functions f
⇤ they can efﬁciently achieve a non-negligiblecorrelation after training.
Deﬁnition 3.2 (Weak learnability) .
Let { µ d } d 2 N be a family of distributions µ d 2 P ( X d ) , and let{ f d } d 2 N be a family of functions f d 2 L 2 ( µ d ) .
Finally, let { ˜ f d } d 2 N be a family of estimators, where˜ f d is a random function in L 2 ( µ d ) .
We say that { f d , µ d } d 2 N is “weakly-learned” by the family ofestimators { ˜ f
d } d 2 N if there are constants d 0 , C > 0 such that for all d > d 0 ,
[Equation]
In words, weak-learning measures whether the family of estimators { ˜ f d } has a non-negligible edgeover simply estimating with the identically zero functions ˜ f d ⌘ 0 .
We study weak-learnability by GD-trained fully-connected networks.
Deﬁnition 3.3.
We say that { f d , µ d } d 2 N is efﬁciently weakly-learnable by GD-trained fully-connected 255 networks if there are fully-connected networks and initializations { f NN ,d , µ ✓ ,d } , and hyperparameters{ ⌘ d , k d , ⌧ d } such that for some constant c > 0 ,
• Hyperparameters are polynomial size: 0  ⌘ d , k d , 1 / ⌧ d , L d  O ( d c ) ;• Networks are polynomially Lipschitz: L d = sup x kr ✓ f NN ,d ( x ; ✓ ) k  O ( d c ) ;
• { ˜ f d } weakly-learns { f d , µ d } in the sense of Deﬁnition 3.2, where ˜ f
d = f NN ( · ; ✓ d ) forweights ✓ d that are GD-trained on D ( f d , µ d ) for k d steps with step size ⌘ d and noise ⌧ d ,starting from initialization µ ✓ ,d .
If µ ✓ ,d is i.i.d copies of a symmetric distribution, we say that the fully-connected networks aresymmetrically-initialized, and Gaussian-initialized if µ ✓ ,d is i.i.d.
copies of a Gaussian distribution.
3.1.1 Functions on hypercube, fully-connected networks with i.i
d. symmetric initialization
Let us ﬁrst consider functions on the Boolean hypercube f : H d ! R .
An important fact is that thesecan be uniquely written as a multilinear polynomial f ( x ) =
P
[Equation] ( S [Equation]
Q i 2 S x i , where ˆ f ( S ) arethe Fourier coefﬁcients of f .
We characterize weak learnability of functions on the hypercube interms of their Fourier coefﬁcients.
The proof is deferred to Appendix B.1.
Theorem 3.4.
Let { f d } d 2 N be a family of functions
f d : H d ! R with k f d k L 2 ( H d )  1 .
Then{ f d , H d } is efﬁciently weakly-learnable by GD-trained symmetrically-initialized fully-connected 271 networks if and only if there is a constant C > 0 such that for each d 2 N there is S d ✓ [ d ] with| S d |  C or | S d |  d  C , and | ˆ f d ( S d ) |  ⌦ ( d  C ) .
The algorithmic result can be achieved by two-layer fully-connected networks, and relies on randomfeatures analysis where each network weight is initialized to 0 with probability 1  p , and +1 or 1 with equal probability p/ 2 .[-
-]
Therefore, for weak learning, two-layer networks are as good asnetworks of any depth.
For the converse impossibility result, we apply Theorem 3.1, recalling that
GD is G sign,perm -equivariant by Proposition 2.5, and noting that G sign,perm -orbit-alignment is:
Lemma 3.5.
Let f : H d ! R .
Then C (( f, H d ); G sign,perm ) = max k 2 [ d ]  dk
P
S ✓ [ d ] | S | =
k ˆ f ( S ) 2 .
This lemma can be proved via direct manipulation or using the representation theory of G sign,perm .
It implies that if the Fourier coefﬁcients of f are negligible for all S s.t. min( | S | , d  | S | )  O (1) , 281 then the G sign,perm -orbit-alignment of f is negligible and f cannot be learned efﬁciently.
3.1.2 Functions on sphere, fully-connected networks with i.i.d. Gaussian initialization
We now study learning a target function on the unit sphere, f 2 L 2 ( S d  1 ) , where we take the standard
Lebesgue measure on S d  1 .
A key fact in harmonic analysis is that L 2 ( S d  1 ) can be written as thedirect sum of subspaces spanned by spherical harmonics of each degree: L 2 ( S d  1 )
=
1 l =
V d,l ,where V d,l ✓ L 2 ( S d  1 ) is the space of degree- ` spherical harmonics (see, e.g., [Hoc12]).
Let
P V d,l : L 2 ( S d  1 ) !
V d,l be the projection operator to the space of degree- l spherical harmonics,which has dimension dim( V d,l ) =
2 l + d  2 l  l + d  3 l  .
In Appendix B.2, we prove the followingcharacterization of when functions on the sphere are weakly-learnable.
Theorem 3.6.
Let { f d } d 2 N be a family of functions
f d : S d  1 !
R with k f d k L 2 ( S d  1 )  1 .
Then { f d , S d  1 } is efﬁciently weakly-learnable by GD-trained Gaussian-initialized fully-connectednetworks if and only if there is a constant C > 0 such that
[Equation]
The algorithmic result can be achieved by two-layer fully-connected networks, and is a consequence ofthe analysis of the random feature kernel in [GMMM21], which shows that the projection of f d ontothe low-degree spherical harmonics can be efﬁciently learned.
For the impossibility result, we apply Theorem 3.1, noting that GD is G rot -equivariant by Proposition 2.5, and the G rot -orbit-alignment is:
Lemma 3.7.
Let f 2 L 2 ( S d  1 ) .
Then C (( S d  1 , f ); G rot )
= max l 2 Z  0 k P V d,l f k 2 / dim( V d,l ) .
The G rot -orbit-alignment is computed using the representation theory of G rot .
In particular, weuse the Schur orthogonality theorem, and the fact that for any degree l 2 Z  0 the representation 300  l : V d,l !
V d,l of G rot given by  l ( g ) f
= f  g is irreducible (see, e.g., [Kna96]). Q
3.2 Application: Extending the merged-staircase property necessity result
In our second application, we study the setting of learning a sparse function on the binary hypercube (a.k.a.
a junta) that depends on only P  d coordinates of the input x , i.e.,
[Equation]
where h ⇤ : H P !
R .
The regime of interest to us is when h ⇤ is ﬁxed and d ! 1 , representing a 303 hidden signal in a high-dimensional dataset.
This setting was studied by [ABM22], who identiﬁedthe “merged-staircase property”, which we generalize below.
Deﬁnition 3.8 ( l -MSP) .
For l 2 Z + and h ⇤ : H P !
R , we say that h ⇤ satisﬁes the merged staircaseproperty with leap l (i.e., l -MSP) if its set of nonzero Fourier coefﬁcients S = { S :
ˆ h
⇤ ( S ) 6 = ; } can 307 be ordered as
S =
{ S 1 , . . .
, S m } such that for all
i 2 [ m ] , | S i \ [ j<i S
j |  l .
For example, h ⇤ ( x ) =
x 1 + x 1 x 2 + x 1 x 2 x 3 satisﬁes 1-MSP; h ⇤ ( x ) = x 1 x 2 + x 1 x 2 x 3 satisﬁes2-MSP, but not 1-MSP because of the leap required to learn x 1 x 2 ; similarly h ⇤ ( x ) = x 1 x 2 x 3 + x 4satisﬁes 3-MSP but not 2-MSP.
If h ⇤ satisﬁes l -MSP for some small l , then the function f
⇤ can belearned greedily in an efﬁcient manner, by iteratively discovering the coordinates on which it depends.
In [ABM22] it was proved that the 1-MSP property nearly characterized which sparse functions couldbe ✏ -learned in O ✏ ,h ⇤ ( d ) samples by one-pass SGD training in the mean-ﬁeld regime.
In this work, we prove the MSP necessity result for GD training.
On the one hand, our necessityresult is for a different training algorithm, GD, which injects noise during training.
On the other hand,our result is much more general in the sense that it applies to any Lipschitz, permutation-equivariantarchitecture (whereas the necessity result of [ABM22] applies only to two-layer architectures in themean-ﬁeld regime).
We also generalize the result to any leap l .
Theorem 3.9 ( l -MSP necessity) .
Let f NN ( · ; ✓ ) : H d !
R be an architecture and µ ✓ 2 P ( R p ) be an 320 initialization such that GD is G perm -equivariant.
Let L = sup ✓ , x kr ✓ f NN ( x ; ✓ )
k be the Lipschitzconstant, and let ✓ k be the random weights after k steps of GD training with noise parameter⌧ > 0 and step size ⌘ on the distribution D
= D ( f ⇤ , H d ) .
Suppose that f ⇤ ( x ) =
h ⇤ ( z ) where 323 h ⇤ : H P !
R does not satisfy l -MSP for some l 2 Z + .
Then there are constants C, ✏ 0 > 0 dependingon h ⇤ such that P ✓ k
[ ` D ( ✓ k )  ✏ 0 ]  C ⌘ L 2 ⌧
[Equation]
The interpretation is that if h ⇤ does not satisfy l -MSP, then to learn f ⇤ to better than ✏ 0 error withconstant probability, we need at least ⌦ h ⇤ , ✏ ( d l +1 ) steps of GD on a O h ⇤ , ✏ (1) -Lipschitz networkwith step size ⌘ =
O h
⇤ , ✏ (1) and noise level ⌧ = ⌦ h ⇤ , ✏ (1) .
The proof is deferred to Appendix C.
It proceeds by ﬁrst isolating the “easily-reachable” coordinates T ✓ [ P ] , and subtracting theircontribution from f ⇤ .
We then bound G -orbit-alignment of the resulting function, where G is thepermutation group on [ d ] \ T .
Hardness for learning with SGD
In this section, for  > 0 , we let D ( f, µ X ,  ) 2 P ( X ⇥ R ) denote the distribution of ( x , f ( x ) + ⇠ )where x ⇠ µ X and ⇠ ⇠ N (0 ,  2 ) is independent noise.
We show that the equivariance of SGD implies that the function f mod8 : H d ! { 0 , . . .
, 7 } given by
[Equation]
[Equation]
is hard for SGD-trained, i.i.d.
symmetrically-initialized fully-connected networks.
Our hardnessresult is the ﬁrst of its kind for neural network training, as it relies on a cryptographic assumption toprove superpolynomial lower bounds for SGD learning. We introduce the assumption.
For any S ✓ [ d ] , let  S : H d ! { +1 ,  1 } d be the parity function  S ( x ) = Q
[Equation]
2 S x i .
Deﬁnition 4.1.
The learning parities with Gaussian noise, ( d, n,  ) -LPGN, problem is parametrizedby d, n 2 Z > 0 and  2 R > 0 .
An instance ( S, q , ( x i , y i ) i 2 [ n ] ) consists of (i) an unknown subset
S ✓ [ d ] of size | S | =
b d/ 2 c , and (ii) a known query vector q ⇠ { +1 ,  1 } d , and i.i.d.
samples 342 ( x i , y i ) i 2 [ n ] ⇠ D (  S , H d ,  ) .
The task is to return  S ( q ) 2 { +1 ,  1 } .
Our cryptographic assumption is that poly( d ) -size circuits cannot succeed on LPGN.
Deﬁnition 4.2.
Let  > 0 .
We say  -LPGN is poly( d ) -time solvable if there is a sequence ofsample sizes { n d } d 2 N and circuits {A d } d 2 N such that n d , size
( A d )  poly( d ) , and A d solves 346 ( d, n d ,  ) -LPGN with success probability at least 9 / 10 , when inputs are rounded to poly( d ) bits.
Assumption 4.3.
Fix  .
The  -LPGN-hardness assumption is:  -LPGN is not poly( d ) -time solvable.
The LPGN problem is the simply standard Learning Parities with Noise problem (LPN)
[BKW03],except with Gaussian noise instead of binary classiﬁcation noise, and we are also promised that| S |
=
b d/ 2 c .
In Appendix D.3, we derive Assumption 4.3 from the standard hardness of LPN.
Wenow state our SGD hardness result.
Theorem 4.4.
Let { f NN ,d , µ ✓ ,d } d 2 N be a family of networks and initializations satisfying Assump- tion 2.4 (fully-connected) with i.i.d.
symmetric initialization.
Let  > 0 , and let { n d } be sample sizes such that ( f NN ,d , µ ✓ ,d ) -SGD training on n d samples from D ( f mod8 , H d ,  ) rounded to poly( d ) bits yields parameters ✓ d with
[Equation]
Then, under (  / 2) -LPGN hardness, ( f NN ,d , µ ✓ ,d ) -SGD on n d samples cannot run in poly( d ) time.
In order to prove Theorem 4.4, we use the sign-ﬂip equivariance of gradient descent guaranteed bythe symmetry in the initialization.
Indeed, a sign-ﬂip equivariant network that learns f mod8 ( x ) from -noisy samples, is capable of solving the harder problem of learning f mod8 ( x  s ) from  -noisysamples, where s 2 H d is an unknown sign-ﬂip vector.
However, through an average-case reductionwe show that this problem is (  / 2) -LPGN-hard.
Discussion
Our general GD lower bound in Theorem 3.1, and our idea of basing hardness for SGD training oncryptographic assumptions could be further developed to other settings, such as architectures andinitializations beyond those considered in this paper.
An outstanding open question is whether computational lower bounds for SGD/GD training canbe shown beyond those implied by equivariance.
For example, consider the function f : H d !{ +1 ,  1 } that computes the “full parity”, i.e., the parity of all of the inputs f ( x ) =
Q [Equation]
365 Past work has empirically shown that SGD on fully-connected networks with Gaussian initialization[SSSS17, AS20, NY21] fails to learn this function.
Proving this would represent a signiﬁcant advancein our theory for lower bounds, since there is no obvious equivariance that implies that the full parityis hard to learn. Indeed, we have shown here that the full parity can be learned by fully-connectednetworks with symmetric Rad(1 / 2) initialization in which case training is G sign,perm -equivariant.
Checklist
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]