Original: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Revised: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: There is mountingempirical evidence of emergent phenomena in the capabilitiesof deep learning methods as we scale up datasets, model sizes, and training times.
Revised: There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times.

Original: While there are some accounts of how these resources modulate statistical capacity,far less is known about their effect on the computational problem of model training.
Revised: While there are some accounts of how these resources modulate statistical capacity, far less is known about their effect on the computational problem of model training.

Original: This work conducts such an exploration through the lens of learning  k -sparseparities of n bits, a canonical family of problems which pose theoretical computational barriers.
Revised: This work conducts such an exploration through the lens of learning a k -sparse parity of n bits, a canonical discrete search problem which is statistically easy but computationally hard.

Original: In this setting, we find thatneural networks exhibit surprisingphase transitions when scaling up dataset size and running time. In particular, wedemonstrate empirically that with standard training, a variety of architectures learnsparse parities with n O ( k ) examples, with loss (and error) curves abruptly droppingafter n O ( k ) iterations.
Revised: Empirically, we ﬁnd that a variety of neural networks successfully learn sparse parities, with discontinuous phase transitions in the training curves. On small instances, learning abruptly occurs at approximately n O ( k ) iterations; this nearly matches SQ lower bounds, despite the apparent lack of a sparse prior.

Original: These positive results nearly match known SQ lower bounds,even without an explicit sparsity-promoting prior. We elucidate the mechanisms ofthese phenomena with a theoretical analysis: we find that the phase transition inperformance is not due to SGD “stumbling in the dark” until it finds the hidden setof features (a natural algorithm which also runs in n O ( k ) time); instead, we showthat SGD gradually amplifies a Fourier gap in the population gradient.
Revised: Our theoretical analysis shows that these observations are not explained by a Langevin-like mechanism, whereby SGD “stumbles in the dark” until it ﬁnds the hidden set of features (a natural algorithm which also runs in n O ( k ) time). Instead, we show that SGD gradually ampliﬁes the sparse solution via a Fourier gap in the population gradient, making continual progress that is invisible to loss and error metrics.

Original:  Introduction
Revised:  Introduction

Original: Neural networks perform better with more resources (data, model size, training time) but differenttasks exhibit qualitatively different dependencies of performance on resources.
Revised: In deep learning, performance improvements are frequently observed upon simply scaling up resources (such as data, model size and training time).

Original: In particular, whilemany learning tasks exhibit continuousimprovement in performance with increasing resources, othercases show discontinuous improvement, where a capability emerges at a certain threshold.
Revised: While these improvements are often continuous in terms of these resources, some of the most surprising recent advances in the ﬁeld have been emergent capabilities : at a certain threshold, behavior changes qualitatively and discontinuously .

Original: Through astatistical lens, it is well-understood that larger models, trained with more data, can fit more complexand expressive functions.
Revised: Through a statistical lens, it is well-understood that larger models, trained with more data, can ﬁt more complex and expressive functions.

Original: However, far less is known about the analogous computational question:how does the scaling of these resources influence the success of gradient-based optimization fortraining these models?
Revised: However, far less is known about the analogous computational question: how does the scaling of these resources inﬂuence the success of gradient-based optimization?

Original: Theseemergent phase transitions cannot be explained via statistical capacity alone. In many cases wesee a phase transition even when the amount of data remains fixed, with only model size or trainingtime increasing.
Revised: These phase transitions cannot be explained via statistical capacity alone: they can appear even when the amount of data remains ﬁxed, with only model size or training time increasing.

Original: A timely example is the emergence of reasoning and few-shot learning capabilitieswhen scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022).
Revised: A timely example is the emergence of reasoning and few-shot learning capabilities when scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022); Srivastava et al.

Revised: (2022) identify various tasks which language models are only able to solve if they are larger than a critical scale.

Original: Power et al.
Revised: Power et al.

Original: (2022) give examples exhibiting discontinuous improvements inpopulation accuracy (“grokking”) when running time increases, while data and model size remainfixed.
Revised: (2022) give examples of discontinuous improvements in population accuracy (“grokking”) when running time increases, while dataset and model sizes remain ﬁxed.

Revised: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: In this work, we analyze the computational aspect of scaling in deep learning in a simple syntheticsetting which already exhibits discontinuous improvements.
Revised: In this work, we analyze the computational aspects of scaling in deep learning, in an elementary synthetic setting which already exhibits discontinuous improvements.

Original: Specifically, we consider the sparseparity problem — where the label is the parity (XOR) of k ≪ n bits in a random length- n binary string.
Revised: Speciﬁcally, we consider the supervised learning problem of learning a sparse parity : the label is the parity (XOR) of k ⌧ n bits in a random length- n binary string.

Original: Thisis a canonical problemwhich is computationally difficult for a range of algorithms, including [Figure] , 1998) and streaming (Kol et al., 2017) algorithms.
Revised: This problem is computationally difﬁcult for a range of algorithms, including gradient-based (Kearns, 1998) and streaming (Kol et al., 2017) algorithms.

Original: We focus on analyzingthe resource measure of training time , and demonstrate that the loss curves for sparse parities displaya phase transition across a variety of architectures and hyperparameters (see Figure 1, left).
Revised: We focus on analyzing the resource measure of training time , and demonstrate that the loss curves for sparse parities display a phase transition across a variety of architectures and hyperparameters (see Figure 1, left).

Original: A strikingobservation is that SGD robustly finds the sparse subset (and hence,effectively, reaches 0 error) witha variety of activation functions and initialization schemes, even with no over-parameterization.
Revised: Strikingly, we observe that SGD ﬁnds the sparse subset (and hence, reaches 0 error) with a variety of activation functions and initialization schemes, even with no over-parameterization.

Original: Thisrobust convergence is the starting point for our investigation. Perhaps the most natural hypothesis to explain SGD’s success in learning parities would be that itsimply “stumbles in the dark”,essentially performing random search for the unknown target (e.g.via stochastic gradient Langevin dynamics).
Revised: A natural hypothesis to explain SGD’s success in learning parities, with no visible progress in error and loss for most of training, would be that it simply “stumbles in the dark”, performing random search for the unknown target (e.g. via stochastic gradient Langevin dynamics).

Original: If that were the case, we might expect to observe aconvergence time of 2 Ω( n ) , like a naive search over parameters or subsets of indices.
Revised: If that were the case, we might expect to observe a convergence time of 2 ⌦ ( n ) , like a naive search over parameters or subsets of indices.

Original: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: theconvergence time is closer to an n O ( k ) scaling.
Revised: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: the convergence time adapts to the sparsity parameter k , with a scaling of n O ( k ) on small instances.

Original: Notably, such a convergence rate implies that SGD iscloser to achieving the optimal computation time among a natural class of algorithms, namely SQ(statistical query) algorithms.
Revised: Notably, such a convergence rate implies that SGD is closer to achieving the optimal computation time among a natural class of algorithms (namely, statistical query algorithms).

Original: In this work, through an extensive empirical analysis of the scaling behavior of a variety of models,as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.
Revised: Through an extensive empirical analysis of the scaling behavior of a variety of models, as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.

Original: Instead, there is a hidden progress measure under which SGD is steadily improving.
Revised: Instead, there is a hidden progress measure under which SGD is steadily improving.

Original: Furthermore, andperhaps surprisingly, we show that SGD achieves a computational runtime much closer to the optimal SQ lower bound than simply doing (non-sparse) parameter search.
Revised: Furthermore, and perhaps surprisingly, we show that SGD achieves a computational runtime much closer to the optimal SQ lower bound than simply doing (non-sparse) parameter search.

Original: More generally, ourtheoreticaland empirical investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources, and we provide further theoretical and empirical evidence of phasetransitions with data, model size, and training time.
Revised: More generally, our investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources: we identify phase transitions when varying data, model size, and training time.

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: SGDefficiently learns sparse parities, in theory and practice.
Revised: SGD learns sparse parities.

Original: It is known from SQ lowerbounds that (noisy) gradient descent on any architecture requires at least n Ω( k ) computational stepsto learn k -sparse n -dimensional parities (for background, see Appendix A). However, with standardarchitectures and initialization, which do not explicitly encode a sparsity prior, one may expect SGD’sperformance to be much worse, on the order of 2 Ω( n ) . We give extensive empirical evidence that thisis not the case.
Revised: It is known from SQ lower bounds that with a constant noise level, gradient descent on any architecture requires at least n ⌦ ( k ) computational steps to learn k -sparse n -dimensional parities (for background, see Appendix A). We ﬁrst show a wide variety of positive empirical results, in which neural networks successfully solve the parity problem in a number of iterations which scales near this computational limit:

Original: Empirical Finding 1.
Revised: Empirical Finding 1.

Original: For all small instances ( n ≤ 30 , k ≤ 4 ) of the sparse parity problem,architectures A ∈ { 2-layer MLPs, Transformers 1 , sinusoidal (and other non-standard) neurons, PolyNets 2 } , initializations in { uniform, Gaussian, Bernoulli } , and batch sizes 1 ≤ B ≤ 1024 , SGDWith on A solves the ( n, k ) -sparse parity problem (w.p. ≥ 0 . 2 ) within at most c · n αk steps, for smallarchitecture-dependent constants c, α .
Revised: For all small instances ( n  30 , k  4 ) of the sparse parity problem, architectures A 2 { 2-layer MLPs, Transformers 1 , sinusoidal/oscillating neurons, PolyNets 2 } , initial- izations in { uniform, Gaussian, Bernoulli } , and batch sizes 1  B  1024 , SGD on A solves the ( n, k ) -sparse parity problem (w.p.  0 . 2 ) within at most c · n ↵ k steps, for small constants c, ↵ .

Original: For a subset of these architectures, we performed more extensive training, to show scaling behaviorsof the computation time; 3 see Figure 1 (right) and Figure 10 in the appendix.

Original: The above results indicate that SGD succeeds at solving parities much faster than the 2 Ω( n ) steps thatwould be required by random search without a sparse prior. This suggests that despite flat loss andaccuracy curves before the phase transition, SGD makes progress “under the hood”. We show thatthis is indeed the case by coming up with a progress measure that continuously improves throughouttraining. We can explain the loss and accuracy curves using this progress measure, by showing thatthey typically rise sharply once the measure passes a certain threshold.
Revised: With a smaller range of hyperparameters

Original: Theoretical Analysis.
Revised: Theoretical analyses of sparse feature emergence.

Original: Our empirical results suggest that, in a number of computational stepsmatching the SQ limit, SGD is able to solve the parity problem and identify the influential coordinates,without an explicit sparse prior.
Revised: Our empirical results suggest that, in a number of computational steps matching the SQ limit, SGD is able to solve the parity problem and identify the inﬂuential coordinates, without an explicit sparse prior.

Original: We give a theoretical analysis which validates this claim.
Revised: We give a theoretical analysis which validates this claim.

Original: Informal Theorem 2.
Revised: Informal Theorem 2.

Original: On 2-layer MLPs of width 2 Θ( k ) , and with batch size n O ( k ) , SGD convergeswith high probability to a solution with at most ϵ error on the ( n, k ) -parity problem in at most2 O ( k ) · poly(1 /ϵ ) iterations.
Revised: On 2-layer MLPs of width 2 ⇥ ( k ) , and with batch size n O ( k ) , SGD converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most 2 O ( k ) · poly(1 / ✏ ) iterations.

Original: We also present a stronger analysis for an idealized architecture (which we call the disjoint-PolyNet ),which allows for any batch size, and captures the phase transitions observed in the error curves.
Revised: We also present a stronger analysis for an idealized architecture (which we call the disjoint-PolyNet ), which allows for any batch size, and captures the phase transitions observed in the error curves.

Original: Informal Theorem 3.
Revised: Informal Theorem 3.

Original: On disjoint-PolyNets, SGD (with any batch size B ≥ 1 ) converges with highprobability to a solution with at most ϵ error on the ( n, k ) -parity problem in at most n O ( k ) · log(1 /ϵ )iterations.
Revised: On disjoint-PolyNets, SGD (with any batch size B  1 ) converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most n O ( k ) · log(1 / ✏ ) iterations.

Original: Continuous-time gradient flow exhibits a phase transition: it spends a 1 − o (1) fraction ofits time before convergence with error ≥ 49% .
Revised: Continuous-time gradient ﬂow exhibits a phase transition: it spends a 1  o (1) fraction of its time before convergence with error  49% .

Original: In addition to refuting the alternative “random search” hypothesis, our work also poses a counterexample to other models for thecomputational mechanisms of deep learning; for instance, itprovides a setting where deep neural nets successfully learn a concept to 100% accuracy while thecorresponding Neural Tangent Kernel (NTK) only achieves trivial performance, hence showing theimportance of  feature learning .
Revised: Our theoretical and empirical results hold in non-overparameterized regimes (including with a widthsinusoidal neuron), in which no ﬁxed kernel, including the neural tangent kernel (NTK) (Jacot et al., 2018), is sufﬁciently expressive to ﬁt all sparse parities with a large margin. Thus, our ﬁndings comprise an elementary example of combinatorial feature learning : SGD can only successfully converge by learning a low-width sparse representation. Further empirical explorations.

Original: We also construct a counterexample to the “deep only works ifshallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating a case where adeep network can get near-perfect accuracy even when greedy layerwise training (e.g. Belilovskyet al. (2019)) cannot beat trivial performance. By providing positive theory and empirics which eludethese simplified explanations of SGD, we hope to point the way to  a more complete understanding oflearning dynamics in  the challenging cases where no apparent progress  is made for extended periodsof time.
Revised: Building upon our core positive results, we provide a wide variety of preliminary experiments, showing sparse parity learning to be a versatile testbed for understanding the challenges and surprises in solving combinatorial problems with neural networks. These include quantities which reveal the continual hidden progress behind uninformative training curves (as predicted by the theory), experiments at small sample sizes which exhibit grokking (Power et al., 2022), as well as an example where greedy layer-wise learning is impossible but end-to-end SGD can learn the layers jointly.

Original: 1.2 Related work
Revised: 1.2 Related work

Original: We present the most directly related workbelow, on feature learning,  learning parities with neuralnets, and the statistical mechanics of machine learning.
Revised: We present the most directly related work on feature learning, and learning parities with neural nets.

Original: A broader discussionof related work can befound in Appendix A.3.
Revised: A broader discussion can be found in Appendix A.3.

Original: SGD and feature learning.
Revised: SGD and feature learning.

Original: Theoretical analysis of gradient descent over neural networks isnotoriously hard, due to the non-convex nature of the optimization problem.
Revised: Theoretical analysis of gradient descent on neural networks is notoriously hard, due to the non-convex nature of the optimization problem.

Original: That said, it has beenestablished that in some settings, the dynamics of GD keep the weights close to their initialization,thus behaving like convex optimization over the Neural Tangent Kernel (see, for example, Jacot et al.(2018); Allen-Zhu et al.
Revised: That said, it has been established that in some settings, the dynamics of GD keep the weights close to their initialization, thus behaving like convex optimization over the Neural Tangent Kernel (see, for example, (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2018)).

Original: Du et al. In contrast, it has been shown that in various tasks,moving away from the fixed features of the NTK is essential for the success of neural networks trainedwith GD (for example Yehudai and Shamir (2019); Allen-Zhu and Li (2019); Wei et al. (2019) andthe review in Malach et al.
Revised: In contrast, it has been shown that in various tasks, moving away from the ﬁxed features of the NTK is essential for the success of neural networks trained with GD (for example (Yehudai and Shamir, 2019; Allen-Zhu and Li, 2019; Wei et al., 2019) and the review in (Malach et al., 2021)).

Original: These results demonstrate that feature learning is an importantpart of the GD optimization process.
Revised: These results demonstrate that feature learning is an important part of the GD optimization process.

Original: Our work also focuses on a setting where feature learning isessential for the target task.
Revised: Our work also focuses on a setting where feature learning is essential for the target task.

Original: In our theoretical analysis, we show that the initial population gradientencodes the relevant features for the problem.
Revised: In our theoretical analysis, we show that the initial population gradient encodes the relevant features for the problem.

Original: The importance of the first gradient step for featurelearning has been recently studied in (Ba et al., 2022).
Revised: The importance of the ﬁrst gradient step for feature learning has been recently studied in (Ba et al., 2022).

Original: Learning parities with neural networks.
Revised: Learning parities with neural networks.

Original: The problem of learning parities using neural networkshas been investigated in prior works from various perspectives.
Revised: The problem of learning parities using neural networks has been investigated in prior works from various perspectives.

Original: It has been demonstrated that paritiesare hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartzet al., 2017; Abbe and Sandon, 2020).
Revised: It has been demonstrated that parities are hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartz et al., 2017; Abbe and Sandon, 2020).

Original: One possible approach for overcoming the computationalhardness is to make favorable assumptions on the input distribution.
Revised: One possible approach for overcoming the computational hardness is to make favorable assumptions on the input distribution.

Original: Indeed, recent works show thatunder various assumptions on the input distribution, neural networks can be efficiently trained tolearn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;
Revised: Indeed, recent works show that under various assumptions on the input distribution, neural networks can be efﬁciently trained to learn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;

Original: Frei et al., 2022; Malach et al.,2021).
Revised: Frei et al., 2022; Malach et al., 2021).

Original: In contrast to these results, this work takes the approach of intentionally focusing on a hardbenchmark task, without assuming that the distribution has some favorable (namely, non-uniform)structure.
Revised: In contrast to these results, this work takes the approach of intentionally focusing on a hard benchmark task, without assuming that the distribution has some favorable (namely, non-uniform) structure.

Original: This setting allows us to probe the performance of deep learning at a known computationallimit.
Revised: This setting allows us to probe the performance of deep learning at a known computational limit.

Original: Notably, the work of Andoni et al.
Revised: Notably, the work of Andoni et al.

Original: (2014) provides analysis for learning polynomials (and inparticular, parities) under the uniform distribution.
Revised: (2014) provides analysis for learning polynomials (and in particular, parities) under the uniform distribution.

Original: However, their main results require a networkof size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical andempirical evidence for the success of smaller networks.
Revised: However, their main results require a network of size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical and empirical evidence for the success of smaller networks.

Original: Studying a related subject, some works haveshown that neural networks display a spectral bias, learning to fit low-frequency coefficients beforehigh-frequency ones (Rahaman et al., 2019; Cao et al., 2019).
Revised: Studying a related subject, some works have shown that neural networks display a spectral bias, learning to ﬁt low-frequency coefﬁcients before high-frequency ones (Rahaman et al., 2019; Cao et al., 2019).

Original: Statistical mechanics of machine learning.

Original: An extensive body of work originating in the statisticalphysics community has studied phase transitions in the learning curves of neural networks (Gardnerand Derrida, 1989;

Original: Watkin et al., 1993; Engel and Van den Broeck, 2001).

Original: These works typicallyfocus on student-teacher learning in the “thermodynamic limit”, in which the number of trainingexamples is α times larger than the input dimension and both are taken to infinity.

Original: One of theclassic toy architectures analyzed in this literature is the parity machine (Mitchison and Durbin,1989; Hansel et al., 1992; Opper, 1994; Kabashima, 1994; Simonetti and Caticha, 1996).

Original: In ourwork, we introduce PolyNets, a variant of parity machines in which the output is real-valued ratherthan ± 1 ; and we theoretically analyze disjoint-Polynets, which are the real-output analogue of theoft-considered parity machines with tree architecture.

Original: While much of the statistical mechanics of

Original: ML literature focuses on an idealized training limit in which the weights reach a Gibbs distributionequilibrium, there is a strand of the literature that aims to characterize the trajectory of SGD trainingin the high-dimensional limit with constant-sized sets of ordinary differential equations (Saad and

Original: Solla, 1995b,a; Goldt et al., 2019).

Original: These papers discuss cases, including problems that share aspectswith 2-sparse parities Refinetti et al.

Original: (2021), where the network gets stuck in (and then escapes from)a plateau of suboptimal generalization error.

Original: Recently, Arous et al.

Original: (2021) studied (for rank-oneparameter estimation problems) the relative amount of time spent by SGD in an initial high-error“search” phase versus a final “descent” phase, which is reminiscent of the framing of Theorem 7.

Original: However, to our knowledge prior work has not shown k -sparse parities can be learned with a numberof iterations that nearly matches known lower bounds, nor has it specifically studied phase transitionsin k -sparse parity learning during gradient descent.

Original:  Preliminaries
Revised:  Preliminaries

Original: We define necessary notation here; see Appendix A for more background and technical ingredients.
Revised: We provide an expanded discussion of background and related work in Appendix A.

Original: Sparse parities.
Revised: Sparse parities.

Original: For integer n ≥ 1 and non-empty  S ⊆ [ n ] , the ( n, S ) - parity function χ S :{± 1 } n → {± 1 } is defined as χ S ( x ) = (cid:81) i ∈ S x i .
Revised: For integer n  1 and non-empty set S ✓ [ n ] , the ( n, S ) - parity function  S : {± 1 } n ! {± 1 } is deﬁned as  S ( x ) = Q i 2 S x i .

Original: We define the ( n, S ) - parity distribution D S asthe distribution over ( x, y = χ S ( x )) 4 where x is drawn fromthe uniform distribution over {± 1 } n ,which we denote by Unif( {± 1 } n ) .
Revised: We deﬁne the ( n, S ) - parity distribution D S as the joint distribution over ( x, y ) 3 where x is drawn from Unif( {± 1 } n ) , the uniform distribution over random length- n sign vectors, and y :=

Revised:  S ( x ) is the product of the inputs at the indices given by the “relevant features” S (thus, ± 1 , depending on whether the number of relevant  1 inputs is even or odd).

Original: We define the ( n, k ) - parity learning problem as the task ofrecovering S using samples from D S , where S is chosen at random in (cid:0) [ n ] k (cid:1) .
Revised: We deﬁne the ( n, k ) - parity learning problem as the task of recovering the S using samples from D S , where S is chosen at random from  [ n ] k  .

Original: Statistically, it is possibleto do so using Θ(log (cid:0) nk (cid:1) )
Revised: A key fact about parities is that they are orthogonal under the correlation inner product: for S 0 ✓ [ n ] , [Equation] That is, a learner who guesses indices S 0 cannot use correlations (equivalently, the accuracy of the hypothesis  S 0 ) as feedback to reveal which indices in S 0 are correct, unless S 0 is exactly the correct subset. This notion of indistinguishability leads to a computational lower bound in the statistical query (SQ) model (Kearns, 1998): ⌦ ( n k ) constant-noise queries are necessary, which is far greater than the statistical limit of ⇥ (log  nk  )

Original: ≈ k log n samples.
Revised: ⇡ k log n samples.

Original: However, in the statistical query (SQ) model (Kearns,1998), (which has been shown to encapsulate gradient-based methods such as GD or SGD (Abbeet al., 2021)), this task requires Ω( n k ) queries (assuming constant level of noise).
Revised: The hardness of parity has been used to derive computational hardness results for other settings, like agnostically learning halfspaces (Klivans and Kothari, 2014) and MLPs (Goel et al., 2019).

Original: While learningnoiseless parities can be solved by Gaussian elimination using O ( n ) samples, learning sparse noisyparities, even at a very small noise level (i.e., o (1) or n − δ ) is believed to inherently require n Ω( k )computational steps and samples, or exponential computation with n o ( k ) samples. This was first
Revised: Beyond the restricted computational model of statistical queries, noiseless parities can be learned in poly( n ) time via Gaussian elimination. However, learning sparse noisy parities, even at a very small noise level (i.e., o (1) or n   ), is believed to inherently require n ⌦ ( k ) computational steps.

Original: [Figure] explicitly conjectured by Alekhnovich (2003), and has been the basis for several cryptographicschemes (e.g., (Ishai et al., 2008; Applebaum et al., 2009, 2010; Bogdanov et al., 2019)).
Revised: In all, learning sparse parities is a well-studied combinatorial problem which exempliﬁes the computational difﬁculty of learning a joint dependence on multiple relevant features.

Original: Notation for neural networks and training.
Revised: Notation for neural networks and training.

Original: Our main results are presented in the online learningsetting, with a stream of i.i.d.
Revised: Our main results are presented in the online learning setting, with a stream of i.i.d.

Original: batches of examples.
Revised: batches of examples.

Original: At each iteration t = 1 , . . .
Revised: At each iteration t = 1 , . . .

Original: , T , a learningalgorithm receives a batch of B examples { ( x t,i , y t,i ) }
Revised: , T , a learning algorithm A receives a batch of B examples { ( x t,i , y t,i ) }

Original: Bi =1 drawn i.i.d.
Revised: Bi =1 drawn i.i.d.

Original: from D S , then outputs aclassifier (cid:98) y t : {± 1 } n → {± 1 } .
Revised: from D S , then outputs a classiﬁer b y t : {± 1 } n ! {± 1 } .

Original: We say that the algorithm solves the parity task in t steps with ϵ error,if with probability at least 1 − ϵ over both training and internal randomness, as well as ( x, y ) ∼ D S ,(cid:98)
Revised: We say that A solves the parity task in t steps (with error ✏ ) if

Original: y t ( x ) = y .
Revised: [Equation]

Original: We will focus on the case that (cid:98) y t = sign( f ( x ; θ t )) for some parameters θ t in a continuousdomain Θ and for a continuous function f : {± 1 } n × Θ → R .
Revised: We will focus on the case that b y t = sign( f ( x ; ✓ t )) for some parameters ✓ t in a continuous domain ⇥ and for a continuous function f : {± 1 } n ⇥ ⇥ !

Original: A ubiquitous online learningalgorithm is gradient descent (GD).
Revised: R 5 , updated with the ubiquitous online learning algorithm of gradient descent (GD), whose update rule is given by

Original: For achoice of loss function ℓ : {± 1 } × R → R ,initializationθ 0 (that are chosen from some distribution), learning rate schedule { η t } Tt [-=
-]
Revised: [Equation] for a loss function ` : {± 1 } ⇥ R ! R , learning rate schedule { ⌘ t } Tt =1 , and weight decay schedule {  t } Tt [-=
-]=1 6 .

Original: ⊆ R and weight-decayschedule { λ t } Tt =1 ⊆ R , GD refers the standard iterative update rule using the regularized, empiricalloss function, which is a function of architecture f . The learning rate η t can also be a vector (e.g.,allowing different rate schedules for different layers).
Revised: The initialization ✓ 0 is drawn randomly from a chosen distribution.

Original:  Empirical findings
Revised:  Empirical ﬁndings 3.1 SGD on neural networks learns sparse parities

Original: 3.1 SGD on neural networks learns sparse parities

Original: The central phenomenon of study in this work is the empirical observation that neural networks, withstandard initialization and training, can solve the ( n, k ) -parity problem in a number of iterationsscaling as n O ( k ) on small instances.
Revised: The central phenomenon of study in this work is the empirical observation that neural networks, with standard initialization and training, can solve the ( n, k ) -parity problem in a number of iterations scaling as n O ( k ) on small instances.

Original: We observed robust positive results for randomly-initialized
Revised: We observed robust positive results for randomly-initialized

Original: SGD on the following architectures, indexed by Roman numerals:
Revised: SGD on the following architectures, indexed by Roman numerals:

Original: • 2-layer MLPs: ReLU ( σ ( z ) =
Revised: • 2-layer MLPs: ReLU (  ( z ) =

Original: ( z ) + ) or polynomial ( σ ( z )
Revised: ( z ) + ) or polynomial (  ( z )

Original: =
Revised: =

Original: z k ) activation, in a wide variety ofwidth regimes r ≥ k .
Revised: z k ) activation, in a wide variety of width regimes r  k .

Original: Settings (i), (ii), (iii) (resp.
Revised: Settings (i), (ii), (iii) (resp.

Original: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU(resp. polynomial) activations.
Revised: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU (resp. polynomial) activations.

Original: We also consider r = k (exceptional settings (*i), (*ii) ), theminimum width for representing a k -wise parity for both activations.
Revised: We also consider r = k (exceptional settings (*i), (*ii) ), the minimum width for representing a k -wise parity for both activations.

Original: • Single neurons: Next, we consider non-standard activation functions σ which allow a one-neuron 
Revised: • 1-neuron networks: Next, we consider non-standard activation functions  which allow a one-neuron architecture f ( x ; w ) =

Original: architecture f (x ; w ) = σ ( w ⊤ x ) to realize k -wise parities.
Revised:  ( w > x ) to realize k -wise parities.

Original: The constructions stem from lettingw ∗ =
Revised: The constructions stem from letting w ⇤ =

Original: (cid:80) i ∈ S e i , and constructing σ ( · ) to interpolate (the appropriate scaling of)
Revised: P i 2 S e i , and constructing  ( · ) to interpolate (the appropriate scaling of)

Original: k − w ∗⊤ xmod 2with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).
Revised: k  w ⇤> xmod 2 with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).

Original: Going a stepfurther, a single ∞ -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.
Revised: Going a step further, a single 1 -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.

Original: We alsoremoved the second trainable layer (setting u = 1 ), obtaining settings (xi), (xii), (xiii), (xiv).
Revised: In settings (xi), (xii), (xiii), (xiv), we remove the second trainable layer (setting u = 1 ).

Original: Wefound that wider architectures with these activations also trained successfully.
Revised: We ﬁnd that wider architectures with these activations also train successfully.

Original: [Figure]
Revised: • Transformers: There is growing interest in using parity as a benchmark for combinatorial function learning, long-range dependency learning, and length generalization in Transformers (Lu et al., 2021;

Original: • Transformers: Motivated by  recent theoretical and empirical work on the ability of self-attention tolearn sparse functions and parities (Lu et al., 2021; Edelman et al., 2021; Hahn, 2020), we considera simplified specialization of the Transformer architecture to this sequence classification problem.[Figure]
Revised: Edelman et al., 2021; Hahn, 2020; Anil et al., 2022; Liu et al., 2022). Motivated by these recent theoretical and empirical works, we consider a simpliﬁed specialization of the Transformer architecture to this sequence classiﬁcation problem.

Original: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.
Revised: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.

Original:  • PolyNets: Our final setting (xv) is the PolyNet, a slightly modified version of the parity machinearchitecture.
Revised: 3. • PolyNets: Our ﬁnal setting (xv) is the PolyNet, a slightly modiﬁed version of the parity machine architecture.

Original: Parity machines have been studied extensively in the statistical mechanics of MLliterature (see the related work section) as well as in a line of work on ‘neural cryptography’(Rosen-Zvi et al., 2002).
Revised: Parity machines have been studied extensively in the statistical mechanics of ML literature (see the related work section) as well as in a line of work on ‘neural cryptography’ (Rosen-Zvi et al., 2002).

Original: A parity machine outputs the sign of the product of k linear functions of the input.
Revised: A parity machine outputs the sign of the product of k linear functions of the input.

Original: A PolyNet simply outputs the product itself.
Revised: A PolyNet simply outputs the product itself.

Original: Both architectures can clearly realizek -sparse parities.
Revised: Both architectures can clearly realize ksparse parities.

Original: The

Original:  PolyNet architecture was originally motivated by the search for an idealized setting where an end-to-end optimization trajectory analysis is tractable (see Section 4.1); we 
Revised: The PolyNet architecture was originally motivated by the search for an idealized setting where an end-to-end optimization trajectory analysis is tractable (see Section 4.1); we found in these experiments that this architecture trains very stably and sample-efﬁciently.

Original: found in

Original: these experiments

Original: that

Original: this architecture

Original: trains

Original: very

Original: stably

Original: and

Original: sample

Original: -efficiently.

Original: Robust space of positive results.
Revised: Robust space of positive results.

Original: All of the networks listed above were observed to successfullylearn sparse parities in a variety of settings.
Revised: All of the networks listed above were observed to successfully learn sparse parities in a variety of settings.

Original: We summarize our findings as follows: for all combinations of n ∈ { 10 , 20 , 30 } , k ∈ { 2 , 3 , 4 } , batch sizes B ∈ { 1 , 2 , 4 , . . . , 1024 } , initializations{ uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architectureconfigurations { (i) , (ii) , . ..
Revised: We summarize our ﬁndings as follows: for all combinations of n 2 { 10 , 20 , 30 } , k 2 { 2 , 3 , 4 } , batch sizes B 2 { 1 , 2 , 4 , . . . , 1024 } , initializations { uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture conﬁgurations { (i) , (ii) , . .

Original: , (xv) } , SGD solved the parity problem (with 100% accuracy, validatedon a batch of 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rateη ∈ { 0 . 001 , 0 . 01 , 0 .
Revised: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batch of 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rate ⌘ 2 { 0 . 001 , 0 . 01 , 0 .

Original: The models converged in t c ≤ c · n αk ≤ 10 5 steps, for small architecturedependent constants c, α , shown in Appendix C. Figure 1 (left) shows some representative trainingcurves.
Revised: The models converged in t c  c · n ↵ k  10 5 steps, for small architecture-dependent constants c, ↵ (see Appendix C). Figure 1 (left) shows some representative training curves.

Original: Less robust configurations.
Revised: Less robust conﬁgurations.

Original: Settings (*i) and (*ii), where the MLP just barely represents a k -sparseparity, and the Transformer setting (*iii), are less robust to small batch sizes.
Revised: Settings (*i) and (*ii), where the MLP just barely represents a k -sparse parity, and the Transformer setting (*iii), are less robust to small batch sizes.

Original: In these settings, thesame positive results as above only held for sufficiently large batch sizes: B ≥ 16 .
Revised: In these settings, the same positive results as above only held for sufﬁciently large batch sizes: B  16 .

Original: Also, setting (*iii)used the Adam optimizer; see Appendix D.1.3 for details.
Revised: Also, setting (*iii) used the Adam optimizer (which is standard for Transformers); see Appendix D.1.3 for details.

Original: Phase transitions in training curves.
Revised: Phase transitions in training curves.

Original: For almost all of the architectures, we find that that thetraining curves exhibit phase transitions in terms of running time (and thus, in the online learningsetting, dataset size as well): long durations of seemingly no progress, followed by periods of rapiddecrease in the validation error.
Revised: For almost all of the architectures, we ﬁnd that that the training curves exhibit phase transitions in terms of running time (and thus, in the online learning setting, dataset size as well): long durations of seemingly no progress, followed by periods of rapid decrease in the validation error.

Original: Strikingly, for architectures (v) and (vi), this plateau is absent: theerror in the initial phase appears to decrease with a linear slope.
Revised: Strikingly, for architectures (v) and (vi), this plateau is absent: the error in the initial phase appears to decrease with a linear slope.

Original: See Appendix C.8 for more plots.
Revised: See Appendix C.8 for more plots.

Original: 3.2 Random search or hidden progress?
Revised: 3.2 Random search or hidden progress?

Original: The remainder of this paper seeks to answer the question: “By what mechanism does deep learningsolve these emblematic computationally-hard optimization problems?”
Revised: The remainder of this paper seeks to answer the question: “By what mechanism does deep learning solve these emblematic computationally-hard optimization problems?”

Original: A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search,“bouncing around” the loss landscape in the absence of a useful gradient signal.
Revised: A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search, “bouncing around” the loss landscape in the absence of a useful gradient signal.

Original: Upon closer inspection,several empirical observations clash with this hypothesis:
Revised: Upon closer inspection, several empirical observations clash with this hypothesis:

Original: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization, it is unclear how to account for then Θ( k ) runtimes observed in experiments.
Revised: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization, it is unclear how to account for the runtimes observed in experiments, which adapt to the sparsity k .

Original: The initializations,which certainly do not prefer sparse functions 6 , are close to the correct solutions with probability 2 − Ω( n ) ≪ n − k .
Revised: The initializations, which certainly do not prefer sparse functions 7 , are close to the correct solutions with probability 2  ⌦ ( n ) ⌧ n  k .

Original: On larger instances ( n, k ) , the power-law exponents worsen; see Figure 2 (right) ,and the discussion in Appendix C.2.

Original: • No early convergence: Over a large number of random trials, no copies of this randomizedalgorithm get “lucky” (i.e. solve the problem in significantly fewer than the median number ofiterations); see Figure 2 (left) .
Revised: • No early convergence: Over a large number of random trials, no copies of this randomized algorithm get “lucky” (i.e. solve the problem in signiﬁcantly fewer than the median number of iterations); see Figure 2 (left) .

Original: The success times of random exhaustive search would be distributedas Geom(1 / (cid:0) nk (cid:1) ) , whose probability mass is highest at t = 0 and decreases monotonically with t .
Revised: The success times of random exhaustive search would be distributed as Geom(1 /  nk  ) , whose probability mass is highest at t = 0 and decreases monotonically with t .

Original: • Sensitivity to initialization, not stochastic batches: Running these training setups over multiplestochastic batches from a common initialization, we find that loss curves and convergence timesare highly correlated with the architecture’s random  initialization; see Figure 2 (center) . 
Revised: • Sensitivity to initialization, not SGD samples: Running these training setups over multiple stochastic batches from a common initialization, we ﬁnd that loss curves and convergence times are highly correlated with the architecture’s random initialization, and are quite concentrated conditioned on initialization; see Figure 2 (center) . • Elbows in the scaling curves: For larger n , the power-law scaling ceases to hold: the exponent worsens (see Figure 2 (right) , as well as the discussion in Appendix C.2). This would not be true for random exhaustive search.

Original: Even these observations, which do not probe the internal state of the algorithm, suggest that exhaustivesearch is an insufficient picture of the training dynamics, and a different mechanism is at play.
Revised: Even these observations, which do not probe the internal state of the algorithm, suggest that exhaustive search is an insufﬁcient picture of the training dynamics, and a different mechanism is at play.

Original:  Theoretical analyses 
Revised:  Theoretical analyses 4.1 Provable emergence of the parity indices in high-precision gradients

Original: 4.1 Provable emergence of the parity indices in high-precision gradients

Original: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.
Revised: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.

Original: Our main theoretical observation is that, in many cases, the population gradient of the weights atinitialization contains enough “information” for solving the parity problem.
Revised: Our main theoretical observation is that, in many cases, the population gradient of the weights at initialization contains enough “information” for solving the parity problem.

Original: That is, given an accurateenough estimate of the initial gradient (by e.g. computing the gradient over a large enough batchsize), the relevant subset S can be found.
Revised: That is, given an accurate enough estimate of the initial gradient (by e.g. computing the gradient over a large enough batch size), the relevant subset S can be found.

Original: As a warm-up example, consider training a single ReLU neuron f ( x ; w ) =
Revised: As a warm-up example, consider training a single ReLU neuron b y ( x ; w ) =

Original: σ ( w ⊤ x ) w.r.t. thecorrelation loss ℓ ( y, (cid:98) y ) 
Revised: ( w > x ) + with the correlation loss ` ( y, b y ) =

Original: = − y (cid:98) y  , from the initialization w =
Revised:  y b y over D S , from an all-ones initialization w =

Original:  While a single neuron cannotexpress the parity, we observe that the population gradient can indicate what the correct subsetis: E ( x,y ) ∼D S [ ∇ w i ℓ ( y, f ( x ; w ))] =
Revised: 1] 2 R n . While a single neuron cannot express the parity, we observe that the correct subset can be extracted from the population gradient at initialization:

Original: E ( x,y ) ∼D S (cid:2) − yx i σ ′ ( w ⊤ x ) (cid:3) which corresponds to either theorder-

Original: ( k − 1 )
Revised: [Equation]

Original: Fourier coefficient S \ { i } of the function x (cid:55)→ σ ′ ( w ⊤ x ) (if i ∈ S is a relevantcoordinate) or the order- ( k + 1) coefficient S ∪ { i } (if i ̸∈ S is irrelevant). When σ isthe ReLUfunction and w = [1 , . . . , 1] , σ ′ ( w ⊤ x ) = sign ( (cid:80) i x i ) +1is just a shifted majority functionof x .
Revised: The key insight is that each coordinate in the above expression is a correlation between a parity and the function x 7!

Original: The
Revised:  [

Original: Fourier spectrum of majority is well-understood: for even k ,there is a gap between these Fouriercoefficients that is detectable using n O ( k ) samples. This analysis can be further extended to a ReLU neuron initialized with weights w ∈ {± 1 } n and asmall bias.
Revised: P i x i  0] , and thus a Fourier coefﬁcient of this Boolean function.

Original: In fact, we can show that taking a single gradient step with large enough batch size on a ReLU network trained with the hinge-loss ℓ ( y, (cid:98) y ) = max { 1 − y (cid:98) y, 0 } , already finds features that cansolve the parity problem. Using this, we show that training a ReLU MLP with SGD solves the parityproblem:
Revised: At each relevant coordinate ( i 2 S ), the population gradient is the order-( k  1 )

Original: Theorem 4. Fix some ϵ ∈ (0 , 1) , let k be some even number, and assume that n is some odd numbersatisfying n ≥ Ω( k 4 log( nk/ϵ )) . There exist a symmetric random initialization scheme and learningrate and weight decay schedules s.t. for every S ⊆ [ n ] of size k , training a ReLU MLP of sizer
Revised: Fourier coefﬁcient

Original: = O (2 k k log( k/ϵ )) with batch size B = n O ( k ) log 2 ( r/ϵ ) for T = poly( k, r, 1 /ϵ ) iterations on D S w.r.t.
Revised: S \ { i } ; for the irrelevant features ( i 62 S ), it is instead the order- ( k + 1) coefﬁcient S [ { i } .

Original: the hinge loss, finds a network f ( x ; θ t ) with expected loss: E [ ℓ ( f ( x ; θ t ) , y )] ≤ ϵ , wherethe expectation is over the randomness of initialization, training and sampling ( x, y ) ∼ D S . The analysis presented above shows that when the batch size scales with n O ( k ) , SGD over MLP solvesthe parity problem. However, in our experimental setting, the number of steps , and not the batch size, scales with n O ( k ) .
Revised: All we require is a detectable gap between these quantities.

Original: While we believe that running SGD with small batch size and small learning rateessentially amplifies the signal in the population gradient, behaving similarly to performing a largestep over a large batch size, we do not have a complete analysis for training MLPs with SGD in thesmall batch size regime. To complement the above result, in Section 4.2 we analyze the trajectory fora variant of the PolyNet architecture trained with gradient flow. We note that, while the above analysis applies for ReLU MLPs with a specific initialization scheme, asimilar feature emergence phenomenon can be observed in a broader set of architectures and setting. Indeed, for feature emergence to occur, we only require that there is a “gap” between the relevant andirrelevant Fourier coefficients.
Revised: Formally, letting f ( x ; w ) =

Original: Formally, denote (cid:98) f ( S ) :
Revised:  ( w > x ) , letting b f ( S ) :

Original: = E [ f ( x ) χ S ( x )]  the Fourier coefficient of fat S , and observe the following definition:
Revised: = E [ f ( x )  S ( x )] denote the Fourier coefﬁcient of f at S , we isolate the desired property:

Original: Definition 1 (Fourier gap) .
Revised: Deﬁnition 1 (Fourier gap) .

Original: For some function f : {± 1 } n → R andsome subset S  of size k , wesay that f has a γ - Fourier gap at S if (1) for every ( k − 1) 
Revised: For a function f : {± 1 } n ! R and S ✓ [ n ] of size k , we say that f has a  - Fourier gap at S if, for every ( k  1) -element subset S  ⇢ S and ( k + 1) -element superset

Original: -element subset S ′ ⊆ S , it holds that| (cid:98) f ( S ′ )
Revised: S +  S , it holds that | [Equation]

Original: | ≥ γ , and (2) for every subset S ′ ⊆
Revised: For the all-ones initialization, observe [

Original: [ n ] of size k + 1 it holds that | (cid:98) f ( S ′ )
Revised: P

Original: | ≤ γ/ 2 .
Revised: [Equation]

Original: Now, given a network architecture where some neuron has a γ -Fourier gap with respect to the targetsubset S , we can generalize the result of the ReLU neuron.
Revised: is just an afﬁne transformation of the majority function of x , for which a Fourier gap can be established, with  = ⇥ ( n  ( k  1) / 2 ) . This arises from closed-form formulas for the Fourier spectrum of majority (see Lemma 2 in

Original: That is, we show that the subset S can bedetermined by observing an estimate of  the population gradient at initialization:
Revised: Appendix B.1), a landmark result from the harmonic analysis of Boolean functions (Titsworth, 1962; O’Donnell, 2014). Thus, the coordinates in S can be recovered from

Original: Proposition 5.
Revised: [Equation]

Original: Let σ be some activation function and let ℓ be some loss function.
Revised: samples; see Proposition 9 in Appendix B.2 for a formal argument. Carefully extending this insight, we obtain an end-to-end convergence result for ReLU-activation MLP networks with a particular symmetric choice of ± 1 initialization, trained with the hinge loss:

Original: Denote f ( x ; w ) = σ ( w ⊤ x ) .Fix some subset S ⊆
Revised: Theorem 4 (SGD on MLPs learns sparse parities) . Let ✏ 2 (0 , 1) .

Original: [ n ] . Let g ∈ R n  be an estimate of the population gradient suchthat (cid:13)(cid:13)
Revised: Let k  2 an even integer, and let n = ⌦ ( k 4 log( nk/ ✏ )) be an odd integer.

Original: g − E ( x,y ) ∼D S  [ ∇ w ℓ (y, f ( x ; w ))] (cid:13)(cid:13) ∞
Revised: Then, there exist a random initialization scheme, ⌘ t , and  t such that for every S ✓ [ n ] of size k , SGD on a ReLU MLP of width r = ⌦ (2 k k log( k/ ✏ )) , with batch size B = ⌦ ( n k log( n/ ✏ )) on D S with the hinge loss, outputs a network f ( x ; ✓ t ) with expectedloss E [ ` ( f ( x ; ✓ t ) , y )]  ✏ in at most O ( k 3 r 2 n/ ✏ 2 ) iterations.

Original: ≤ γ/ 4 .
Revised: This does not capture the full range of settings in which we empirically observe successful convergence.

Original: Then, for every w s.t. σ ′ ( w ⊤ x ) has a γ -Fouriergap w.r.t.
Revised: First, it requires a sign vector initialization, while we observe convergence with other random initialization schemes (namely, uniform and Gaussian).

Original:  to S and ℓ ′ ( f ( x ; w ) , y ) = − y for all x , the target subset S is detected by g , namely S = { i ∈ [ n ] : | g i | ≥ 3 γ/ 4 } .
Revised: Second, it requires the batch size to scale with n ⌦ ( k ) 9 , while we also obtain positive results when B is small (even B = 1 ).

Original: Comparison with NTK analysis.
Revised: Analogous statements for these cases (as well as other activations and losses) would require Fourier gaps for population gradient functions other than majority; lower bounds on the degree- ( k  1) coefﬁcients ( “Fourier anti-concentration” ) are particularly elusive in the literature, and we leave it as an open challenge to establish them in more general settings. We provide preliminary empirics in Appendix C.1, suggesting that the Fourier gaps in our empirical settings are sufﬁciently large. Low width necessitates feature learning.

Original: In recent years, many theoretical works have studied the behaviorof SGD on neural networks through the lens of the neural tangent kernel(NTK) (Jacot et al., 2018). It istherefore important to highlight the fact that the NTK (or, in fact, any kernel) cannot solve thesparse parity problem.
Revised: We note that in the low-width (non-overparameterized) regimes considered in this work, no ﬁxed kernel (including the neural tangent kernel (Jacot et al., 2018), whose dimensionality is the network’s parameter count) can solve the sparse parity problem.

Original: The followingresult shows that no kernel can achieve small error on the sparseparity problem, unless the size of the kernel is n Ω( k ) : Theorem 6.
Revised: The following is a consequence of results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022): Theorem 5 (Low-width NTK cannot ﬁt all parities) .

Original: LetΨ : {± 1 } n → R D be some D -dimensional embedding with supx ∥ Ψ( x ) ∥ 2 ≤ 1 , and let R > 0 be some number. If DR 2 < ϵ 2 · (cid:0) nk (cid:1) , then there exists some ( n, k ) -parity distribution D S s.t.
Revised: Let : {± 1 } n ! R D be any D -dimensional embedding with sup

Original: [Equation]
Revised: [Equation]

Original: The above is a corollary of the results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022).
Revised: Let R, ✏ > 0 , and let ` denote the 0-1 loss or hinge loss.

Original: 4.2 Disjoint-PolyNet: an idealized architecture for trajectory analyses
Revised: If [Equation] , then there exists some S ✓ [ n ] of size k such that [Equation]

Revised: Thus, our low-width results lie outside the NTK regime, which requires far larger models (size n ⌦ ( k ) ) to express parities.

Revised: However, we note that better sample complexity bounds are possible in the NTK regime, with an algorithm more similar to standard SGD (see (Telgarsky, 2022) and Appendix A.3).

Revised: 4.2 Disjoint-PolyNet: exact trajectory analysis for an idealized architecture

Original: In this section, we present anidealized architecture (a version of PolyNets (xv)) that exhibits similarbehavior to MLPs(experimentally) and is technically easier to analyze.
Revised: In this section, we present an architecture (a version of PolyNets (xv)) which empirically exhibits similar behavior to MLPs and bypasses the difﬁculty of analyzing Fourier gaps.

Original: More specifically, we considerthe disjoint-PolyNet which takes a product over k linear functionswhere the linear functions arecomputed on k disjoint partitions of the input P 1 , . . . , P k with each Pi = { ( i − 1) n ′ + 1 , . ..
Revised: The disjointPolyNet takes a product over k linear functions of an equal-sized 11 partition P 1 , . .

Original: , in ′ }with n ′ = n/k 7 , that is, f ( x ; w 1: k ) : = (cid:81)
Revised: , P k of the input coordinates: f ( x ; w 1: k ) :=

Original:  ki =1  w⊤ i  x P i  .
Revised: Q ki =1 h w i , x P i i .

Original: As noted in the related work section, this isequivalent to a tree parity machine but with real-valued rather than ± 1 output.
Revised: As noted in the Section 1.2, this is equivalent to a tree parity machine, with real-valued (instead of ± 1 ) outputs.

Original: In order for the disjoint-PolyNet to be able toexpress the class of parity problems, we assume thatthe set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to eachdisjoint partition, that is, for alli ∈
Revised: This architecture also requires us to assume that the set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to each disjoint partition, that is, for all

Original:  [ k ] , S ∩ P i = 1 .
Revised: i 2 [ k ] , S \ P i = 1 .

Original: We refer to this problem as the ( n, k ) -disjointparity problem.
Revised: We refer to this problem as the ( n, k ) -disjoint parity problem.

Original: Note that there are still ( n ′ )
Revised: Note that there are still ( n 0 )

Original: k
Revised: k

Original: = ( n/k ) k different possibilities for set S under thisrestriction.
Revised: = ( n/k ) k different possibilities for set S under this restriction.

Original: For fixed k , these represent a constant portion of the(cid:0) nk (cid:1)
Revised: For ﬁxed k , these represent a constant fraction of the

Original: ≈ ( ne/k ) k
Revised: [Equation]

Original: (by Stirling’sapproximation) possibilities for S in the general non-disjoint case.
Revised: Stirling’s approximation) possibilities for S in the general non-disjoint case.

Original: Consider training a disjoint-PolyNet w.r.t.
Revised: Consider training a disjoint-PolyNet w.r.t.

Original: the correlation loss.
Revised: the correlation loss.

Original: WLOG, let S = { 1 , n ′ + 1 , . . . , ( k −1) n ′ + 1 } and e 1 = (1 , 0 , .. 0) .
Revised: Without loss of generality, assume that each relevant coordinate in S is the ﬁrst element P i .

Original: The population gradient is non-zero  at i iff i ∈ S : g i ( w 1: k ) = E [ ∇ w i ℓ ( f ( x ; w 1: k ) , y )] = − E (cid:104) y (cid:16)(cid:81)
Revised: Then, the population gradient is non-zero only at indices

Original: j ̸ = i w ⊤ j x P j (cid:17) x P i (cid:105) = − (cid:16)(cid:81) j ̸ = i w j, 1 (cid:17) e 1 .
Revised: i 2 S :

Original: [Figure]
Revised: [Equation]

Original: Now we consider the gradient flow dynamics ofdisjoint-PolyNet, which provide a mathematicallytractable case study for the trajectory of sparse parity learning.
Revised: This allows us to analyze the gradient ﬂow dynamics of the disjoint-PolyNet, without needing to establish Fourier gaps.

Original: For each i ∈ [ k ] , in this sectionwe treat w i as a function from R [-≥ 
-]
Revised: For each i 2 [ k ] , in this section we treat w i as a function from R [-≥ 
-] 0 !

Original: → R n ′ which satisfies the following differential equation:˙ w i = − g i ( w 1: k ( t )) .
Revised: R nwhich satisﬁes the following differential equation: ˙ w i =  g i ( w 1: k ( t )) .

Original: For clarity of exposition, assume all-ones initialization.
Revised: For clarity of exposition, assume all-ones initialization.

Original: Then all of the relevantweights { w i, 1 : i ∈ [ k ] } follow the same [-trajectory, which we denote by v : R ≥ 
 → R .-]
Revised: Then, all of the relevant weights { w i, 1 : i 2 [ k ] } follow the same [-trajectory, which we denote by v : R ≥ 
 → R .-]trajectory.

Original: By analyzing the resulting differential equations, we can formally exhibit “phase transition”-likebehavior in the fully deterministic gradient flow setting.
Revised: By analyzing the resulting differential equations, we can formally exhibit “phase transition”-like behavior in the fully deterministic gradient ﬂow setting.

Original: Theorem 7 (Gradient flow on disjoint-PolyNets) .
Revised: Theorem 6 (Loss plateau for gradient ﬂow on disjoint-PolyNets) .

Original: Suppose k ≥ 3 .
Revised: Suppose k  3 .

Original: Let T ( . 49) be the  time it takes forerror to fall below . 49 , and let T (0) be the time it takes to reach zero error.
Revised: Let T ( ✏ ) denote the smallest time at which the error is at most ✏ .

Original: Then
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: In other words, the network takes much longer to reachslightly betterthan trivial accuracy than ittakes togo from slightly better than trivial to perfect accuracy.
Revised: Informally, the network takes much longer to reach better trivial accuracy than takes to trivial the trajectory of disjoint-PolyNets trained online size

Original: We can also analyze the trajectory of disjoint-PolyNets trained with online SGD, confirming that aneural network trained with batch size 1 SGD can learn k -sparse parities within n O ( k ) iterations.

Original: Theorem 8 (SGD on disjoint-PolyNets) .
Revised: Theorem 7 (SGD on disjoint-PolyNets learns disjoint parities) .

Original: Suppose we train the disjoint-PolyNet, initialized as above,with online SGD with any batch size B ≥ 1 .
Revised: Suppose we train a disjoint-PolyNet, initialized as above, with online SGD.

Original: Then there exists an adaptive learning rate schedulesuch that for any ϵ > 0 , with probability .99, the error falls below ϵ within [Equation] steps.
Revised: Then there exists an adaptive learning rate schedule such that for any ✏ > 0 , with probability 0.99, the error falls below ✏ within ˜ O  ( n 0 ) (2 k  1) log(1 / ✏ )  steps.

Original: Extended versions of these theorems, along with proofs, can be found in Appendix B.3.
Revised: Extended versions of these theorems, along with proofs, can be found in Appendix B.3.

Original:  Hidden progress: discussion and additional experiments
Revised:  Hidden progress: discussion and additional experiments

Original: In this section, we advocate for sparse parities as an idealized testbed for understanding algorithmsand phenomena in modern deep learning. These are accompanied by experimental vignettes whichare auxiliary to the core results from Section 3, with more systematic studies deferred to future work.
Revised: So far, we have shown that sparse parity learning provides an idealized setting in which neural networks successfully learn sparse combinatorial features, with a mechanism of continual progress hiding behind discontinuous training curves. In this section, we outline preliminary explorations on a broader range of interesting phenomena which arise in this setting.

Original: Details are given inAppendix D.
Revised: Details are provided in

Revised: Appendix C, while more systematic investigations are deferred to future work.

Original: A progress measure for parity.
Revised: Hidden progress measures for learning parities.

Original: To begin, the black-box experiments in Section 3 suggest thatrandom search is the incorrect model of SGD’s behavior in this setting.
Revised: The theoretical and (black-box) empirical results suggest that SGD does not learn parities via the memoryless process of random exhaustive search.

Original: Using the theoretical insight that amplifying a precise initial population gradient suffices for learning parities, we construct onepossible progress measure, which is a function of the sequence of weights w 0 , . . .
Revised: This suggests the existence of progress measures : scalar quantities which are functions of the training algorithm’s state (i.e. the model weights w t ) and are predictive of the time to successful convergence. We provide some white-box investigations which further support the hypothesis of hidden progress, by examining the gradual improvement in quantities other than the training loss. In Appendix C.1, we directly plot the Fourier gaps of the population gradient, as a function of t , ﬁnding that they are large (within a small constant factor of those of majority) in practice.

Original: w t ∈ R n so far:ρ ( w 0: t ) :
Revised: In Figure 3 and Appendix C.3, we examine the weight movement norm ⇢ ( w 0: t ) :

Original: =
Revised: =

Original: ∥ w t − w 0 ∥ ∞ , using the fact that w t − w 0 is an estimate for the initial population gradientin the linearized setting.
Revised: k w t  w 0 k 1 to reveal hidden progress, motivated by the fact that w t  w 0 is a linearized estimate for the initial population gradient.

Original: Figure 3 shows how gradual weight movement (and thus, progress) can behidden behind plateauing losses.

Original: Roles of overparameterization vs. oversampling .
Revised: Roles of overparameterization vs. oversampling .

Original: An interesting consequence of our analysisis that it illuminates scaling behaviors with respect to a third fundamental resource parameter: modelsize , which we study in terms of network width r .
Revised: An interesting consequence of our analysis is that it illuminates scaling behaviors with respect to a third fundamental resource parameter: model size , which we study in terms of network width r .

Original: If SGD operated by a “random search” mechanism,one would expect width to provide a parallel speedup.
Revised: If SGD operated by a “random search” mechanism, one would expect width to provide a parallel speedup.

Original: Instead, we  SGD sequentially amplifiesprogress.
Revised: Instead, we ﬁnd that SGD sequentially ampliﬁes progress.

Original: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD doesnot give (1 /r ) × speedups; more directly, in  Figure 4 (left) , convergence times for sparse paritiesempirically plateau at large model sizes.
Revised: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD does not give (1 /r ) ⇥ speedups; more directly, in Appendix C.4 (previewed in Figure 4 (left) ), we ﬁnd that convergence times for sparse parities empirically plateau at large model sizes.

Original: It is a significant challenge to understand the interactions between network depth and computation,and largely outside the scope of this work.

Original: However, in Appendix C.7, we provide a brief note onusing parities and polynomial-activation MLPs to construct a simple counterexample to the “deeponly works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating acase where a deep network can get near-perfect accuracy even when greedy layerwise training (e.g.(Belilovsky et al., 2019)) cannot beat trivial performance.

Original: Learning and grokking in the finite-sample multi-pass setting.
Revised: Emergence of grokking in the ﬁnite-sample (multi-pass) setting.

Original: The maintheoretical and empirical results  in this work consider online learning algorithms which couple the resources of training timeand independent samples. However, due to the computational-statistical gap in parity learning, thesepositive results are suboptimal in terms of sample efficiency.
Revised: Our main results are presented in the online learning setting (fresh minibatches from D S at each iteration). While this mitigates the confounding factor of overﬁtting, it couples the resources of training time and independent samples in a suboptimal way, due to the computational-statistical gap for parity learning.

Original: We find that minibatch SGD (with weightdecay) can empirically solve sparse parities, even from a sample of size m ≪ n k .
Revised: In Appendix C.5, we ﬁnd empirically that minibatch SGD (with weight decay) can learn sparse parities, even with smaller sample sizes m ⌧ n k .

Original: For small values ofm , we reliably observe the grokking phenomenon (Power et al., 2022): overfitting for along time, thena delayed phase transition for the generalization error; see the two center panels of Figure 4 (right) .
Revised: We reliably observe the grokking phenomenon (Power et al., 2022): an initial overﬁtting phase, then a delayed phase transition in the generalization error; see the two center panels of Figure 4 (right) .

Original: Conclusion
Revised: These results complement and corroborate the ﬁndings of Nanda and Lieberum (2022), who analyze the hidden progress of Transformers trained on arithmetic tasks (a setting which also exhibits grokking). Deeper networks.

Original: This work puts forward parity learning as a stylized test case to explore some of the puzzling featuresof the role of computational (as opposed to statistical) resources in deep learning.
Revised: It is a signiﬁcant challenge (and generally outside the scope of this work) to understand the interactions between network depth and computational/statistical efﬁciency.

Original: These includediscontinuous improvements (a.k.a.
Revised: In Appendix C.7, we show that learning parities with deeper polynomial-activation MLPs comprises a simple counterexample to the “deep only works if shallow is good” principle of Malach and Shalev-Shwartz (2019): a deep network can get near-perfect accuracy, even when greedy layer-wise training (e.g. (Belilovsky et al., 2019)) cannot beat trivial performance. By providing positive theory and empirics which elude these simpliﬁed explanations of SGD, we hope to point the way to a more complete understanding of learning dynamics in the challenging cases where no apparent progress is made for extended periods of time.

Original: emergent capabilities ), feature learning, and universality ofarchitectures.
Revised: This work puts forward sparse parity learning as an elementary test case to explore the puzzling features of the role of computational (as opposed to statistical) resources in deep learning.

Original: In particular, we show thatdeep learning on parities exhibits a phase transition behavior,that it is successfully learned by a variety of deep-net architectures, and that this  success cannot beexplained as a “random exhaustive search”, nor through  frameworks such as the neural tangent kernelor layer-by-layer learning.
Revised: In particular, we have shown that a variety of neural architectures solve this combinatorial search problem, with a number of computational steps scaling near the sparsity-dependent SQ lower bound; furthermore, the loss curves exhibit phase transitions . This success cannot be explained by a model of random exhaustive search, nor through simpliﬁed frameworks such as the neural tangent kernel or layer-by-layer learning.

Original: However, there are more experimental and theoretical questions, even for this simplified case of paritylearning.
Revised: Even in this simple setting, there are several open experimental and theoretical questions.

Original: Our focus in this work was on the online learning case, where training time and samplesarise in tandem.
Revised: Our focus in this work was on the online learning case, where training time and samples arise in tandem.

Original: However, we believe it would be instructive to investigate parity learning when threeresources of samples, time, and model size are scaled separately.
Revised: We believe it would be instructive to investigate parity learning when three resources of samples, time, and model size are scaled separately.

Original: Some very preliminary findingsalong these lines are presented in Section 3.
Revised: Some very preliminary ﬁndings along these lines are presented in Section 3.

Original: Extending our theoretical results to the small-batch setting, as well as to more architectures, is anopen problem.
Revised: Extending our theoretical results to the small-batch setting, as well as to more architectures, is an open problem.

Original: Resolving it would require a better understanding of anti-concentration(lower boundon deviation from mean) of Fourier coefficients, a phenomenon that is much less studied than theconcentration of these coefficients.
Revised: Resolving it would require a better understanding of anti-concentration of Fourier coefﬁcients, a phenomenon that is much less studied than the concentration of these coefﬁcients.

Original: We would also want to extend the analysis beyond parities totasks that are not aligned with the elementary basis such as low-rank tensor recovery. Another important follow-up direction is understanding the extent that these insights extend fromparity learning to real-world problems, as well as the extent into which non-synthetic tasks (in,e.g., natural language processing and program synthesis) embed within them parity-like subtasks ofexhaustive combinatorial search. 
Revised: Another important follow-up direction is understanding the extent to which these insights extend from parity learning to more complex (including real-world) combinatorial problem settings, as well as the extent to which non-synthetic tasks (in, e.g., natural language processing and program synthesis) embed within them parity-like subtasks of exhaustive combinatorial search. We hope that our results will lead to further progress towards understanding and improving the optimization dynamics behind the recent slew of dramatic empirical successes of deep learning in these types of domains.

Original: Broader impact.
Revised: Broader impact.

Original: This work seeks to contribute to the foundational understanding of computationalscaling behaviors in deep learning.
Revised: This work seeks to contribute to the foundational understanding of computational scaling behaviors in deep learning.

Original: Our theoretical and empirical analyses are in a heavily-idealizedsynthetic problem setting.
Revised: Our theoretical and empirical analyses are in a heavily-idealized synthetic problem setting.

Original: Hence, we see no direct societal impacts of the results in this study.
Revised: Hence, we see no direct societal impacts of the results in this study.

Revised: Acknowledgements.

Revised: We would like to thank Lenka Zdeborová for providing us with references to the statistical physics literature on phase transitions in the learning curves of neural networks, and

Revised: Matus Telgarsky for bringing to our attention the better sample complexity guarantees of 2-sparse parity learning in the NTK regime.

Revised: Sham Kakade acknowledges funding from the Ofﬁce of Naval

Revised: Research under award N00014-22-1-2377.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?[Yes]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?[Yes]
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original:  (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable?

Original: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] [Table] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]