Original: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Revised: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Why is it that so many novel capabilities in  deep learning are unlocked by scalingup datasets, model sizes, and training times?
Revised: There is mounting empirical evidence of emergent phenomena in the capabilitiesof deep learning methods as we scale up datasets, model sizes, and training times.

Original: While there are some accounts ofhow these factors modulate statistical capacity, far less is known about how thesefactors affect the computational problem of model training.
Revised: While there are some accounts of how these resources modulate statistical capacity,far less is known about their effect on the computational problem of model training.

Original: This work exploresthese computational issues through the lens of learning k-sparse parities of nbits, a canonical family of problems which pose  computational barriers.
Revised: This work conducts such an exploration through the lens of learning k -sparseparities of n bits, a canonical family of problems which pose theoretical computational barriers.

Original: Perhapssurprisingly, standard deep learning approaches, when applied to this problem,already exhibit a number of discontinuous impacts of increasing resources (suchas dataset size, training time, or model size) on model performance, making it anatural candidate for rigorously studying these phase transitions.
Revised: In this setting, we find that neural networks exhibit surprisingphase transitions when scaling up dataset size and running time.

Original: In particular, weempirically demonstrate that with standardinitialization and training, a varietyofarchitectures learn sparse parities in a number of iterations that nearly matchesoptimal lower bounds (in the SQ learning model), even without an explicit sparsitypromoting prior.
Revised: In particular, wedemonstrate empirically that with standard training, a variety of architectures learnsparse parities with n O ( k ) examples, with loss (and error) curves abruptly droppingafter n O ( k ) iterations. These positive results nearly match known SQ lower bounds,even without an explicit sparsity-promoting prior.

Original: Furthermore, we elucidate the mechanisms of these positive resultswith a theoretical analysis: we argue that the phase transition in performance isnot due to SGD "stumbling in the dark" (with a Langevin-like mechanism) until itfinds the hidden set of features; instead, we show that SGD gradually amplifies a Fourier gap(the hidden progress measure) in the population gradient.
Revised: We elucidate the mechanisms ofthese phenomena with a theoretical analysis: we find that the phase transition inperformance is not due to SGD “stumbling in the dark” until it finds the hidden setof features (a natural algorithm which also runs in n O ( k ) time); instead, we showthat SGD gradually amplifies a Fourier gap in the population gradient.

Original:  Introduction
Revised:  Introduction

Original: Neural networks perform better with more resources (data, model size, training time), but differenttasks exhibit qualitatively different dependencies of performance on resources.
Revised: Neural networks perform better with more resources (data, model size, training time) but differenttasks exhibit qualitatively different dependencies of performance on resources.

Original: In particular, whilemany learning tasks exhibit continuous improvement in performance with increasing resources, othercases show discontinuous improvement, where a capability emerges at a certain threshold .
Revised: In particular, whilemany learning tasks exhibit continuous improvement in performance with increasing resources, othercases show discontinuous improvement, where a capability emerges at a certain threshold.

Revised: Through astatistical lens, it is well-understood that larger models, trained with more data, can fit more complexand expressive functions.

Revised: However, far less is known about the analogous computational question:how does the scaling of these resources influence the success of gradient-based optimization fortraining these models?

Original: Thesephase transitions cannot be explained via statistical capacity alone.
Revised: These emergent phase transitions cannot be explained via statistical capacity alone.

Original: Indeed, in many cases we seea phase transition even when the amount of data remains fixed, with only model size or trainingtime increasing.
Revised: In many cases wesee a phase transition even when the amount of data remains fixed, with only model size or trainingtime increasing.

Original: A timely example is the emergence of reasoning and few-shot learning capabilitieswhen scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al.
Revised: A timely example is the emergence of reasoning and few-shot learning capabilitieswhen scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022;

Original: , 2022),including computational resource only (Hoffmann et al., 2022).
Revised: Hoffmann et al., 2022).

Original: Power et al.
Revised: Power et al.

Original: (2022) give examplesexhibiting discontinuous improvements in population accuracy (“grokking”) when running timeincreases, while data and model size remain fixed.
Revised: (2022) give examples exhibiting discontinuous improvements inpopulation accuracy (“grokking”) when running time increases, while data and model size remainfixed.

Original: In this work, we analyze the computational aspect of scaling in deep learning in an idealized problemsetting which already exhibits discontinuous improvements.
Revised: In this work, we analyze the computational aspect of scaling in deep learning in a simple syntheticsetting which already exhibits discontinuous improvements.

Original: Specifically, we consider the sparseparity problem — where the concept is the parity  of k ≪ n bits of an n -dimensional input.
Revised: Specifically, we consider the sparseparity problem — where the label is the parity (XOR) of k ≪ n bits in a random length- n binary string.

Original: This is acanonical problem that is computationally difficult for a range of algorithms, including gradient-basedmethods (Kearns, 1998) and streamingalgorithms (Kol et al., 2017).
Revised: This is a canonical problem which is computationally difficult for a range of algorithms, including [Figure] , 1998) and streaming (Kol et al., 2017) algorithms.

Original: We focus on analyzing theresource measure of training time , and demonstrate that the loss curves for sparse parities display a [Figure] phase transition across a variety of architectures and hyperparameters (see Figure 1, left).
Revised: We focus on analyzingthe resource measure of training time , and demonstrate that the loss curves for sparse parities displaya phase transition across a variety of architectures and hyperparameters (see Figure 1, left).

Original: A strikingobservation is that SGD robustly finds the sparse subset (and hence, effectively, reaches 0 error)with a variety of activation functions, initialization schemes,and, as shown later on, even with noover-parameterization.
Revised: A strikingobservation is that SGD robustly finds the sparse subset (and hence, effectively, reaches 0 error) witha variety of activation functions and initialization schemes, even with no over-parameterization.

Original: This robust convergence is the starting point for our investigation.
Revised: Thisrobust convergence is the starting point for our investigation.

Original: Perhaps the most natural hypothesis to explain SGD’s success in learning parities is that it simply“stumbles in the dark”, essentially performing random search for the unknown target (e.g. throughsome stochastic  Langevin dynamics).
Revised: Perhaps the most natural hypothesis to explain SGD’s success in learning parities would be that itsimply “stumbles in the dark”, essentially performing random search for the unknown target (e.g.via stochastic gradient Langevin dynamics).

Original: If that was the case, we may expect to observe a convergencetime of 2 O ( n ) (which is consistent that of a naive parameter search).
Revised: If that were the case, we might expect to observe aconvergence time of 2 Ω( n ) , like a naive search over parameters or subsets of indices.

Original: However, Figure 1, right,already provides some evidence against this “random search” hypothesis. This figure shows that theconvergence time is closer to an n O ( k ) scaling.
Revised: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: theconvergence time is closer to an n O ( k ) scaling.

Original: Notably, such a convergence rate implies  SGD iscloser to achieving the optimal computation time among a natural class of algorithms, namely SQ(statistical query) algorithms.
Revised: Notably, such a convergence rate implies that SGD iscloser to achieving the optimal computation time among a natural class of algorithms, namely SQ(statistical query) algorithms.

Original: In this work, through an extensive empirical analysis of the scaling behavior of a variety of models,as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.
Revised: In this work, through an extensive empirical analysis of the scaling behavior of a variety of models,as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.

Original: Instead, there is a hidden progress measurewhich under  SGD is steadily improving.
Revised: Instead, there is a hidden progress measure under which SGD is steadily improving.

Original: Furthermore,and perhaps surprisingly, we also give empirical and theoretical evidence that SGD is achieving acomputational runtimethat is much closer to the optimal SQcomputational lower bound than simplydoing (random) parameter search.
Revised: Furthermore, andperhaps surprisingly, we show that SGD achieves a computational runtime much closer to the optimal SQ lower bound than simply doing (non-sparse) parameter search.

Original: More generally, our theoretical and empirical investigations reveala number of notable phenomena regarding the dependence of SGD’s performance on resources, andwe provide further theoretical and empirical evidence of phase transitions with data, model size, andtraining time.
Revised: More generally, our theoreticaland empirical investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources, and we provide further theoretical and empirical evidence of phasetransitions with data, model size, and training time.

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: SGD efficiently learns sparse parities, in theory and practice.
Revised: SGD efficiently learns sparse parities, in theory and practice.

Original: It is known  that (a noisy SQabstraction of) SGD on any architecture requires at least n Ω( k ) computational steps to learn k -sparsen -dimensional parities (for background, see Appendix A).
Revised: It is known from SQ lowerbounds that (noisy) gradient descent on any architecture requires at least n Ω( k ) computational stepsto learn k -sparse n -dimensional parities (for background, see Appendix A).

Original: However, with standard architectures andinitialization, which do not  encode the sparsity prior, one may expect SGD’s performance to be muchworse, on the order of 2 Ω( n ) .
Revised: However, with standardarchitectures and initialization, which do not explicitly encode a sparsity prior, one may expect SGD’sperformance to be much worse, on the order of 2 Ω( n ) .

Original: We give extensive empirical evidence that this is not the case.
Revised: We give extensive empirical evidence that thisis not the case.

Original: Empirical Finding 1.
Revised: Empirical Finding 1.

Original: For all small instances ( n ≤ 30 , k ≤ 4 ) of the sparse parity problem,architectures A ∈ { 2-layer MLPs, Transformers 1 , single neurons with ReLU or oscillating activations, PolyNets 2 } , and initialization in { uniform, Gaussian, Bernoulli } , andfor all batch sizesin{ 1  , . .
Revised: For all small instances ( n ≤ 30 , k ≤ 4 ) of the sparse parity problem,architectures A ∈ { 2-layer MLPs, Transformers 1 , sinusoidal (and other non-standard) neurons, PolyNets 2 } , initializations in { uniform, Gaussian, Bernoulli } , and batch sizes 1 ≤ B ≤ 1024 , SGD

Original: . , 1024 } , SGD on A solves the ( n, k ) -sparse parity problem with probability ≥ 1 / 2  within atmost c · n αk steps, for small architecture-dependent constants c, α .
Revised: With on A solves the ( n, k ) -sparse parity problem (w.p. ≥ 0 . 2 ) within at most c · n αk steps, for smallarchitecture-dependent constants c, α .

Original: For a subset of these architectures, we performed more extensive training, to show scaling behaviorsof the computation time. 3 See Figure 1, right.
Revised: For a subset of these architectures, we performed more extensive training, to show scaling behaviorsof the computation time; 3 see Figure 1 (right) and Figure 10 in the appendix.

Original: The above results indicate that SGD succeeds in solving parities much faster than the 2 Ω( n ) stepsthat would be required by random search without a sparse prior.
Revised: The above results indicate that SGD succeeds at solving parities much faster than the 2 Ω( n ) steps thatwould be required by random search without a sparse prior.

Original: This suggests that despiteprogressseeming flat in thefirst phase of learning, SGD does in fact make progress “under the hood”.
Revised: This suggests that despite flat loss andaccuracy curves before the phase transition, SGD makes progress “under the hood”.

Original: Weshow that this is indeed the case by coming up with a progress measure that continuously improvesthroughout training.
Revised: We show thatthis is indeed the case by coming up with a progress measure that continuously improves throughouttraining.

Original: We can explain the loss and accuracy curves using this progress measure, byshowing that they typically rise sharply once the measure passes a certain threshold.
Revised: We can explain the loss and accuracy curves using this progress measure, by showing thatthey typically rise sharply once the measure passes a certain threshold.

Original: Theoretical analysis.
Revised: Theoretical Analysis.

Original: Our empirical results suggest that SGD is able to do feature learning for theparity problem, and identify the influential coordinates, even without a sparsity-inducing prior.
Revised: Our empirical results suggest that, in a number of computational stepsmatching the SQ limit, SGD is able to solve the parity problem and identify the influential coordinates,without an explicit sparse prior.

Original: Wegive a theoretical analysis that validates this claim.
Revised: We give a theoretical analysis which validates this claim.

Original: For MLPs, we prove that the population gradientcontains enough signal on the influential coordinates to enable learning in the large-batch limit.

Original: Informal Theorem 2.
Revised: Informal Theorem 2.

Original: For 2-layer MLPs, SGD in the large batch limit learns the relevant featuresusing n O ( k ) samples.
Revised: On 2-layer MLPs of width 2 Θ( k ) , and with batch size n O ( k ) , SGD convergeswith high probability to a solution with at most ϵ error on the ( n, k ) -parity problem in at most2 O ( k ) · poly(1 /ϵ ) iterations.

Original: For an idealized architecture (which we calldisjoint-PolyNet), we theoretically explain the qualitativebehavior of the error curves.
Revised: We also present a stronger analysis for an idealized architecture (which we call the disjoint-PolyNet ),which allows for any batch size, and captures the phase transitions observed in the error curves.

Original: Informal Theorem 3.
Revised: Informal Theorem 3.

Original: Gradient flow ondisjoint-PolyNet exhibits two properties: (1) early progressis more visible in the weights than in the error, and (2) for a large fraction of the total running time(until perfect accuracy), accuracy remains nearly trivial.
Revised: On disjoint-PolyNets, SGD (with any batch size B ≥ 1 ) converges with highprobability to a solution with at most ϵ error on the ( n, k ) -parity problem in at most n O ( k ) · log(1 /ϵ )iterations. Continuous-time gradient flow exhibits a phase transition: it spends a 1 − o (1) fraction ofits time before convergence with error ≥ 49% .

Original: In addition to refuting a “random search” model, our work also poses a counterexample to othermodels for the computational mechanisms of deep learning; for instance, it provides a setting wheredeep neural nets successfully learn a concept to 100% accuracy while the corresponding Neural Tangent Kernel (NTK) only achieves trivial performance, hence showing the importance of featurelearning .
Revised: In addition to refuting the alternative “random search” hypothesis, our work also poses a counterexample to other models for the computational mechanisms of deep learning; for instance, itprovides a setting where deep neural nets successfully learn a concept to 100% accuracy while thecorresponding Neural Tangent Kernel (NTK) only achieves trivial performance, hence showing theimportance of feature learning .

Original: We also construct a counterexample to the “deep only works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating a case where a deep network can get near-perfectaccuracy even when greedy layerwise training (a la Belilovsky et al.
Revised: We also construct a counterexample to the “deep only works ifshallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating a case where adeep network can get near-perfect accuracy even when greedy layerwise training (e.g. Belilovskyet al.

Original: (2019)) cannot beat trivialperformance.
Revised: (2019)) cannot beat trivial performance.

Original: By providing positive theory and empirics which elude these simplified explanationsof SGD, we hope to point the way to a more complete understanding of learning dynamics in thechallenging cases where no apparent progress is made for extended periods of time.
Revised: By providing positive theory and empirics which eludethese simplified explanations of SGD, we hope to point the way to a more complete understanding oflearning dynamics in the challenging cases where no apparent progress is made for extended periodsof time.

Original: 1.2 Related work
Revised: 1.2 Related work

Original: We present the most directly related work below, on feature learning and learning parities with neuralnets.
Revised: We present the most directly related work below, on feature learning, learning parities with neuralnets, and the statistical mechanics of machine learning.

Original: A broader discussion of related work can be found in Appendix A.3.
Revised: A broader discussion of related work can befound in Appendix A.3.

Original: SGD and feature learning.
Revised: SGD and feature learning.

Original: Theoretical analysis of gradient descent over neural-networks is notoriously hard, due to the non-convex nature of the optimization problem.
Revised: Theoretical analysis of gradient descent over neural networks isnotoriously hard, due to the non-convex nature of the optimization problem.

Original: That said, it has beenestablished that in some settings, the dynamics of GD keep the weights close to their initialization,thus behaving likea convex optimization over the Neural Tangent Kernel (see for example Jacot et al.(2018);
Revised: That said, it has beenestablished that in some settings, the dynamics of GD keep the weights close to their initialization,thus behaving like convex optimization over the Neural Tangent Kernel (see, for example, Jacot et al.(2018);

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: Du et al.
Revised: Du et al.

Original: In contrast, it has been shown that in various tasks,moving away from the fixed features of the NTK is essential for the success  neural networks trainedwith GD (for example Yehudai and Shamir (2019); Allen-Zhu and Li (2019);
Revised: In contrast, it has been shown that in various tasks,moving away from the fixed features of the NTK is essential for the success of neural networks trainedwith GD (for example Yehudai and Shamir (2019); Allen-Zhu and Li (2019);

Original: Wei et al.
Revised: Wei et al.

Original: (2019) andthe review in Malach et al.
Revised: (2019) andthe review in Malach et al.

Original: These results demonstrate that feature learning is an importantpart of the GD optimization process.
Revised: These results demonstrate that feature learning is an importantpart of the GD optimization process.

Original: Our work also focuses on a setting where feature learning isessential for the target task.
Revised: Our work also focuses on a setting where feature learning isessential for the target task.

Original: In our theoretical analysis, we show that the initial population gradientencodes the relevant features for the problem.
Revised: In our theoretical analysis, we show that the initial population gradientencodes the relevant features for the problem.

Original: The importance of the first gradient step for featurelearning has been recently studied in Ba et al.
Revised: The importance of the first gradient step for featurelearning has been recently studied in (Ba et al., 2022).

Original: Learning parities with neural networks.
Revised: Learning parities with neural networks.

Original: The problem of learning parities using neural networkshas been investigated in prior works from various perspectives.
Revised: The problem of learning parities using neural networkshas been investigated in prior works from various perspectives.

Original: It has been demonstrated that paritiesare hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartzet al., 2017; Abbe and Sandon, 2020).
Revised: It has been demonstrated that paritiesare hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartzet al., 2017; Abbe and Sandon, 2020).

Original: One possible approach for overcoming the computationalhardness is to make favorable assumptions on the input distribution.
Revised: One possible approach for overcoming the computationalhardness is to make favorable assumptions on the input distribution.

Original: Indeed, recent works show thatunder various assumptions on the input distribution, neural networks can be efficiently trained to learn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;
Revised: Indeed, recent works show thatunder various assumptions on the input distribution, neural networks can be efficiently trained tolearn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;

Original: Frei et al., 2022; Malach et al.,2021).
Revised: Frei et al., 2022; Malach et al.,2021).

Original: In contrast to these results, this work takes the approach of intentionally focusing on a hardbenchmark task, without assuming that the distribution has some favorable (namely, non-uniform)structure.
Revised: In contrast to these results, this work takes the approach of intentionally focusing on a hardbenchmark task, without assuming that the distribution has some favorable (namely, non-uniform)structure.

Original: This setting allows us to probe the performance of deep learning at a known computationallimit.
Revised: This setting allows us to probe the performance of deep learning at a known computationallimit.

Original: Notably, the work of Andoni et al.
Revised: Notably, the work of Andoni et al.

Original: (2014) provides analysis for learning polynomials (and inparticular, parities) under the uniform distribution.
Revised: (2014) provides analysis for learning polynomials (and inparticular, parities) under the uniform distribution.

Original: However, their main results require a networkof size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical andempirical evidence for the success of smaller networks.
Revised: However, their main results require a networkof size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical andempirical evidence for the success of smaller networks.

Original: Studying a related subject, some works haveshown that neural networks display a spectral bias, learning to fit low-frequency coefficients beforehigh-frequency ones (Rahaman et al., 2019; Cao et al., 2019). 
Revised: Studying a related subject, some works haveshown that neural networks display a spectral bias, learning to fit low-frequency coefficients beforehigh-frequency ones (Rahaman et al., 2019; Cao et al., 2019). Statistical mechanics of machine learning.

Revised: An extensive body of work originating in the statisticalphysics community has studied phase transitions in the learning curves of neural networks (Gardnerand Derrida, 1989;

Original: Preliminaries
Revised: Watkin et al., 1993; Engel and Van den Broeck, 2001). These works typicallyfocus on student-teacher learning in the “thermodynamic limit”, in which the number of trainingexamples is α times larger than the input dimension and both are taken to infinity. One of theclassic toy architectures analyzed in this literature is the parity machine (Mitchison and Durbin,1989; Hansel et al., 1992; Opper, 1994; Kabashima, 1994; Simonetti and Caticha, 1996). In ourwork, we introduce PolyNets, a variant of parity machines in which the output is real-valued ratherthan ± 1 ; and we theoretically analyze disjoint-Polynets, which are the real-output analogue of theoft-considered parity machines with tree architecture.

Revised: While much of the statistical mechanics of

Revised: ML literature focuses on an idealized training limit in which the weights reach a Gibbs distributionequilibrium, there is a strand of the literature that aims to characterize the trajectory of SGD trainingin the high-dimensional limit with constant-sized sets of ordinary differential equations (Saad and

Revised: Solla, 1995b,a; Goldt et al., 2019).

Revised: These papers discuss cases, including problems that share aspectswith 2-sparse parities Refinetti et al.

Revised: (2021), where the network gets stuck in (and then escapes from)a plateau of suboptimal generalization error.

Revised: Recently, Arous et al.

Revised: (2021) studied (for rank-oneparameter estimation problems) the relative amount of time spent by SGD in an initial high-error“search” phase versus a final “descent” phase, which is reminiscent of the framing of Theorem 7.

Revised: However, to our knowledge prior work has not shown k -sparse parities can be learned with a numberof iterations that nearly matches known lower bounds, nor has it specifically studied phase transitionsin k -sparse parity learning during gradient descent.

Original: We define necessary notation here; see Appendix A for more background and technical ingredients.
Revised: We define necessary notation here; see Appendix A for more background and technical ingredients.

Original: Sparse parities.
Revised: Sparse parities.

Original: For integer n ≥ 1 and non-empty S ⊆ [ n ] , the ( n, S ) - parity function χ S : {± 1 } n →{± 1 } is defined as χ S ( x ) = (cid:81) 
Revised: For integer n ≥ 1 and non-empty S ⊆ [ n ] , the ( n, S ) - parity function χ S :{± 1 } n → {± 1 } is defined as χ S ( x ) = (cid:81) i ∈ S x i .

Original: i ∈ S x i . We define the ( n, S ) - parity distribution D S as the distributionover ( x, y = χ S ( x )) 4 where x is drawn from the uniform distribution over {± 1 } n , which we denoteby Unif( {± 1 } n ) .
Revised: We define the ( n, S ) - parity distribution D S asthe distribution over ( x, y = χ S ( x )) 4 where x is drawn from the uniform distribution over {± 1 } n ,which we denote by Unif( {± 1 } n ) .

Original: We define the ( n, k ) - parity learning problem as the task of recovering S usingsamples from D S , where S is chosen at random in (cid:0) [ n ] k (cid:1) .
Revised: We define the ( n, k ) - parity learning problem as the task ofrecovering S using samples from D S , where S is chosen at random in (cid:0) [ n ] k (cid:1) .

Original: Statistically, it is possible to do so usingΘ(log (cid:0) nk (cid:1) )
Revised: Statistically, it is possibleto do so using Θ(log (cid:0) nk (cid:1) )

Original: ≈ k log n samples.
Revised: ≈ k log n samples.

Original: However, in the statistical query (SQ) model (Kearns, 1998), (whichhas been shown to encapsulate gradient-based methods such as GD or SGD (Abbe et al., 2021)), thistask requires Ω( n k ) queries (assuming constant level of noise).
Revised: However, in the statistical query (SQ) model (Kearns,1998), (which has been shown to encapsulate gradient-based methods such as GD or SGD (Abbeet al., 2021)), this task requires Ω( n k ) queries (assuming constant level of noise).

Original: While learning noiseless paritiescan be solved by Gaussian elimination using O ( n ) samples, learning sparse noisy parities, even at avery small noise level (i.e., o (1) or n − δ ) is believed to inherently require n Ω( k ) computational stepsand samples, or exponential computation with n o ( k ) samples
Revised: While learningnoiseless parities can be solved by Gaussian elimination using O ( n ) samples, learning sparse noisyparities, even at a very small noise level (i.e., o (1) or n − δ ) is believed to inherently require n Ω( k )computational steps and samples, or exponential computation with n o ( k ) samples.

Original: This was first  explicitly conjecturedby Alekhnovich (2003), and has been the basis for several cryptographic schemes (e.g., (Ishai et al.
Revised: This was first [Figure] explicitly conjectured by Alekhnovich (2003), and has been the basis for several cryptographicschemes

Original: ,2008; Applebaum et al., 2009, 2010; Bogdanov et al., 2019)).
Revised: (e.g., (Ishai et al., 2008; Applebaum et al., 2009, 2010; Bogdanov et al., 2019)).

Original: Notation for neural networks and training.
Revised: Notation for neural networks and training.

Original: Our main results are presented in the online learningsetting, with a stream of i.i.d.
Revised: Our main results are presented in the online learningsetting, with a stream of i.i.d.

Original: batches of examples.
Revised: batches of examples.

Original: At each iteration t = 1 , . . .
Revised: At each iteration t = 1 , . . .

Original: , T , a learningalgorithm receives a batch of B examples { ( x t,i , y t,i ) }
Revised: , T , a learningalgorithm receives a batch of B examples { ( x t,i , y t,i ) }

Original: Bi =1 drawn i.i.d.
Revised: Bi =1 drawn i.i.d.

Original: from D S , then outputs aclassifier (cid:98) y t : {± 1 } n → {± 1 } .
Revised: from D S , then outputs aclassifier (cid:98) y t : {± 1 } n → {± 1 } .

Original: We say that the algorithm solves the parity task in t steps with ϵ error,if with probability at least 1 − ϵ over both training and internal randomness, as well as ( x, y ) ∼ D S ,(cid:98)
Revised: We say that the algorithm solves the parity task in t steps with ϵ error,if with probability at least 1 − ϵ over both training and internal randomness, as well as ( x, y ) ∼ D S ,(cid:98)

Original: y t ( x ) =
Revised: y t ( x ) =

Original: y .
Revised: y .

Original: We will focus on the case that (cid:98)
Revised: We will focus on the case that (cid:98)

Original: y t
Revised: y t

Original: = sign( f ( x ; θ t )) for some parameters θ t in a continuousdomain Θ and for a continuous function f : {± 1 } n × Θ → R .
Revised: = sign( f ( x ; θ t )) for some parameters θ t in a continuousdomain Θ and for a continuous function f : {± 1 } n × Θ → R .

Original: A ubiquitous online learningalgorithm is gradient descent (GD).
Revised: A ubiquitous online learningalgorithm is gradient descent (GD).

Original: For a choice of loss function ℓ : {± 1 } × R → R , initializationθ 0 (that are chosen from some distribution), learning rate schedule { η t } Tt =

Revised: For a choice of loss function ℓ : {± 1 } × R → R , initializationθ 0 (that are chosen from some distribution), learning rate schedule { η t } Tt =

Original: ⊆
Revised: ⊆

Original: R and weight-decayschedule { λ t } Tt =1 ⊆ R , GD refers the standard iterative update rule using the regularized, empiricalloss function, which is a function of architecture f .
Revised: R and weight-decayschedule { λ t } Tt =1 ⊆ R , GD refers the standard iterative update rule using the regularized, empiricalloss function, which is a function of architecture f .

Original: The learning rate η t can also be a vector (e.g.,allowing different rate schedules for different layers).
Revised: The learning rate η t can also be a vector (e.g.,allowing different rate schedules for different layers).

Original:  Empirical findings
Revised:  Empirical findings

Original: 3.1 SGD on neural networks learns sparse parities
Revised: 3.1 SGD on neural networks learns sparse parities

Original: The central phenomenon of study in this work is the empirical observation that neural networks, withstandard initialization and training, can solve the ( n, k ) -parity problem in a number of iterationsscaling as n O ( k ) on small instances.
Revised: The central phenomenon of study in this work is the empirical observation that neural networks, withstandard initialization and training, can solve the ( n, k ) -parity problem in a number of iterationsscaling as n O ( k ) on small instances.

Original: We observed robust positive results for randomly-initialized
Revised: We observed robust positive results for randomly-initialized

Original: SGD on the following architectures, indexed by Roman numerals:
Revised: SGD on the following architectures, indexed by Roman numerals:

Original: • 2-layer MLPs: ReLU ( σ ( z ) =
Revised: • 2-layer MLPs: ReLU ( σ ( z ) =

Original: ( z ) + ) or polynomial ( σ ( z )
Revised: ( z ) + ) or polynomial ( σ ( z )

Original: =
Revised: =

Original: z k ) activation, in a wide variety ofwidth regimes r ≥ k .
Revised: z k ) activation, in a wide variety ofwidth regimes r ≥ k .

Original: Settings (i), (ii), (iii) (resp.
Revised: Settings (i), (ii), (iii) (resp.

Original: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU
Revised: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU(resp. polynomial) activations.

Original: When f ( x ; θ ) = 0 in practice (e.g. with sign initialization), we break the tie arbitrarily.

Original: We ensure in the theoretical analysis that this does not happen.

Original: [Figure] (resp. polynomial) activations. We also consider r = k (exceptional settings (*i), (*ii) ), theminimum width for representing a k -wise parity for both activations.• Single neurons: Next, we consider non-standard activation functions σ which allow a one-neuronarchitecture f ( x ; w )
Revised: We also consider r = k (exceptional settings (*i), (*ii) ), theminimum width for representing a k -wise parity for both activations. • Single neurons: Next, we consider non-standard activation functions σ which allow a one-neuron

Original:  = σ ( w ⊤ x ) to realize k -wise parities.
Revised: architecture f ( x ; w ) = σ ( w ⊤ x ) to realize k -wise parities.

Original: The constructions stem from lettingw ∗ =
Revised: The constructions stem from lettingw ∗ =

Original: (cid:80) i ∈ S e i , and constructing σ ( · ) to interpolate (the appropriate scaling of)
Revised: (cid:80) i ∈ S e i , and constructing σ ( · ) to interpolate (the appropriate scaling of)

Original: k − w ∗⊤ xmod 2with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).
Revised: k − w ∗⊤ xmod 2with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).

Original: Going a stepfurther, a single ∞ -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.
Revised: Going a stepfurther, a single ∞ -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.

Original: We alsoremoved the second trainable layer (setting u = 1 ), obtaining settings (xi), (xii), (xiii), (xiv).
Revised: We alsoremoved the second trainable layer (setting u = 1 ), obtaining settings (xi), (xii), (xiii), (xiv).

Original: We found that wider architectures with these activations also trained successfully.
Revised: Wefound that wider architectures with these activations also trained successfully.

Original:  • Transformers: Motivated by recent theoretical and empirical work on the ability of self-attention tolearn sparse functions and parities (Lu et al., 2021; Edelman et al., 2021; Hahn, 2020), we consider a simplified specialization of the Transformer architecture to this sequence classification problem. 
Revised: [Figure] • Transformers: Motivated by recent theoretical and empirical work on the ability of self-attention tolearn sparse functions and parities (Lu et al., 2021; Edelman et al., 2021; Hahn, 2020), we considera simplified specialization of the Transformer architecture to this sequence classification problem. [Figure]

Original: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.1.3.
Revised: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.

Original: • PolyNets: Our final setting (xv) introduces a non-standard architecture, which may be of inde-
Revised: • PolyNets: Our final setting (xv) is the PolyNet, a slightly modified version of the parity machinearchitecture.

Original: pendent interest.
Revised: Parity machines have been studied extensively in the statistical mechanics of MLliterature (see the related work section) as well as in a line of work on ‘neural cryptography’(Rosen-Zvi et al., 2002).

Original: The PolyNet takes a product over k linear functions f ( x ; w 1: k ) : = (cid:81)
Revised: A parity machine outputs the sign of the product of k linear functions of the input. A PolyNet

Original: ki
Revised: simply

Original: =1 w ⊤ i x ;
Revised: outputs the product itself. Both

Original: this can clearly realize k -way parities.
Revised: architectures can clearly realizek -sparse parities.

Original: Originally motivated by the search for an idealizedsetting
Revised: The PolyNet architecture was originally motivated by the search for an idealized

Original:  where an end-to-end optimization trajectory analysis is tractable (see Section 4.1), we found in
Revised: setting where an end-to-end optimization trajectory analysis is tractable (see Section 4.1); we found in

Original: these experiments that this architecture trainsvery stably and sample-efficiently
Revised: these experiments that this architecture trains

Revised: very

Revised: stably

Original:  Robust space of positive results.
Revised: and sample -efficiently. Robust space of positive results.

Original: All of the networks listed above were observed to successfullylearn sparse parities in a variety of settings; it would be uninformative to enumerate all of them in atable or show training curves.
Revised: All of the networks listed above were observed to successfullylearn sparse parities in a variety of settings.

Original: Instead, we summarize our findings as follows: for all combinationsof n ∈ { 10 , 20 , 30 } , k ∈ { 2 , 3 , 4 } , batch sizes B ∈ { 1 , 2 , 4 , . . . , 1024 } , initializations { uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture configurations{ (i) , (ii) , . . 
Revised: We summarize our findings as follows: for all combinations of n ∈ { 10 , 20 , 30 } , k ∈ { 2 , 3 , 4 } , batch sizes B ∈ { 1 , 2 , 4 , . . . , 1024 } , initializations{ uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architectureconfigurations { (i) , (ii) , . . .

Original: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batch of 2 13samples) in at least 5 out of 10 random trials, for at least one choice of learning rate η ∈ { 0 . 01 , 0 . 1 , 1 } .
Revised: , (xv) } , SGD solved the parity problem (with 100% accuracy, validatedon a batch of 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rateη ∈ { 0 . 001 , 0 . 01 , 0 .

Original: The models converged in t c ≤ c · n αk ≤ 10 5 steps, for small architecture-dependent constants c, α ,shown in Appendix C.1. Figure 1 (left) shows some representative training curves.
Revised: The models converged in t c ≤ c · n αk ≤ 10 5 steps, for small architecturedependent constants c, α , shown in Appendix C. Figure 1 (left) shows some representative trainingcurves.

Original: Less robust configurations.
Revised: Less robust configurations.

Original: The settings (*i) and (*ii), where the MLP just barely representsa k -sparse parity, and the Transformer setting (*iii), were less robust to hyperparameter choices(especially the batch size B ).
Revised: Settings (*i) and (*ii), where the MLP just barely represents a k -sparseparity, and the Transformer setting (*iii), are less robust to small batch sizes. In these settings, thesame positive results as above only held for sufficiently large batch sizes: B ≥ 16 . Also, setting (*iii)used the Adam optimizer; see Appendix D.1.3 for details.

Original: We present these results in a few settings, described in  Appendix [-D.
-]
Revised: Phase transitions in training curves. For almost all of the architectures, we find that that thetraining curves exhibit phase transitions in terms of running time (and thus, in the online learningsetting, dataset size as well): long durations of seemingly no progress, followed by periods of rapiddecrease in the validation error. Strikingly, for architectures (v) and (vi), this plateau is absent: theerror in the initial phase appears to decrease with a linear slope. See Appendix [-D.
-]C.8 for more plots.

Original: 3.2 Random search or hidden progress?
Revised: 3.2 Random search or hidden progress?

Original: The remainder of this paper seeks to answer the question: “By what mechanism does deep learningsolve these emblematic computationally-hard optimization problems?”
Revised: The remainder of this paper seeks to answer the question: “By what mechanism does deep learningsolve these emblematic computationally-hard optimization problems?”

Original: [Figure] A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search,“bouncing around” the loss landscape in the absence of a useful gradient signal.
Revised: A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search,“bouncing around” the loss landscape in the absence of a useful gradient signal.

Original: Several empiricalobservations clash with this hypothesis:
Revised: Upon closer inspection,several empirical observations clash with this hypothesis:

Original: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization,
Revised: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization,

Original: it is unclear how to account for the n Θ( k ) runtimes observed in experiments.
Revised: it is unclear how to account for the n Θ( k ) runtimes observed in experiments.

Original: The initializations,which certainly do not prefer sparse functions 6 , are close to the correct solutions with probability2 − Ω( n ) ≪ n − k .
Revised: The initializations,which certainly do not prefer sparse functions 6 , are close to the correct solutions with probability 2 − Ω( n ) ≪ n − k .

Original: On larger instances ( n, k ) , the power-law exponents worsen; see Figure 2 (right) ,and the discussion in Appendix C.4.•
Revised: On larger instances ( n, k ) , the power-law exponents worsen; see Figure 2 (right) ,and the discussion in Appendix C.2.

Original:  No early convergence: Over a large number of random trials, no copies of this randomizedalgorithm get “lucky” (i e. solve the problem in significantly fewer than the median number ofiterations); see Figure 2 (left) .
Revised: • No early convergence: Over a large number of random trials, no copies of this randomizedalgorithm get “lucky” (i.e. solve the problem in significantly fewer than the median number ofiterations); see Figure 2 (left) .

Original: The success times of random exhaustive search would be distributedas Geom(1 / (cid:0) nk (cid:1) ) ; it is more likely to succeed (and terminate) atsmaller t .•
Revised: The success times of random exhaustive search would be distributedas Geom(1 / (cid:0) nk (cid:1) ) , whose probability mass is highest at t = 0 and decreases monotonically with t .

Original:  Sensitivity to initialization, not stochastic batches: Running these training setups over multiplestochastic batches from a common initialization, we find that loss curves and convergence timesare highly correlated with the architecture’s random initialization; see Figure 2 (center) .
Revised: • Sensitivity to initialization, not stochastic batches: Running these training setups over multiplestochastic batches from a common initialization, we find that loss curves and convergence timesare highly correlated with the architecture’s random initialization; see Figure 2 (center) .

Original: Even these observations, which do not probe the internal state of the algorithm, suggest  exhaustivesearch is an insufficient picture of the training dynamics and a different mechanism is at play.
Revised: Even these observations, which do not probe the internal state of the algorithm, suggest that exhaustivesearch is an insufficient picture of the training dynamics, and a different mechanism is at play.

Original:  Theoretical analyses
Revised:  Theoretical analyses

Original: 4.1 Provable emergence of the parity indices in high-precision gradients
Revised: 4.1 Provable emergence of the parity indices in high-precision gradients

Original: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.
Revised: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.

Original: Our main theoretical observation is that, in many cases, the population gradient of the weights atinitialization contains enough “information” for solving the parity problem.
Revised: Our main theoretical observation is that, in many cases, the population gradient of the weights atinitialization contains enough “information” for solving the parity problem.

Original: That is, given an accurateenough estimate of the initial gradient (by e.g. computing the gradient over a large enough batchsize), the relevant subset S can be found.
Revised: That is, given an accurateenough estimate of the initial gradient (by e.g. computing the gradient over a large enough batchsize), the relevant subset S can be found.

Original: As a warm-up example, consider training a single ReLU neuron f ( x ; w ) =
Revised: As a warm-up example, consider training a single ReLU neuron f ( x ; w ) =

Original: σ ( w ⊤ x ) w.r.t.
Revised: σ ( w ⊤ x ) w.r.t.

Original: thecorrelation loss ℓ ( y, (cid:98) y )
Revised: thecorrelation loss ℓ ( y, (cid:98) y )

Original: = − y (cid:98) y , from the initialization w =
Revised: = − y (cid:98) y , from the initialization w =

Original: While a single neuron cannotexpress the parity, we observe that the population gradient can indicate what the correct subset is E ( x,y ) ∼D S [ ∇ w i ℓ ( y, f ( x ; w ))]
Revised: While a single neuron cannotexpress the parity, we observe that the population gradient can indicate what the correct subsetis: E ( x,y ) ∼D S [ ∇ w i ℓ ( y, f ( x ; w ))]

Original: =
Revised: =

Original: E ( x,y ) ∼D S (cid:2) − yx i σ ′ ( w ⊤ x ) (cid:3) which corresponds to either the order-( k − 1 )
Revised: E ( x,y ) ∼D S (cid:2) − yx i σ ′ ( w ⊤ x ) (cid:3) which corresponds to either theorder- ( k − 1 )

Original: Fourier coefficient S \ { i } of the function x (cid:55)→ σ ′ ( w ⊤ x ) (if i ∈ S is a relevant coordinate)or the order- ( k + 1) coefficient S ∪ { i } (if i ̸∈ S is irrelevant).
Revised: Fourier coefficient S \ { i } of the function x (cid:55)→ σ ′ ( w ⊤ x ) (if i ∈ S is a relevantcoordinate) or the order- ( k + 1) coefficient S ∪ { i } (if i ̸∈ S is irrelevant).

Original: When σ is the ReLU function andw = 
Revised: When σ is the ReLUfunction and w = [1 , . .

Original:  , 1] , σ ′ ( w ⊤ x ) = sign ( (cid:80) i x i ) +1is just a shifted Majority function of x .
Revised: . , 1] , σ ′ ( w ⊤ x ) = sign ( (cid:80) i x i ) +1is just a shifted majority function of x .

Original: For Majority, the
Revised: The

Original: Fourier spectrum  is well understood: for even k , there is a gap between these Fourier coefficients thatis detectable using n O ( k ) samples.
Revised: Fourier spectrum of majority is well-understood: for even k , there is a gap between these Fouriercoefficients that is detectable using n O ( k ) samples.

Original: This analysis can be further extended to  ReLU neuron initialized with weights w ∈ {± 1 } n and asmall bias.
Revised: This analysis can be further extended to a ReLU neuron initialized with weights w ∈ {± 1 } n and asmall bias.

Original: In fact, we can show that taking a single gradient step with large enough batch size on a
Revised: In fact, we can show that taking a single gradient step with large enough batch size on a

Original: ReLU network trained with the square-loss, already finds features that can solve the parity problem.
Revised: ReLU network trained with the hinge-loss ℓ ( y, (cid:98) y ) = max { 1 − y (cid:98) y, 0 } , already finds features that cansolve the parity problem.

Original: Using this, we show that training a ReLU MLP with SGD solves the parity problem:
Revised: Using this, we show that training a ReLU MLP with SGD solves the parityproblem:

Original: Theorem 4.
Revised: Theorem 4.

Original: Fix some ϵ ∈ (0 , 1) , let k be some even number, and assume that n is some odd numbersatisfying n ≥ Ω( k 4 log( k/ϵ )) .
Revised: Fix some ϵ ∈ (0 , 1) , let k be some even number, and assume that n is some odd numbersatisfying n ≥ Ω( k 4 log( nk/ϵ )) .

Original: There exists a symmetric random initialization scheme, learningrate and weight-decay schedules, s.t. for every S ⊆
Revised: There exist a symmetric random initialization scheme and learningrate and weight decay schedules s.t. for every S ⊆

Original: [ n ] of size k , training a ReLU MLP of sizer
Revised: [ n ] of size k , training a ReLU MLP of sizer

Original: = O (2 k k log( k/ϵ )) with batch-size B = n O ( k ) log(1 /ϵ ) for T = poly( k, r, 1 /ϵ ) iterations on D Sw.r.t.
Revised: = O (2 k k log( k/ϵ )) with batch size B = n O ( k ) log 2 ( r/ϵ ) for T = poly( k, r, 1 /ϵ ) iterations on D S w.r.t.

Original: the square-loss, finds a network f
Revised: the hinge loss, finds a network f

Original: ( x ; θ t ) with expected loss: E [ ℓ ( f ( x ; θ t ) , y )] ≤ ϵ , where theexpectation is over the randomness of initialization, training and sampling ( x, y ) ∼ D S .
Revised: ( x ; θ t ) with expected loss: E [ ℓ ( f ( x ; θ t ) , y )] ≤ ϵ , wherethe expectation is over the randomness of initialization, training and sampling ( x, y ) ∼ D S .

Original: The analysis presented above shows that when the batch size scales with n O ( k ) , SGD over MLP solvesthe parity problem.
Revised: The analysis presented above shows that when the batch size scales with n O ( k ) , SGD over MLP solvesthe parity problem.

Original: However, in our experimental setting, the number of steps , and not the batch size,scales with n O ( k ) .
Revised: However, in our experimental setting, the number of steps , and not the batch size, scales with n O ( k ) .

Original: While we believe that running SGD with small batch size and small learning rateessentially amplifies the signal in the population gradient, behaving similarly to performing a largestep over a large batch size, we do not have a complete analysis for training MLPs with SGD in thesmall batch size regime.
Revised: While we believe that running SGD with small batch size and small learning rateessentially amplifies the signal in the population gradient, behaving similarly to performing a largestep over a large batch size, we do not have a complete analysis for training MLPs with SGD in thesmall batch size regime.

Original: To complement the above result, in Section 4.2 we analyze the trajectory fora variant of the PolyNet architecture trained with gradient flow.
Revised: To complement the above result, in Section 4.2 we analyze the trajectory fora variant of the PolyNet architecture trained with gradient flow.

Original: We note that, while the above analysis applies for ReLU MLPs with a specific initialization scheme, asimilar feature emergence phenomenon can be observed in a broader set of architectures and setting.
Revised: We note that, while the above analysis applies for ReLU MLPs with a specific initialization scheme, asimilar feature emergence phenomenon can be observed in a broader set of architectures and setting.

Original: Indeed, for feature emergence to occur, we only require that there is a “gap” between the relevant andirrelevant Fourier coefficients.
Revised: Indeed, for feature emergence to occur, we only require that there is a “gap” between the relevant andirrelevant Fourier coefficients.

Original: Formally, denote (cid:98) f
Revised: Formally, denote (cid:98) f

Original: ( S ) :
Revised: ( S ) :

Original: =
Revised: =

Original: E [ f ( x ) χ S ( x )] the Fourier coefficient of fat S , and observe the following definition:
Revised: E [ f ( x ) χ S ( x )] the Fourier coefficient of fat S , and observe the following definition:

Original: Definition 1 (Fourier gap) .
Revised: Definition 1 (Fourier gap) .

Original: For some function f : {± 1 } n → R and some subset S of size k , wesay that f has a γ - Fourier gap at S if (1) for every ( k − 1)
Revised: For some function f : {± 1 } n → R and some subset S of size k , wesay that f has a γ - Fourier gap at S if (1) for every ( k − 1)

Original: -element subset S ′ ⊆ S , it holds that| (cid:98) f ( S ′ )
Revised: -element subset S ′ ⊆ S , it holds that| (cid:98) f ( S ′ )

Original: | ≥ γ , and (2) for every subset S ′ ⊆
Revised: | ≥ γ , and (2) for every subset S ′ ⊆

Original: [ n ] of size k + 1 it holds that | (cid:98) f ( S ′ )
Revised: [ n ] of size k + 1 it holds that | (cid:98) f ( S ′ )

Original: | ≤ γ/ 2 .
Revised: | ≤ γ/ 2 .

Original: Now, given a network architecture where some neuron has a γ -Fourier gap with respect to the targetsubset S , we can generalize the result of the ReLU neuron.
Revised: Now, given a network architecture where some neuron has a γ -Fourier gap with respect to the targetsubset S , we can generalize the result of the ReLU neuron.

Original: That is, we show that the subset S can bedetermined by observing an estimate of the population gradient at initialization:
Revised: That is, we show that the subset S can bedetermined by observing an estimate of the population gradient at initialization:

Original: Proposition 5.
Revised: Proposition 5.

Original: Let σ be some activation function and let ℓ be some loss function.
Revised: Let σ be some activation function and let ℓ be some loss function.

Original: Denote f ( x ; w ) =
Revised: Denote f ( x ; w ) =

Original: σ ( w ⊤ x ) .
Revised: σ ( w ⊤ x ) .

Original: Fix some subset S ⊆
Revised: Fix some subset S ⊆

Original: [ n ] .
Revised: [ n ] .

Original: Let g ∈ R n be an estimate of the population gradient suchthat (cid:13)(cid:13)
Revised: Let g ∈ R n be an estimate of the population gradient suchthat (cid:13)(cid:13)

Original: g − E ( x,y ) ∼D S [ ∇ w ℓ ( y, f ( x ; w ))] (cid:13)(cid:13) ∞
Revised: g − E ( x,y ) ∼D S [ ∇ w ℓ ( y, f ( x ; w ))] (cid:13)(cid:13) ∞

Original: ≤ γ/ 4 .
Revised: ≤ γ/ 4 .

Original: Then, for every w s.t.
Revised: Then, for every w s.t.

Original: σ ′ ( w ⊤ x ) has a γ -Fouriergap w.r.t.
Revised: σ ′ ( w ⊤ x ) has a γ -Fouriergap w.r.t.

Original: to S and ℓ ′ ( f ( x ; w ) , y ) = − y for all x , the target subset S is detected by g , namely
Revised: to S and ℓ ′ ( f ( x ; w ) , y ) = − y for all x , the target subset S is detected by g , namely

Original: S =
Revised: S =

Original: { i ∈ [ n ] :
Revised: { i ∈ [ n ] :

Original: | g i | ≥ 3 γ/ 4 } .
Revised: | g i | ≥ 3 γ/ 4 } .

Original: Comparison with NTK analysis.
Revised: Comparison with NTK analysis.

Original: In recent years, many theoretical works have studied the behaviorof SGD on neural networks through the lens of the neural tangent kernel (NTK)
Revised: In recent years, many theoretical works have studied the behaviorof SGD on neural networks through the lens of the neural tangent kernel (NTK)

Original: (Jacot et al., 2018).
Revised: (Jacot et al., 2018).

Original: It is therefore important to highlight the fact that the NTK (or, in fact, any kernel) cannot solve thesparse parity problem.
Revised: It is therefore important to highlight the fact that the NTK (or, in fact, any kernel) cannot solve thesparse parity problem.

Original: The following result shows that no kernel can achieve small error on the sparseparity problem, unless the size of the kernel is n Ω( k ) :
Revised: The following result shows that no kernel can achieve small error on the sparseparity problem, unless the size of the kernel is n Ω( k ) :

Original: Theorem 6.
Revised: Theorem 6.

Original: Let Ψ : {± 1 } n → R D be some D -dimensional embedding.
Revised: Let Ψ : {± 1 } n → R D be some D -dimensional embedding with sup x ∥ Ψ( x ) ∥ 2 ≤ 1 , and let R > 0 be some number.

Original: If D < 2  ϵ  · (cid:0) nk (cid:1) , then thereexists some ( n, k ) -parity distribution D S s.t.inf
Revised: If DR 2 < ϵ 2 · (cid:0) nk (cid:1) , then there exists some ( n, k ) -parity distribution D S s.t.

Original: w ∈ R D E ( x,y ) ∼D S (cid:2) ℓ (Ψ( x ) ⊤ w, y ) (cid:3) > 1 / 2 − ϵ .
Revised: [Equation]

Original: The above is a corollary of the results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022).
Revised: The above is a corollary of the results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022).

Original: 4.2 PolyNet: an idealized architecture for trajectory analyses
Revised: 4.2 Disjoint-PolyNet: an idealized architecture for trajectory analyses

Original: In this section, we present an idealized architecture (a version of PolyNets (xv)) that exhibits similarbehavior as MLPs (experimentally) and is technically easier to analyze.More specifically, we [Figure] [Figure]
Revised: In this section, we present an idealized architecture (a version of PolyNets (xv)) that exhibits similarbehavior to MLPs (experimentally) and is technically easier to analyze.

Original: [Figure]

Original: ( r

Original: [Figure]

Original: )  21e 0.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: c l a ss i f

Original: i c a t i o n e rr o r m

Original: [Figure]

Original: [Figure]

Original: m

Original: [Figure]

Original: 4 1e m =

Original: [Figure] [Figure] [Figure] m =iteration t consider the disjoint PolyNet which takes a product over k linear functions where the linear functionsare computed on k disjoint partitions of the input P 1 , . . 
Revised: More specifically, we considerthe disjoint-PolyNet which takes a product over k linear functions where the linear functions arecomputed on k disjoint partitions of the input P 1 , . . .

Original: , P k with each P i =
Revised: , P k with each P i =

Original: { ( i − 1) n ′ +1 , . . 
Revised: { ( i − 1) n ′ + 1 , . . .

Original: , im }with n ′ = n/k 7 , that is, f ( x ; w 1: k ) :
Revised: , in ′ }with n ′ = n/k 7 , that is, f ( x ; w 1: k ) :

Original: = (cid:81)
Revised: = (cid:81)

Original: ki
Revised: ki

Original: =1 w ⊤ i x P i .
Revised: =1 w ⊤ i x P i .

Revised: As noted in the related work section, this isequivalent to a tree parity machine but with real-valued rather than ± 1 output.

Original: In order for the disjoint-PolyNet to be able to express the class of parity problems, we assume thatthe set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to eachdisjoint partition, that is, for all i ∈
Revised: In order for the disjoint-PolyNet to be able to express the class of parity problems, we assume thatthe set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to eachdisjoint partition, that is, for all i ∈

Original: [ k ] , S ∩ P i = 1 .
Revised: [ k ] , S ∩ P i = 1 .

Original: We refer to this problem as the ( n, k ) -disjointparity problem.
Revised: We refer to this problem as the ( n, k ) -disjointparity problem.

Original: Note that there are stillm k = ( n/k )
Revised: Note that there are still ( n ′ )

Original: k
Revised: k

Original:  different possibilities for set S under thisrestriction.
Revised: = ( n/k ) k different possibilities for set S under thisrestriction.

Original: For fixed k , these represent a constant portion of the (cid:0) nk (cid:1)
Revised: For fixed k , these represent a constant portion of the (cid:0) nk (cid:1)

Original: ≈ ( ne/k ) k
Revised: ≈ ( ne/k ) k

Original: (by Stirling’sapproximation) possibilities for S in the general non-disjoint case.
Revised: (by Stirling’sapproximation) possibilities for S in the general non-disjoint case.

Original: Consider training a disjoint-PolyNet w.r.t.
Revised: Consider training a disjoint-PolyNet w.r.t.

Original: the correlation loss.
Revised: the correlation loss.

Original: WLOG, let S = { 1 , n ′ + 1 , . . .
Revised: WLOG, let S = { 1 , n ′ + 1 , . . .

Original: , ( k −1) n ′ + 1 } and e 1 = (1 , 0 , .. 0) .
Revised: , ( k −1) n ′ + 1 } and e 1 = (1 , 0 , .. 0) .

Original: The population gradient is non-zero at i iff i ∈ S : g i ( w 1: k ) =
Revised: The population gradient is non-zero at i iff i ∈ S : g i ( w 1: k ) =

Original: E 
Revised: E [ ∇ w i ℓ ( f ( x ; w 1: k ) , y )] = − E (cid:104) y (cid:16)(cid:81) j ̸ = i w ⊤ j x P j (cid:17) x

Original: [Equation] ∇
Revised: P i (cid:105) =

Original: [Equation]
Revised: − (cid:16)(cid:81) j ̸ = i w j, 1 (cid:17) e 1 .

Original: [Equation]
Revised: [Figure]

Original: Now we consider the gradient flow dynamics of disjoint-PolyNet, which provide a mathematicallytractable case study for the trajectory of sparse parity learning.
Revised: Now we consider the gradient flow dynamics of disjoint-PolyNet, which provide a mathematicallytractable case study for the trajectory of sparse parity learning.

Original: For each i ∈ [ k ] , in this sectionwe treat w i as a function from R ≥ 

Revised: For each i ∈ [ k ] , in this sectionwe treat w i as a function from R ≥

Original: → R m which satisfies the following differential equation˙ w i = − g i ( w 1: k ( t )) .
Revised: → R n ′ which satisfies the following differential equation:˙ w i = − g i ( w 1: k ( t )) .

Original: For clarity of exposition, assume all-ones initialization.
Revised: For clarity of exposition, assume all-ones initialization.

Original: Then all of the relevantweights { w i, 1 : i ∈ [ k ] } follow the same trajectory, which we denote by v : R ≥ 

Revised: Then all of the relevantweights { w i, 1 : i ∈ [ k ] } follow the same trajectory, which we denote by v : R ≥

Original: → R .
Revised: → R .

Revised: By analyzing the resulting differential equations, we can formally exhibit “phase transition”-likebehavior in the fully deterministic gradient flow setting.

Original: Theorem 7.
Revised: Theorem 7 (Gradient flow on disjoint-PolyNets) .

Original: The error at time t is 1 / 2 − ( v (t ) / √ m ) ck for some constant c > 0 . 
Revised: Suppose k ≥ 3 . Let T ( . 49) be the time it takes forerror to fall below . 49 , and let T (0) be the time it takes to reach zero error.

Original: Thus, the trajectory of error appears to display a sharper “phase transition” than that of v ( t ) .
Revised: Then

Original: Theorem 8.
Revised: [Equation]

Original: For k ≥ 3 and ϵ > 0 , the fraction of the time it takes for the error to fall below 12 − ϵ is1 − ˜ O (cid:0) 1 ϵ 1 − 2 /k
Revised: In other words, the network takes much longer to reach slightly better than trivial accuracy than ittakes to go from slightly better than trivial to perfect accuracy. We can also analyze the trajectory of disjoint-PolyNets trained with online SGD, confirming that aneural network trained with batch size 1 SGD can learn k -sparse parities within n O ( k ) iterations. Theorem 8 (SGD on disjoint-PolyNets) .

Original: m k/ 2 − 1 (cid:1) of the running time required to achieve zero error. In other words, for sufficiently large m , almost no progress is visible in the accuracy for most of therunning time.
Revised: Suppose we train the disjoint-PolyNet, initialized as above,with online SGD with any batch size B ≥ 1 . Then there exists an adaptive learning rate schedulesuch that for any ϵ > 0 , with probability .99, the error falls below ϵ within [Equation]

Original:   Hidden progress: discussion and additional experiments
Revised: steps. Extended versions of these theorems, along with proofs, can be found in Appendix B.3.  Hidden progress: discussion and additional experiments

Original: In this section, we advocate for sparse parities as an idealized testbed for understanding algorithmsand phenomena in modern deep learning.
Revised: In this section, we advocate for sparse parities as an idealized testbed for understanding algorithmsand phenomena in modern deep learning.

Original: These are accompanied by experimental vignettes whichare auxiliary to the core results from Section 3, with more systematic studies deferred to future work.
Revised: These are accompanied by experimental vignettes whichare auxiliary to the core results from Section 3, with more systematic studies deferred to future work.

Original: Details are given in Appendix D.
Revised: Details are given in Appendix D.

Original: [Table]

Original: [Table]

Original: [Table]

Original: A progress measure for parity.
Revised: A progress measure for parity.

Original: To begin, the black-box experiments in Section 3 suggest thatrandom search is the incorrect model of SGD’s behavior in this setting.
Revised: To begin, the black-box experiments in Section 3 suggest thatrandom search is the incorrect model of SGD’s behavior in this setting.

Original: Using the theoretical insightthat amplifying a precise initial population gradient suffices for learning parities, we construct onepossible progress measure, which is a function of the sequence of weights w 0 , . . .
Revised: Using the theoretical insight that amplifying a precise initial population gradient suffices for learning parities, we construct onepossible progress measure, which is a function of the sequence of weights w 0 , . . .

Original: w t ∈ R n so far:ρ ( w 0: t ) :
Revised: w t ∈ R n so far:ρ ( w 0: t ) :

Original: =
Revised: =

Original: ∥ w t − w 0 ∥ ∞ , using the fact that w t − w 0 is an estimate for the initial population gradientin the linearized setting.
Revised: ∥ w t − w 0 ∥ ∞ , using the fact that w t − w 0 is an estimate for the initial population gradientin the linearized setting.

Original: Figure 3 shows how gradual weight movement (and thus, progress) can behidden behind plateauing losses.
Revised: Figure 3 shows how gradual weight movement (and thus, progress) can behidden behind plateauing losses.

Original: Roles of overparameterization vs. oversampling .
Revised: Roles of overparameterization vs. oversampling .

Original: An interesting consequence of our analysis is thatit illuminates scaling behaviors with respect to a third fundamental resource parameter: model size ,which we study in terms of network width r .
Revised: An interesting consequence of our analysisis that it illuminates scaling behaviors with respect to a third fundamental resource parameter: modelsize , which we study in terms of network width r .

Original: If SGD operated by a “random search” mechanism, onewould expect width to provide a parallel speedup.
Revised: If SGD operated by a “random search” mechanism,one would expect width to provide a parallel speedup.

Original: Instead, we SGD sequentially amplifies progress.
Revised: Instead, we SGD sequentially amplifiesprogress.

Original: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD does not give× 1 /r  speedups; more directly, in Figure 4 (left) , convergence times for sparse parities empiricallyplateau at large model sizes.
Revised: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD doesnot give (1 /r ) × speedups; more directly, in Figure 4 (left) , convergence times for sparse paritiesempirically plateau at large model sizes.

Original: It is a significant challenge to understand the interactions between network depth and computation,and largely outside the scope of this work.
Revised: It is a significant challenge to understand the interactions between network depth and computation,and largely outside the scope of this work.

Original: However, in Appendix C.3, we provide a brief note onusing parities and polynomial-activation MLPs to construct a simple counterexample to the “deeponly works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating acase where a deep network can get near-perfect accuracy even when greedy layerwise training (e.g.(Belilovsky et al., 2019)) cannot beat trivial performance.
Revised: However, in Appendix C.7, we provide a brief note onusing parities and polynomial-activation MLPs to construct a simple counterexample to the “deeponly works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating acase where a deep network can get near-perfect accuracy even when greedy layerwise training (e.g.(Belilovsky et al., 2019)) cannot beat trivial performance.

Original: Learning and grokking in the finite-sample multi-pass setting.
Revised: Learning and grokking in the finite-sample multi-pass setting.

Original: The main theoretical and empiricalresults in this work consider online learning algorithms which couple the resources of training timeand independent samples.
Revised: The main theoretical and empirical results in this work consider online learning algorithms which couple the resources of training timeand independent samples.

Original: However, due to the computational-statistical gap in parity learning, thesepositive results are suboptimal in terms of sample efficiency.
Revised: However, due to the computational-statistical gap in parity learning, thesepositive results are suboptimal in terms of sample efficiency.

Original: We find that minibatch SGD (with weightdecay) can empirically solve sparse parities, even from a sample of size m ≪ n k .
Revised: We find that minibatch SGD (with weightdecay) can empirically solve sparse parities, even from a sample of size m ≪ n k .

Original: For small values ofm , we reliably observe the grokking phenomenon (Power et al., 2022): overfitting for a long time, thena delayed phase transition for the generalization error; see the two center panels of Figure 4 (right) .
Revised: For small values ofm , we reliably observe the grokking phenomenon (Power et al., 2022): overfitting for a long time, thena delayed phase transition for the generalization error; see the two center panels of Figure 4 (right) .

Original:  Conclusion
Revised:  Conclusion

Original: This work puts forward parity learning as a stylized test case to explore some of the puzzling featuresof the role of computational (as opposed to statistical) resources in deep learning.
Revised: This work puts forward parity learning as a stylized test case to explore some of the puzzling featuresof the role of computational (as opposed to statistical) resources in deep learning.

Original: These includediscontinuous improvements (a.k.a.
Revised: These includediscontinuous improvements (a.k.a.

Original: emergent capabilities ), feature learning, and universality ofarchitectures.
Revised: emergent capabilities ), feature learning, and universality ofarchitectures.

Original: In particular, we show that deep learning on parities exhibits a phase transition behavior,that it is successfully learned by a variety of deep-net architectures, and that this success cannot beexplained as a “random exhaustive search”, nor through frameworks such as the neural tangent kernelor layer-by-layer learning.
Revised: In particular, we show that deep learning on parities exhibits a phase transition behavior,that it is successfully learned by a variety of deep-net architectures, and that this success cannot beexplained as a “random exhaustive search”, nor through frameworks such as the neural tangent kernelor layer-by-layer learning.

Original: However, there are more experimental and theoretical questions, even for this simplified case of paritylearning.
Revised: However, there are more experimental and theoretical questions, even for this simplified case of paritylearning.

Original: Our focus in this work was on the online learning case, where training time and samplesarise in tandem.
Revised: Our focus in this work was on the online learning case, where training time and samplesarise in tandem.

Original: However, we believe it would be instructive to investigate parity learning when threeresources of samples, time, and model size are scaled separately.
Revised: However, we believe it would be instructive to investigate parity learning when threeresources of samples, time, and model size are scaled separately.

Original: Some very preliminary findingsalong these lines are presented in Section 3.
Revised: Some very preliminary findingsalong these lines are presented in Section 3.

Original: Extending our theoretical results to the small-batch setting, as well as to more architectures, is anopen problem.
Revised: Extending our theoretical results to the small-batch setting, as well as to more architectures, is anopen problem.

Original: Resolving it would require a better understanding of anti-concentration (lower boundon deviation from mean) of Fourier coefficients, a phenomenon that is much less studied than theconcentration of these coefficients.
Revised: Resolving it would require a better understanding of anti-concentration (lower boundon deviation from mean) of Fourier coefficients, a phenomenon that is much less studied than theconcentration of these coefficients.

Original: We would also want to extend the analysis beyond parities totasks that are not aligned with the elementary basis such as low-rank tensor recovery.
Revised: We would also want to extend the analysis beyond parities totasks that are not aligned with the elementary basis such as low-rank tensor recovery.

Original: Another important follow-up direction is understanding the extent that these insights extend fromparity learning to real-world problems, as well as the extent into which non-synthetic tasks (in,e.g., natural language processing and program synthesis) embed within them parity-like subtasks ofexhaustive combinatorial search.
Revised: Another important follow-up direction is understanding the extent that these insights extend fromparity learning to real-world problems, as well as the extent into which non-synthetic tasks (in,e.g., natural language processing and program synthesis) embed within them parity-like subtasks ofexhaustive combinatorial search.

Original: Broader impact.
Revised: Broader impact.

Original: This work seeks to contribute to the foundational understanding of computationalscaling behaviors in deep learning.
Revised: This work seeks to contribute to the foundational understanding of computationalscaling behaviors in deep learning.

Original: Our theoretical and empirical analyses are in a heavily-idealizedsynthetic problem setting.
Revised: Our theoretical and empirical analyses are in a heavily-idealizedsynthetic problem setting.

Original: Hence, we see no direct societal impacts of these results in this study.
Revised: Hence, we see no direct societal impacts of the results in this study.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type
Revised: g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?

Original: [N/A]
Revised: [N/A]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original:  (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]