Original: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Revised: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: There is mounting empirical evidence of emergent phenomena inthe capabilitiesof deep learning methods as we scale up datasets, model sizes, and training times.
Revised: Why is it that so many novel capabilities in deep learning are unlocked by scalingup datasets, model sizes, and training times?

Original: While there are some accounts of how these resources modulate statistical capacity,far less is known about their effect on the computational problem of model training.
Revised: While there are some accounts ofhow these factors modulate statistical capacity, far less is known about how thesefactors affect the computational problem of model training.

Original: This work conducts such an exploration through the lens of learning k -sparseparities of n bits, a canonical family of problems which posetheoretical computational barriers.
Revised: This work exploresthese computational issues through the lens of learning k-sparse parities of nbits, a canonical family of problems which pose computational barriers.

Original: In this setting, we find that neural networks exhibit surprisingphase transitions when scaling up dataset size and running time.
Revised: Perhapssurprisingly, standard deep learning approaches, when applied to this problem,already exhibit a number of discontinuous impacts of increasing resources (suchas dataset size, training time, or model size) on model performance, making it anatural candidate for rigorously studying these phase transitions.

Original: In particular, wedemonstrate empirically that with standard  training, a variety of architectures learnsparse parities with n O ( k ) examples, with loss (and error) curves abruptly droppingafter n O ( k ) iterations. These positive results nearly match known SQ lower bounds,even without an explicit sparsity-promoting prior.
Revised: In particular, weempirically demonstrate that with standard initialization and training, a variety ofarchitectures learn sparse parities in a number of iterations that nearly matchesoptimal lower bounds (in the SQ learning model), even without an explicit sparsitypromoting prior.

Original: We elucidate the mechanisms ofthese phenomena with a theoretical analysis: we find that the phase transition inperformance is not due to SGD “stumbling in the dark” until it finds the hidden setof features (a natural algorithm which also runs in n O ( k ) time); instead, we showthat SGD gradually amplifies a Fourier gap  in the population gradient.
Revised: Furthermore, we elucidate the mechanisms of these positive resultswith a theoretical analysis: we argue that the phase transition in performance isnot due to SGD "stumbling in the dark" (with a Langevin-like mechanism) until itfinds the hidden set of features; instead, we show that SGD gradually amplifies a Fourier gap (the hidden progress measure) in the population gradient.

Original:  Introduction
Revised:  Introduction

Original: Neural networks perform better with more resources (data, model size, training time) but differenttasks exhibit qualitatively different dependencies of performance on resources.
Revised: Neural networks perform better with more resources (data, model size, training time), but differenttasks exhibit qualitatively different dependencies of performance on resources.

Original: In particular, whilemany learning tasks exhibit continuous improvement in performance with increasing resources, othercases show discontinuous improvement, where a capability emerges at a certain threshold.
Revised: In particular, whilemany learning tasks exhibit continuous improvement in performance with increasing resources, othercases show discontinuous improvement, where a capability emerges at a certain threshold .

Original: Through astatistical lens, it is well-understood that larger models, trained with more data, can fit more complexand expressive functions.

Original: However, far less is known about the analogous computational question:how does the scaling of these resources influence the success of gradient-based optimization fortraining these models?

Original: These emergent phase transitions cannot be explained via statistical capacity alone.
Revised: Thesephase transitions cannot be explained via statistical capacity alone.

Original: In many caseswe see a phase transition even when the amount of data remains fixed, with only model size ortraining time increasing.
Revised: Indeed, in many cases we seea phase transition even when the amount of data remains fixed, with only model size or trainingtime increasing.

Original: A timely example is the emergence of reasoning and few-shot learningcapabilities when scaling up language models (56, 16, 18, 36).
Revised: A timely example is the emergence of reasoning and few-shot learning capabilitieswhen scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al. , 2022),including computational resource only (Hoffmann et al., 2022).

Original: Power et al.
Revised: Power et al.

Original: (55) give examplesexhibiting discontinuous improvements in population accuracy (“grokking”) when running timeincreases, while data and model size remain fixed.
Revised: (2022) give examplesexhibiting discontinuous improvements in population accuracy (“grokking”) when running timeincreases, while data and model size remain fixed.

Original: In this work, we analyze the computational aspect of scaling in deep learning in a simple syntheticsetting which already exhibits discontinuous improvements.
Revised: In this work, we analyze the computational aspect of scaling in deep learning in an idealized problemsetting which already exhibits discontinuous improvements.

Original: Specifically, we consider the sparseparity problem — where the label is the parity(XOR) of k ≪ n bits in a random length- n binary string.
Revised: Specifically, we consider the sparseparity problem — where the concept is the parity of k ≪ n bits of an n -dimensional input.

Original: This is a canonical problem which is computationally difficult for a range of algorithms, includinggradient-based (41) and streaming (44) algorithms.
Revised: This is acanonical problem that is computationally difficult for a range of algorithms, including gradient-basedmethods (Kearns, 1998) and streaming algorithms (Kol et al., 2017).

Original: We focus on analyzing the resource measure of 
Revised: We focus on analyzing theresource measure of training time , and demonstrate that the loss curves for sparse parities display a

Original: [Figure]
Revised: [Figure]

Original: that the loss curves for sparse parities display a phase transition acrossa variety of architectures and hyperparameters (see Figure 1, left).
Revised: phase transition across a variety of architectures and hyperparameters (see Figure 1, left).

Original: A striking observation is that SGDrobustly finds the sparse subset (and hence, effectively, reaches 0 error) with a variety of activationfunctions and initialization schemes,  even with no over-parameterization.
Revised: A strikingobservation is that SGD robustly finds the sparse subset (and hence, effectively, reaches 0 error)with a variety of activation functions, initialization schemes, and, as shown later on, even with noover-parameterization.

Original: This robust convergence isthe starting point for our investigation.
Revised: This robust convergence is the starting point for our investigation.

Original: Perhaps the most natural hypothesis to explain SGD’s success in learning parities would be that itsimply “stumbles in the dark”, essentially performing random search for the unknown target (e.g.via stochasticgradient Langevin dynamics).
Revised: Perhaps the most natural hypothesis to explain SGD’s success in learning parities is that it simply“stumbles in the dark”, essentially performing random search for the unknown target (e.g. throughsome stochastic Langevin dynamics).

Original: If that were the case, we might expect to observe aconvergence time of 2 Ω( n ) , like a naive search over parameters or subsets of indices.
Revised: If that was the case, we may expect to observe a convergencetime of 2 O ( n ) (which is consistent that of a naive parameter search).

Original: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: theconvergence time is closer to an n O ( k ) scaling.
Revised: However, Figure 1, right,already provides some evidence against this “random search” hypothesis. This figure shows that theconvergence time is closer to an n O ( k ) scaling.

Original: Notably, such a convergence rate impliesthat SGD iscloser to achieving the optimal computation time among a natural class of algorithms, namely SQ(statistical query) algorithms.
Revised: Notably, such a convergence rate implies SGD iscloser to achieving the optimal computation time among a natural class of algorithms, namely SQ(statistical query) algorithms.

Original: Through an extensive empirical analysis of the scaling behavior of a variety of models, as well astheoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.
Revised: In this work, through an extensive empirical analysis of the scaling behavior of a variety of models,as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.

Original: Instead,there is a hidden progress measureunder which  SGD is steadily improving.
Revised: Instead, there is a hidden progress measure which under SGD is steadily improving.

Original: Furthermore, andperhaps surprisingly, we show that SGD achieves a computational runtime  much closer to the optimal SQ  lower bound than simply doing (non-sparse) parameter search.
Revised: Furthermore,and perhaps surprisingly, we also give empirical and theoretical evidence that SGD is achieving acomputational runtime that is much closer to the optimal SQ computational lower bound than simplydoing (random) parameter search.

Original: More generally, our theoreticaland empirical investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources, and we provide further theoretical and empirical evidence of phasetransitions with data, model size, and training time.
Revised: More generally, our theoretical and empirical investigations reveala number of notable phenomena regarding the dependence of SGD’s performance on resources, andwe provide further theoretical and empirical evidence of phase transitions with data, model size, andtraining time.

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: SGD efficiently learns sparse parities, in theory and practice.
Revised: SGD efficiently learns sparse parities, in theory and practice.

Original: It is known from SQ lower boundsthat (noisy) gradient descent on any architecture requires at least n Ω( k ) computational steps tolearn k -sparse n -dimensional parities (for background, see Appendix A).
Revised: It is known that (a noisy SQabstraction of) SGD on any architecture requires at least n Ω( k ) computational steps to learn k -sparsen -dimensional parities (for background, see Appendix A).

Original: However, with standardarchitectures and initialization, which do notexplicitly encode a sparsity prior, one may expect SGD’sperformance to be much worse, on the order of 2 Ω( n ) .
Revised: However, with standard architectures andinitialization, which do not encode the sparsity prior, one may expect SGD’s performance to be muchworse, on the order of 2 Ω( n ) .

Original: We give extensive empirical evidence that thisis not the case.
Revised: We give extensive empirical evidence that this is not the case.

Original: Empirical Finding 1.
Revised: Empirical Finding 1.

Original: For all small instances ( n ≤ 30 , k ≤ 4 ) of the sparse parity problem,architectures A ∈ { 2-layer MLPs, Transformers 1 , sinusoidal (and other non-standard) neurons, PolyNets 2 } , initializations in { uniform, Gaussian, Bernoulli } , and  batch sizes  1 ≤ B ≤ 1024  , SGDon A solves the ( n, k ) -sparse parity problem (w.p. ≥ 0 . 2) within at most c · n αk steps, for smallarchitecture-dependent constants c, α .
Revised: For all small instances ( n ≤ 30 , k ≤ 4 ) of the sparse parity problem,architectures A ∈ { 2-layer MLPs, Transformers 1 , single neurons with ReLU or oscillating activations, PolyNets 2 } , and initialization in { uniform, Gaussian, Bernoulli } , and for all batch sizes in{ 1 , . . . , 1024 } , SGD on A solves the ( n, k ) -sparse parity problem with probability ≥ 1 / 2 within atmost c · n αk steps, for small architecture-dependent constants c, α .

Original: For a subset of these architectures, we performed more extensive training, to show scaling behaviorsof the computation time; 3 see Figure 1 (right) and Figure 10 in the appendix.
Revised: For a subset of these architectures, we performed more extensive training, to show scaling behaviorsof the computation time. 3 See Figure 1, right.

Original: The above results indicate that SGD succeeds at solving parities much faster than the 2 Ω( n ) steps thatwould be required by random search without a sparse prior.
Revised: The above results indicate that SGD succeeds in solving parities much faster than the 2 Ω( n ) stepsthat would be required by random search without a sparse prior.

Original: This suggests that despite  flat loss andaccuracy curves before the  phase transition, SGD makes progress “under the hood”.
Revised: This suggests that despite progressseeming flat in the first phase of learning, SGD does in fact make progress “under the hood”.

Original: We show thatthis is indeed the case by coming up with a progress measure that continuously improves throughouttraining.
Revised: Weshow that this is indeed the case by coming up with a progress measure that continuously improvesthroughout training.

Original: We can explain the loss and accuracy curves using this progress measure, by showing thatthey typically rise sharply once the measure passes a certain threshold.
Revised: We can explain the loss and accuracy curves using this progress measure, byshowing that they typically rise sharply once the measure passes a certain threshold.

Original: Theoretical Analysis.
Revised: Theoretical analysis.

Original: Our empirical results suggest that, in a number of computational stepsmatching the SQ limit, SGD is able to solve the parity problem and identify the influential coordinates,without an explicit sparse prior.
Revised: Our empirical results suggest that SGD is able to do feature learning for theparity problem, and identify the influential coordinates, even without a sparsity-inducing prior.

Original: We give a theoretical analysis which validates this claim.
Revised: Wegive a theoretical analysis that validates this claim.

Revised: For MLPs, we prove that the population gradientcontains enough signal on the influential coordinates to enable learning in the large-batch limit.

Original: Informal Theorem 2.
Revised: Informal Theorem 2.

Original: On 2-layer MLPs of width 2 Θ( k ) , and with batch size n O ( k ) , SGDconvergeswith high probability to a solution with at most ϵ error on the ( n, k ) -parity problem in at most2 O ( k ) · poly(1 /ϵ ) iterations.
Revised: For 2-layer MLPs, SGD in the large batch limit learns the relevant featuresusing n O ( k ) samples.

Original: We also present a stronger analysis for an idealized architecture (which we call  the disjoint-PolyNet ),which allows for any batch size, and captures the phase transitions observed in the error curves.
Revised: For an idealized architecture (which we call disjoint-PolyNet), we theoretically explain the qualitativebehavior of the error curves.

Original: Informal Theorem 3.
Revised: Informal Theorem 3.

Original: On disjoint-PolyNets, SGD (with any batch size B ≥ 1 ) converges with highprobability to a solution with at most ϵ error on  the ( n, k ) -parity problem in at most n O ( k ) · log(1 /ϵ )iterations. Continuous-time gradient flow exhibits a phase transition: it spends a 1 − o (1) fraction ofits time before convergence with error ≥ 49% .
Revised: Gradient flow on disjoint-PolyNet exhibits two properties: (1) early progressis more visible in the weights than in the error, and (2) for a large fraction of the total running time(until perfect accuracy), accuracy remains nearly trivial.

Original: In addition to refuting the alternative “random search” hypothesis, our work also poses a counterexample to other models for the computational mechanisms of deep learning; for instance, it provides asetting where deep neural nets successfully learn a concept to 100% accuracy while the corresponding 
Revised: In addition to refuting a “random search” model, our work also poses a counterexample to othermodels for the computational mechanisms of deep learning; for instance, it provides a setting wheredeep neural nets successfully learn a concept to 100% accuracy while the corresponding Neural

Original: Neural Tangent Kernel (NTK) only achieves trivial performance, hence showing the importance offeature learning .
Revised: Tangent Kernel (NTK) only achieves trivial performance, hence showing the importance of featurelearning .

Original: We also construct a counterexample to the “deep only works if shallow is good”principle of (48), demonstrating a case where a deep network can get near-perfect accuracy evenwhen greedy layerwise training (e.g. Belilovsky et al.
Revised: We also construct a counterexample to the “deep only works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating a case where a deep network can get near-perfectaccuracy even when greedy layerwise training (a la Belilovsky et al.

Original: (13)) cannot beat trivial performance.
Revised: (2019)) cannot beat trivialperformance.

Original: Byproviding positive theory and empirics which elude these simplified explanations of SGD, we hope topoint the way to a more complete understanding of learning dynamics in the challenging cases whereno apparent progress is made for extended periods of time.
Revised: By providing positive theory and empirics which elude these simplified explanationsof SGD, we hope to point the way to a more complete understanding of learning dynamics in thechallenging cases where no apparent progress is made for extended periods of time.

Original: 1.2 Related work
Revised: 1.2 Related work

Original: We present the most directly related work  on feature learning, and learning parities with neural nets.
Revised: We present the most directly related work below, on feature learning and learning parities with neuralnets.

Original: A broader discussion  can be found in Appendix A.3.
Revised: A broader discussion of related work can be found in Appendix A.3.

Original: SGD and feature learning.
Revised: SGD and feature learning.

Original: Theoretical analysis of gradient descent over neural networks isnotoriously hard, due to the non-convex nature of the optimization problem.
Revised: Theoretical analysis of gradient descent over neural-networks is notoriously hard, due to the non-convex nature of the optimization problem.

Original: That said, it has beenestablished that in some settings, the dynamics of GD keep the weights close to their initialization,thus behaving like  convex optimization over the Neural Tangent Kernel (see, for example, (38, 7, 22)).
Revised: That said, it has beenestablished that in some settings, the dynamics of GD keep the weights close to their initialization,thus behaving like a convex optimization over the Neural Tangent Kernel (see for example Jacot et al.(2018); Allen-Zhu et al.

Original:  In contrast, it has been shown that in various tasks, moving away from the fixed features of the NTKis essential for the successof neural networks trained with GD (for example (71, 6, 69) and the reviewin (50)).
Revised: Du et al. In contrast, it has been shown that in various tasks,moving away from the fixed features of the NTK is essential for the success neural networks trainedwith GD (for example Yehudai and Shamir (2019); Allen-Zhu and Li (2019); Wei et al. (2019) andthe review in Malach et al.

Original: These results demonstrate that feature learning is an important part of the GD optimizationprocess.
Revised: These results demonstrate that feature learning is an importantpart of the GD optimization process.

Original: Our work also focuses on a setting where feature learning is essential for the target task.
Revised: Our work also focuses on a setting where feature learning isessential for the target task.

Original: Inour theoretical analysis, we show that the initial population gradient encodes the relevant features forthe problem.
Revised: In our theoretical analysis, we show that the initial population gradientencodes the relevant features for the problem.

Original: The importance of the first gradient step for feature learning has been recently studiedin (12).
Revised: The importance of the first gradient step for featurelearning has been recently studied in Ba et al.

Original: Learning parities with neural networks.
Revised: Learning parities with neural networks.

Original: The problem of learning parities using neural networkshas been investigated in prior works from various perspectives.
Revised: The problem of learning parities using neural networkshas been investigated in prior works from various perspectives.

Original: It has been demonstrated that paritiesare hard for gradient-based algorithms, using similar arguments as in the SQ analysis (63, 1).
Revised: It has been demonstrated that paritiesare hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartzet al., 2017; Abbe and Sandon, 2020).

Original: Onepossible approach for overcoming the computational hardness is to make favorable assumptionson the input distribution.
Revised: One possible approach for overcoming the computationalhardness is to make favorable assumptions on the input distribution.

Original: Indeed, recent works show that under various assumptions on the inputdistribution, neural networks can be efficiently trained to learn parities (XORs) (20, 64, 27, 50).
Revised: Indeed, recent works show thatunder various assumptions on the input distribution, neural networks can be efficiently trained to learn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021; Frei et al., 2022; Malach et al.,2021).

Original: In contrast to these results, this work takes the approach of intentionally focusing on a hard benchmarktask, without assuming that the distribution has some favorable (namely, non-uniform) structure.
Revised: In contrast to these results, this work takes the approach of intentionally focusing on a hardbenchmark task, without assuming that the distribution has some favorable (namely, non-uniform)structure.

Original: This setting allows us to probe the performance of deep learning at a known computational limit.
Revised: This setting allows us to probe the performance of deep learning at a known computationallimit.

Original: Notably, the work of (8) provides analysis for learning polynomials (and in particular, parities) underthe uniform distribution.
Revised: Notably, the work of Andoni et al. (2014) provides analysis for learning polynomials (and inparticular, parities) under the uniform distribution.

Original: However, their main results require a network of size n O ( k ) (i.e., extremelyoverparameterized network), and provides only partial theoretical and empirical evidence for thesuccess of smaller networks.
Revised: However, their main results require a networkof size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical andempirical evidence for the success of smaller networks.

Original: Studying a related subject, some works have shown that neural networksdisplay a spectral bias, learning to fit low-frequency coefficients before high-frequency ones (57, 17).
Revised: Studying a related subject, some works haveshown that neural networks display a spectral bias, learning to fit low-frequency coefficients beforehigh-frequency ones (Rahaman et al., 2019; Cao et al., 2019).

Original:  Preliminaries
Revised:  Preliminaries

Original: We define necessary notation here; see Appendix A for more background and technical ingredients.
Revised: We define necessary notation here; see Appendix A for more background and technical ingredients.

Original: Sparse parities.
Revised: Sparse parities.

Original: For integer n ≥ 1 and non-empty S ⊆ [ n ] , the ( n, S ) - parity function χ S :{± 1 } n → {± 1 } is defined as χ S ( x ) = (cid:81) i ∈S x i .
Revised: For integer n ≥ 1 and non-empty S ⊆ [ n ] , the ( n, S ) - parity function χ S : {± 1 } n →{± 1 } is defined as χ S ( x ) = (cid:81) i ∈

Original:  We define the ( n, S ) - parity distribution D S as thedistribution over ( x, y = χ S ( x )) 4 where x is drawn from the uniform distribution over {± 1 } n , whichwe denote by Unif( {± 1 } n ) .
Revised: S x i . We define the ( n, S ) - parity distribution D S as the distributionover ( x, y = χ S ( x )) 4 where x is drawn from the uniform distribution over {± 1 } n , which we denoteby Unif( {± 1 } n ) .

Original: We define the ( n, k ) - parity learning problem as the task of recovering S using samples from D S , where S is chosen at random in (cid:0) [ n ] k (cid:1) .
Revised: We define the ( n, k ) - parity learning problem as the task of recovering S usingsamples from D S , where S is chosen at random in (cid:0) [ n ] k (cid:1) .

Original: Statistically, it is possible to doso using Θ(log (cid:0) n k (cid:1) )
Revised: Statistically, it is possible to do so usingΘ(log (cid:0) nk (cid:1) )

Original: ≈ k log n samples.
Revised: ≈ k log n samples.

Original: However, in the statistical query (SQ) model (41), (whichhas been shown to encapsulate gradient-based methods such as GD or SGD (2)), this task requiresΩ( n k ) queries (assuming constant level of noise).
Revised: However, in the statistical query (SQ) model (Kearns, 1998), (whichhas been shown to encapsulate gradient-based methods such as GD or SGD (Abbe et al., 2021)), thistask requires Ω( n k ) queries (assuming constant level of noise).

Original: While learning noiseless parities can be solved by Gaussian elimination using O ( n ) samples, learning sparse noisy parities, even at a very small noiselevel (i.e., o (1) or n − δ ) is believed to inherently require n Ω( k ) computational steps and samples, orexponential computation with n o ( k ) samples.
Revised: While learning noiseless paritiescan be solved by Gaussian elimination using O ( n ) samples, learning sparse noisy parities, even at avery small noise level (i.e., o (1) or n − δ ) is believed to inherently require n Ω( k ) computational stepsand samples, or exponential computation with n o ( k ) samples

Original: This was first explicitly conjectured by Alekhnovich(5), and has been the basis for several cryptographic schemes (e.g., (37, 9, 10, 15)).
Revised: This was first explicitly conjecturedby Alekhnovich (2003), and has been the basis for several cryptographic schemes (e.g., (Ishai et al.

Revised: ,2008; Applebaum et al., 2009, 2010; Bogdanov et al., 2019)).

Original: Notation for neural networks and training.
Revised: Notation for neural networks and training.

Original: Our main results are presented in the online learningsetting, with a stream of i.i.d.
Revised: Our main results are presented in the online learningsetting, with a stream of i.i.d.

Original: batches of examples.
Revised: batches of examples.

Original: At each iteration t = 1 , . . .
Revised: At each iteration t = 1 , . . .

Original: , T , a learningalgorithm receives a batch of B examples { ( x t,i , y t,i ) }
Revised: , T , a learningalgorithm receives a batch of B examples { ( x t,i , y t,i ) }

Original: Bi =1 drawn i.i.d.
Revised: Bi =1 drawn i.i.d.

Original: from D S , then outputs aclassifier (cid:98) y t : {± 1 } n → {± 1 } .
Revised: from D S , then outputs aclassifier (cid:98) y t : {± 1 } n → {± 1 } .

Original: We say that the algorithm solves the parity task in t steps with ϵ error,if with probability at least 1 − ϵ over both training and internal randomness, as well as ( x, y ) ∼ D S ,(cid:98)
Revised: We say that the algorithm solves the parity task in t steps with ϵ error,if with probability at least 1 − ϵ over both training and internal randomness, as well as ( x, y ) ∼ D S ,(cid:98)

Original: y t ( x ) =
Revised: y t ( x ) =

Original: y .
Revised: y .

Original: We will focus on the case that (cid:98)
Revised: We will focus on the case that (cid:98)

Original: y t
Revised: y t

Original: = sign( f ( x ; θ t )) for some parameters θ t in a continuousdomain Θ and for a continuous function f : {± 1 } n × Θ → R .
Revised: = sign( f ( x ; θ t )) for some parameters θ t in a continuousdomain Θ and for a continuous function f : {± 1 } n × Θ → R .

Original: A ubiquitous online learningalgorithm is gradient descent (GD).
Revised: A ubiquitous online learningalgorithm is gradient descent (GD).

Original: For a choice of loss function ℓ : {± 1 } × R → R , initializationθ 0 (that are chosen from some distribution), learning rate schedule { η t } Tt =

Revised: For a choice of loss function ℓ : {± 1 } × R → R , initializationθ 0 (that are chosen from some distribution), learning rate schedule { η t } Tt =

Original: ⊆
Revised: ⊆

Original: R and weight-decayschedule { λ t } Tt =1 ⊆ R , GD refers the standard iterative update rule using the regularized, empiricalloss function, which is a function of architecture f .
Revised: R and weight-decayschedule { λ t } Tt =1 ⊆ R , GD refers the standard iterative update rule using the regularized, empiricalloss function, which is a function of architecture f .

Original: The learning rate η t can also be a vector (e.g.,allowing different rate schedules for different layers).
Revised: The learning rate η t can also be a vector (e.g.,allowing different rate schedules for different layers).

Original:  Empirical findings
Revised:  Empirical findings

Original: 3.1 SGD on neural networks learns sparse parities
Revised: 3.1 SGD on neural networks learns sparse parities

Original: The central phenomenon of study in this work is the empirical observation that neural networks, withstandard initialization and training, can solve the ( n, k ) -parity problem in a number of iterationsscaling as n O ( k ) on small instances.
Revised: The central phenomenon of study in this work is the empirical observation that neural networks, withstandard initialization and training, can solve the ( n, k ) -parity problem in a number of iterationsscaling as n O ( k ) on small instances.

Original: We observed robust positive results for randomly-initialized
Revised: We observed robust positive results for randomly-initialized

Original: SGD on the following architectures, indexed by Roman numerals:
Revised: SGD on the following architectures, indexed by Roman numerals:

Original: • 2-layer MLPs: ReLU ( σ ( z ) =
Revised: • 2-layer MLPs: ReLU ( σ ( z ) =

Original: ( z ) + ) or polynomial ( σ ( z )
Revised: ( z ) + ) or polynomial ( σ ( z )

Original: =
Revised: =

Original: z k ) activation, in a wide variety ofwidth regimes r ≥ k .
Revised: z k ) activation, in a wide variety ofwidth regimes r ≥ k .

Original: Settings (i), (ii), (iii) (resp.
Revised: Settings (i), (ii), (iii) (resp.

Original: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU(resp. polynomial) activations.
Revised: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU When f ( x ; θ ) = 0 in practice (e.g. with sign initialization), we break the tie arbitrarily.

Revised: We ensure in the theoretical analysis that this does not happen.

Original:  We also consider r = k (exceptional settings (*i), (*ii) ), theminimum width for representing a k -wise parity for both activations. • Single neurons: Next, we consider non-standard activation functions σ which allow a one-neuron
Revised: [Figure] (resp. polynomial) activations. We also consider r = k (exceptional settings (*i), (*ii) ), theminimum width for representing a k -wise parity for both activations.• Single neurons: Next, we consider non-standard activation functions σ which allow a one-neuronarchitecture f ( x ; w )

Original: architecture f ( x ; w ) = σ ( w ⊤ x ) to realize k -wise parities.
Revised: = σ ( w ⊤ x ) to realize k -wise parities.

Original: The constructions stem from letting [Figure] w ∗ =
Revised: The constructions stem from lettingw ∗ =

Original: (cid:80) i ∈ S e i , and constructing σ ( · ) to interpolate (the appropriate scaling of)
Revised: (cid:80) i ∈ S e i , and constructing σ ( · ) to interpolate (the appropriate scaling of)

Original: k − w ∗⊤ xmod 2with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).
Revised: k − w ∗⊤ xmod 2with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).

Original: Going a stepfurther, a single ∞ -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.
Revised: Going a stepfurther, a single ∞ -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.

Original: We alsoremoved the second trainable layer (setting u = 1 ), obtaining settings (xi), (xii), (xiii), (xiv).
Revised: We alsoremoved the second trainable layer (setting u = 1 ), obtaining settings (xi), (xii), (xiii), (xiv).

Original: Wefound that wider architectures with these activations also trained successfully.
Revised: We found that wider architectures with these activations also trained successfully.

Original: • Transformers: Motivated by recent theoretical and empirical work on the ability of self-attention 
Revised: • Transformers: Motivated by recent theoretical and empirical work on the ability of self-attention tolearn sparse functions and parities (Lu et al., 2021; Edelman et al., 2021; Hahn, 2020), we consider

Original: to learn sparse functions and parities (47, 23, 32), we consider a simplified specialization of theTransformer architecture to this sequence classification problem.
Revised: a simplified specialization of the Transformer architecture to this sequence classification problem.

Original: This is the less-robust setting(*iii); the architecture and optimizer are described in Appendix D.1.3.
Revised: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.1.3.

Original: • PolyNets: Our final setting (xv) is the PolyNet, a slightly modified version of the parity machinearchitecture.
Revised: • PolyNets: Our final setting (xv) introduces a non-standard architecture, which may be of inde-

Original: Parity machines have been studied extensively in the statistical mechanics of MLliterature (see the related work section) as well as in a line of work on ‘neural cryptography’ (59).
Revised: pendent interest.

Original: Aparity machine outputs the sign of the product of k linear functions of the input. A PolyNet simplyoutputs the product itself.
Revised: The PolyNet takes a product over k linear functions f ( x ; w 1: k ) : = (cid:81)

Original: Both architectures can clearly realize k -sparse parities.The PolyNet
Revised: ki =1 w ⊤ i x ; this can clearly realize k -way parities.

Original: architecture was originally motivated by the search for an idealized settingwhere an end-to-end
Revised: Originally motivated by the search for an idealized setting

Original:  optimization trajectory analysis is tractable (see Section 4.1); we found in these experiments thatthis architecture trains very stably and sample-efficiently.
Revised: where an end-to-end optimization trajectory analysis is tractable (see Section 4.1), we found in these experiments that this architecture trains very stably and sample-efficiently

Original: Robust space of positive results.
Revised: Robust space of positive results.

Original: All of the networks listed above were observed to successfully learnsparse parities in a variety of settings.
Revised: All of the networks listed above were observed to successfullylearn sparse parities in a variety of settings; it would be uninformative to enumerate all of them in atable or show training curves.

Original: We summarize our findings as follows: for all combinationsof n ∈ { 10 , 20 , 30 } , k ∈ { 2 , 3 , 4 } , batch sizes B ∈ { 1 , 2 , 4 , . . . , 1024 } , initializations { uniform,
Revised: Instead, we summarize our findings as follows: for all combinationsof n ∈ { 10 , 20 , 30 } , k ∈ { 2 , 3 , 4 } , batch sizes B ∈ { 1 , 2 , 4 , . . . , 1024 } , initializations { uniform,

Original: Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture configurations{ (i) , (ii) , . .
Revised: Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture configurations{ (i) , (ii) , . .

Original: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batchof 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rate η ∈{ 0 . 001 , 0 . 01 , 0 .
Revised: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batch of 2 13samples) in at least 5 out of 10 random trials, for at least one choice of learning rate η ∈ { 0 . 01 , 0 . 1 , 1 } .

Original: The models converged in t c ≤ c · n αk ≤ 10 5 steps, for small architecturedependent constants c, α (see Appendix C).
Revised: The models converged in t c ≤ c · n αk ≤ 10 5 steps, for small architecture-dependent constants c, α ,shown in Appendix C.1.

Original: Figure 1 (left) shows some representative training curves.
Revised: Figure 1 (left) shows some representative training curves.

Original: Less robust configurations.
Revised: Less robust configurations.

Original: Settings (*i) and (*ii), where the MLP just barely represents a k -sparseparity, and the Transformer setting (*iii), are less robust to small batch sizes. In these settings, thesame positive results as above only held for sufficiently large batch sizes: B ≥ 16 . Also, setting (*iii)used the Adam optimizer; see Appendix D.1.3 for details.
Revised: The settings (*i) and (*ii), where the MLP just barely representsa k -sparse parity, and the Transformer setting (*iii), were less robust to hyperparameter choices(especially the batch size B ).

Original: Phase transitions in training curves. For almost all of the architectures, we find that that the trainingcurves exhibit phase transitions in terms of running time (and thus, in the online learning setting,dataset size as well): long durations of seemingly no progress, followed by periods of rapid decreasein the validation error. Strikingly, for architectures (v) and (vi), this plateau is absent: the error intheinitial phase appears to decrease with a linear slope. See Appendix C.8 for more plots.{+D.
+}
Revised: We present these results in a few settings, described in Appendix {+D.
+}

Original: 3.2 Random search or hidden progress?
Revised: 3.2 Random search or hidden progress?

Original: The remainder of this paper seeks to answer the question: “By what mechanism does deep learningsolve these emblematic computationally-hard optimization problems?”
Revised: The remainder of this paper seeks to answer the question: “By what mechanism does deep learningsolve these emblematic computationally-hard optimization problems?”

Original:  A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search,“bouncing around” the loss landscape in the absence of a useful gradient signal.
Revised: [Figure] A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search,“bouncing around” the loss landscape in the absence of a useful gradient signal.

Original: Upon closer inspection,several empirical observations clash with this hypothesis:
Revised: Several empiricalobservations clash with this hypothesis:

Original: [Figure] Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization,it is unclear how to account for the n Θ( k ) runtimes observed in experiments.
Revised: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization, it is unclear how to account for the n Θ( k ) runtimes observed in experiments.

Original: The initializations,which certainly do not prefer sparse functions 6 , are close to the correct solutions with probability2 − Ω( n ) ≪ n − k .
Revised: The initializations,which certainly do not prefer sparse functions 6 , are close to the correct solutions with probability2 − Ω( n ) ≪ n − k .

Original: On larger instances ( n, k ) , the power-law exponents worsen; see Figure 2 (right) , and the discussion in Appendix C.2.
Revised: On larger instances ( n, k ) , the power-law exponents worsen; see Figure 2 (right) ,and the discussion in Appendix C.4.•

Original: • No early convergence: Over a large number of random trials, no copies of this randomizedalgorithm get “lucky” (i.e. solve the problem in significantly fewer than the median number ofiterations); see Figure 2 (left) .
Revised: No early convergence: Over a large number of random trials, no copies of this randomizedalgorithm get “lucky” (i e. solve the problem in significantly fewer than the median number ofiterations); see Figure 2 (left) .

Original: The success times of random exhaustive search would be distributedas Geom(1 / (cid:0) nk (cid:1) ) , whose probability mass is highest at t = 0 and decreases monotonically with t .•
Revised: The success times of random exhaustive search would be distributedas Geom(1 / (cid:0) nk (cid:1) ) ; it is more likely to succeed (and terminate) at smaller t .•

Original: Sensitivity to initialization, not stochastic batches: Running these training setups over multiplestochastic batches from a common initialization, we find that loss curves and convergence timesare highly correlated with the architecture’s random initialization; see Figure 2 (center) .Even
Revised: Sensitivity to initialization, not stochastic batches: Running these training setups over multiplestochastic batches from a common initialization, we find that loss curves and convergence timesare highly correlated with the architecture’s random initialization; see Figure 2 (center) .

Original:  these observations, which do not probe the internal state of the algorithm, suggest that exhaustive search is an insufficient picture of the training dynamics and a different mechanism is at play
Revised: Even these observations, which do not probe the internal state of the algorithm, suggest exhaustivesearch is an insufficient picture of the training dynamics and a different mechanism is at play.

Original:  Theoretical analyses
Revised:  Theoretical analyses

Original: 4.1 Provable emergence of the parity indices in high-precision gradients
Revised: 4.1 Provable emergence of the parity indices in high-precision gradients

Original: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.
Revised: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.

Original: Our main theoretical observation is that, in many cases, the population gradient of the weights atinitialization contains enough “information” for solving the parity problem.
Revised: Our main theoretical observation is that, in many cases, the population gradient of the weights atinitialization contains enough “information” for solving the parity problem.

Original: That is, given an accurateenough estimate of the initial gradient (by e.g. computing the gradient over a large enough batchsize), the relevant subset S can be found.
Revised: That is, given an accurateenough estimate of the initial gradient (by e.g. computing the gradient over a large enough batchsize), the relevant subset S can be found.

Original: As a warm-up example, consider training a single ReLU neuron f ( x ; w ) =
Revised: As a warm-up example, consider training a single ReLU neuron f ( x ; w ) =

Original: σ ( w ⊤ x ) w.r.t.
Revised: σ ( w ⊤ x ) w.r.t.

Original: thecorrelation loss ℓ ( y, (cid:98) y )
Revised: thecorrelation loss ℓ ( y, (cid:98) y )

Original: = − y (cid:98) y , from the initialization w =
Revised: = − y (cid:98) y , from the initialization w =

Original: While a single neuron cannotexpress the parity, we observe that the population gradient can indicate what the correct subsetis: E ( x,y )
Revised: While a single neuron cannotexpress the parity, we observe that the population gradient can indicate what the correct subset is

Original: [Equation]
Revised: E ( x,y ) ∼D S [ ∇ w i ℓ ( y, f ( x ; w ))]

Original: −
Revised: =

Original: [Equation] which corresponds to either theorder- ( k − 1 )
Revised: E ( x,y ) ∼D S (cid:2) − yx i σ ′ ( w ⊤ x ) (cid:3) which corresponds to either the order-( k − 1 )

Original: Fourier coefficient S \ { i } of the function x (cid:55)→ σ ′ ( w ⊤ x ) (if i ∈ S is a relevantcoordinate) or the order- ( k + 1) coefficient S ∪ { i } (if i ̸∈ S is irrelevant).
Revised: Fourier coefficient S \ { i } of the function x (cid:55)→ σ ′ ( w ⊤ x ) (if i ∈ S is a relevant coordinate)or the order- ( k + 1) coefficient S ∪ { i } (if i ̸∈ S is irrelevant).

Original: When σ is the ReLUfunction and w =[1 , . .
Revised: When σ is the ReLU function andw =

Original: . , 1] , σ ′ ( w ⊤ x ) = sign ( (cid:80) i x i ) +1is just a shifted majority function of x .
Revised: , 1] , σ ′ ( w ⊤ x ) = sign ( (cid:80) i x i ) +1is just a shifted Majority function of x .

Original: The
Revised: For Majority, the

Original: Fourier spectrumof majority is well-understood: for even k , there is a gap between these Fouriercoefficients that is detectable using n O ( k ) samples.
Revised: Fourier spectrum is well understood: for even k , there is a gap between these Fourier coefficients thatis detectable using n O ( k ) samples.

Original: This analysis can be further extended toa ReLU neuron initialized with weights w ∈ {± 1 } n and asmall bias.
Revised: This analysis can be further extended to ReLU neuron initialized with weights w ∈ {± 1 } n and asmall bias.

Original: In fact, we can show that taking a single gradient step with large enough batch size on a
Revised: In fact, we can show that taking a single gradient step with large enough batch size on a

Original: ReLU network trained with the hinge-loss ℓ ( y, (cid:98) y ) = max { 1 − y (cid:98) y, 0 } , already finds features that cansolve the parity problem.
Revised: ReLU network trained with the square-loss, already finds features that can solve the parity problem.

Original: Using this, we show that training a ReLU MLP with SGD solves the parityproblem:
Revised: Using this, we show that training a ReLU MLP with SGD solves the parity problem:

Original: Theorem 4.
Revised: Theorem 4.

Original: Fix some ϵ ∈ (0 , 1) , let k be some even number, and assume that n is some odd numbersatisfying n ≥ Ω( k 4 log( nk/ϵ )) .
Revised: Fix some ϵ ∈ (0 , 1) , let k be some even number, and assume that n is some odd numbersatisfying n ≥ Ω( k 4 log( k/ϵ )) .

Original: There exist a symmetric random initialization scheme and learningrate and weight decay schedules s.t. for every S ⊆
Revised: There exists a symmetric random initialization scheme, learningrate and weight-decay schedules, s.t. for every S ⊆

Original: [ n ] of size k , training a ReLU MLP of sizer
Revised: [ n ] of size k , training a ReLU MLP of sizer

Original: = O (2 k k log( k/ϵ )) with batch size B = n O ( k ) log 2 ( r/ϵ ) for T = poly( k, r, 1 /ϵ ) iterations on D S w.r.t.
Revised: = O (2 k k log( k/ϵ )) with batch-size B = n O ( k ) log(1 /ϵ ) for T = poly( k, r, 1 /ϵ ) iterations on D Sw.r.t.

Original: the hinge loss, finds a network f
Revised: the square-loss, finds a network f

Original: ( x ; θ t ) with expected loss: E [ ℓ ( f ( x ; θ t ) , y )] ≤ ϵ , wherethe expectation is over the randomness of initialization, training and sampling ( x, y ) ∼ D S .
Revised: ( x ; θ t ) with expected loss: E [ ℓ ( f ( x ; θ t ) , y )] ≤ ϵ , where theexpectation is over the randomness of initialization, training and sampling ( x, y ) ∼ D S .

Original: The analysis presented above shows that when the batch size scales with n O ( k ) , SGD over MLP solvesthe parity problem.
Revised: The analysis presented above shows that when the batch size scales with n O ( k ) , SGD over MLP solvesthe parity problem.

Original: However, in our experimental setting, the number of steps , and not the batch size,scales with n O ( k ) .
Revised: However, in our experimental setting, the number of steps , and not the batch size,scales with n O ( k ) .

Original: While we believe that running SGD with small batch size and small learning rateessentially amplifies the signal in the population gradient, behaving similarly to performing a largestep over a large batch size, we do not have a complete analysis for training MLPs with SGD in thesmall batch size regime.
Revised: While we believe that running SGD with small batch size and small learning rateessentially amplifies the signal in the population gradient, behaving similarly to performing a largestep over a large batch size, we do not have a complete analysis for training MLPs with SGD in thesmall batch size regime.

Original: To complement the above result, in Section 4.2 we analyze the trajectory fora variant of the PolyNet architecture trained with gradient flow.
Revised: To complement the above result, in Section 4.2 we analyze the trajectory fora variant of the PolyNet architecture trained with gradient flow.

Original: We note that, while the above analysis applies for ReLU MLPs with a specific initialization scheme, asimilar feature emergence phenomenon can be observed in a broader set of architectures and setting.
Revised: We note that, while the above analysis applies for ReLU MLPs with a specific initialization scheme, asimilar feature emergence phenomenon can be observed in a broader set of architectures and setting.

Original: Indeed, for feature emergence to occur, we only require that there is a “gap” between the relevant andirrelevant Fourier coefficients.
Revised: Indeed, for feature emergence to occur, we only require that there is a “gap” between the relevant andirrelevant Fourier coefficients.

Original: Formally, denote (cid:98) f
Revised: Formally, denote (cid:98) f

Original: ( S ) :
Revised: ( S ) :

Original: =
Revised: =

Original: E [ f ( x ) χ S ( x )] the Fourier coefficient of fat S , and observe the following definition:
Revised: E [ f ( x ) χ S ( x )] the Fourier coefficient of fat S , and observe the following definition:

Original: Definition 1 (Fourier gap) .
Revised: Definition 1 (Fourier gap) .

Original: For some function f : {± 1 } n → R and some subset S of size k , wesay that f has a γ - Fourier gap at S if (1) for every ( k − 1)
Revised: For some function f : {± 1 } n → R and some subset S of size k , wesay that f has a γ - Fourier gap at S if (1) for every ( k − 1)

Original: -element subset S ′ ⊆ S , it holds that| (cid:98) f ( S ′ )
Revised: -element subset S ′ ⊆ S , it holds that| (cid:98) f ( S ′ )

Original: | ≥ γ , and (2) for every subset S ′ ⊆
Revised: | ≥ γ , and (2) for every subset S ′ ⊆

Original: [ n ] of size k + 1 it holds that | (cid:98) f ( S ′ )
Revised: [ n ] of size k + 1 it holds that | (cid:98) f ( S ′ )

Original: | ≤ γ/ 2 .
Revised: | ≤ γ/ 2 .

Original: Now, given a network architecture where some neuron has a γ -Fourier gap with respect to the targetsubset S , we can generalize the result of the ReLU neuron.
Revised: Now, given a network architecture where some neuron has a γ -Fourier gap with respect to the targetsubset S , we can generalize the result of the ReLU neuron.

Original: That is, we show that the subset S can bedetermined by observing an estimate of the population gradient at initialization:
Revised: That is, we show that the subset S can bedetermined by observing an estimate of the population gradient at initialization:

Original: Proposition 5.
Revised: Proposition 5.

Original: Let σ be some activation function and let ℓ be some loss function.
Revised: Let σ be some activation function and let ℓ be some loss function.

Original: Denote f ( x ; w ) =
Revised: Denote f ( x ; w ) =

Original: σ ( w ⊤ x ) .
Revised: σ ( w ⊤ x ) .

Original: Fix some subset S ⊆
Revised: Fix some subset S ⊆

Original: [ n ] .
Revised: [ n ] .

Original: Let g ∈ R n be an estimate of the population gradient suchthat (cid:13)(cid:13)
Revised: Let g ∈ R n be an estimate of the population gradient suchthat (cid:13)(cid:13)

Original: g − E ( x,y ) ∼D S [ ∇ w ℓ ( y, f ( x ; w ))] (cid:13)(cid:13) ∞
Revised: g − E ( x,y ) ∼D S [ ∇ w ℓ ( y, f ( x ; w ))] (cid:13)(cid:13) ∞

Original: ≤ γ/ 4 .
Revised: ≤ γ/ 4 .

Original: Then, for every w s.t.
Revised: Then, for every w s.t.

Original: σ ′ ( w ⊤ x ) has a γ -Fouriergap w.r.t.
Revised: σ ′ ( w ⊤ x ) has a γ -Fouriergap w.r.t.

Original: to S and ℓ ′ ( f ( x ; w ) , y ) = − y for all x , the target subset S is detected by g , namely
Revised: to S and ℓ ′ ( f ( x ; w ) , y ) = − y for all x , the target subset S is detected by g , namely

Original: S =
Revised: S =

Original: { i ∈ [ n ] :
Revised: { i ∈ [ n ] :

Original: | g i | ≥ 3 γ/ 4 } .
Revised: | g i | ≥ 3 γ/ 4 } .

Original: Comparison with NTK analysis.
Revised: Comparison with NTK analysis.

Original: In recent years, many theoretical works have studied the behaviorof SGD on neural networks through the lens of the neural tangent kernel (NTK)(38).
Revised: In recent years, many theoretical works have studied the behaviorof SGD on neural networks through the lens of the neural tangent kernel (NTK)

Revised: (Jacot et al., 2018).

Original: It is thereforeimportant to highlight the fact that the NTK (or, in fact, any kernel) cannot solve the sparse parityproblem.
Revised: It is therefore important to highlight the fact that the NTK (or, in fact, any kernel) cannot solve thesparse parity problem.

Original: The following result(see (40, 49)) shows that no kernel can achieve small error on thesparse parity problem, unless the size of the kernel is n Ω( k ) :
Revised: The following result shows that no kernel can achieve small error on the sparseparity problem, unless the size of the kernel is n Ω( k ) :

Original: Theorem 6.
Revised: Theorem 6.

Original: Let Ψ : {± 1 } n → R D be some D -dimensional embedding with sup x ∥ Ψ( x ) ∥ 2 ≤ 1 ,and let R > 0 be some number.
Revised: Let Ψ : {± 1 } n → R D be some D -dimensional embedding.

Original: If DR 2 <ϵ 2  · (cid:0) nk (cid:1) , then there exists some ( n, k ) -parity distribution 
Revised: If D < 2 ϵ · (cid:0) nk (cid:1) , then thereexists some ( n, k ) -parity distribution D S s.t. inf

Original: D S s.t. inf ∥ w ∥≤ R  E ( x,y ) ∼D S (cid:2) ℓ (Ψ( x ) ⊤ w, y ) (cid:3) > 1  − ϵ.
Revised: w ∈ R D E ( x,y ) ∼D S (cid:2) ℓ (Ψ( x ) ⊤ w, y ) (cid:3) > 1 / 2 − ϵ .

Revised: The above is a corollary of the results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022).

Original: 4.2 Disjoint-PolyNet: an idealized architecture for trajectory analyses
Revised: 4.2 PolyNet: an idealized architecture for trajectory analyses

Original: In this section, we present an idealized architecture (a version of PolyNets (xv)) that exhibits similarbehavior to MLPs (experimentally) and is technically easier to analyze. 
Revised: In this section, we present an idealized architecture (a version of PolyNets (xv)) that exhibits similarbehavior as MLPs (experimentally) and is technically easier to analyze. More specifically, we [Figure] [Figure]

Original: More specifically, we considerthe disjoint-PolyNet which takes a product over k linear functions where the linear functions arecomputed on k disjoint partitions of the input P 1 , . . .
Revised: [Figure] ( r [Figure] )  21e 0.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: c l a ss i f

Revised: i c a t i o n e rr o r m

Revised: [Figure]

Revised: [Figure]

Revised: m

Revised: [Figure]

Revised: 4 1e m =

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: m =iteration t consider the disjoint PolyNet which takes a product over k linear functions where the linear functionsare computed on k disjoint partitions of the input P 1 , . .

Original: , P k with each P i =
Revised: , P k with each P i =

Original: { ( i − 1) n ′ + 1 , . ..
Revised: { ( i − 1) n ′ +1 , . .

Original: , in ′ }with n ′ = n/k 7 , that is, f ( x ; w 1: k ) :
Revised: , im }with n ′ = n/k 7 , that is, f ( x ; w 1: k ) :

Original: = (cid:81) k
Revised: = (cid:81) ki

Original: i =1 w ⊤ i x P i .
Revised: =1 w ⊤ i x P i .

Original: As noted in the related work section, this isequivalent to a tree parity machine but with real-valued rather than ± 1 output.

Original: In order for the disjoint-PolyNet to be able to express the class of parity problems, we assume thatthe set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to each n divisible by k .
Revised: In order for the disjoint-PolyNet to be able to express the class of parity problems, we assume thatthe set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to eachdisjoint partition, that is, for all i ∈

Original: [Figure] disjoint partition, that is, for all i ∈ [ k ] , S ∩ P i = 1 .
Revised: [ k ] , S ∩ P i = 1 .

Original: We refer to this problem as the ( n, k ) -disjointparity problem.
Revised: We refer to this problem as the ( n, k ) -disjointparity problem.

Original: Note that there are still  ( n ′ )
Revised: Note that there are still m k = ( n/k )

Original: k
Revised: k

Original: = ( n/k ) k different possibilities for set S under thisrestriction.
Revised: different possibilities for set S under thisrestriction.

Original: For fixed k , these represent a constant portion of the (cid:0) nk (cid:1)
Revised: For fixed k , these represent a constant portion of the (cid:0) nk (cid:1)

Original: ≈ ( ne/k ) k
Revised: ≈ ( ne/k ) k

Original: (by Stirling’sapproximation) possibilities for S in the general non-disjoint case.
Revised: (by Stirling’sapproximation) possibilities for S in the general non-disjoint case.

Original: Consider training a disjoint-PolyNet w.r.t.
Revised: Consider training a disjoint-PolyNet w.r.t.

Original: the correlation loss.
Revised: the correlation loss.

Original: WLOG, let S = { 1 , n ′ + 1 , . . .
Revised: WLOG, let S = { 1 , n ′ + 1 , . . .

Original: , ( k −1) n ′ + 1 } and e 1 = (1 , 0 , .. 0) .
Revised: , ( k −1) n ′ + 1 } and e 1 = (1 , 0 , .. 0) .

Original: The population gradient is non-zero at i iff i ∈ S : g i ( w 1: k ) =
Revised: The population gradient is non-zero at i iff i ∈ S : g i ( w 1: k ) =

Original: E[ ∇ w i ℓ ( f ( x ; w 1: k ) , y )] = − E (cid:104) y (cid:16)(cid:81) j ̸ = i w ⊤ j x P j (cid:17) x
Revised: E

Original: P i (cid:105) =
Revised: [Equation] ∇

Original: − (cid:16)(cid:81)
Revised: [Equation]

Original: j ̸ = i w j, 1 (cid:17) e 1 .
Revised: [Equation]

Original: Now we consider the gradient flow dynamics of disjoint-PolyNet, which provide a mathematicallytractable case study for the trajectory of sparse parity learning.
Revised: Now we consider the gradient flow dynamics of disjoint-PolyNet, which provide a mathematicallytractable case study for the trajectory of sparse parity learning.

Original: For each i ∈ [ k ] , in this sectionwe treat w i as a function from R ≥ 

Revised: For each i ∈ [ k ] , in this sectionwe treat w i as a function from R ≥

Original: → R n ′ which satisfies the following differential equation:˙ w i = − g i ( w 1: k ( t )) .
Revised: → R m which satisfies the following differential equation˙ w i = − g i ( w 1: k ( t )) .

Original: For clarity of exposition, assume all-ones initialization.
Revised: For clarity of exposition, assume all-ones initialization.

Original: Then all of the relevantweights { w i, 1 : i ∈ [ k ] } follow the same trajectory, which we denote by v : R ≥ 

Revised: Then all of the relevantweights { w i, 1 : i ∈ [ k ] } follow the same trajectory, which we denote by v : R ≥

Original: → R .
Revised: → R .

Original: By analyzingthe resulting differential equations, we can formally exhibit “phase transition”-like behavior in thefully deterministic gradient flow setting. Theorem 7 (Gradient flow on disjoint-PolyNets) . Suppose k ≥ 3 . Let T ( . 49) be the time it takesfor error to fall below .
Revised: Theorem 7.

Original: 49 , and let T (0) be the time it takes to reach zero error. Then T ( . 49) T (0) = 1  −O (cid:0) ( n ′ ) 1 − k/ 2 (cid:1) .
Revised: The error at time t is 1 / 2 − ( v ( t ) / √ m ) ck for some constant c > 0 .

Original: In other words, the network takes much longer to reach slightly better than trivial accuracy than ittakes to go from slightly better than trivial to perfect accuracy.
Revised: Thus, the trajectory of error appears to display a sharper “phase transition” than that of v ( t ) .

Original: We can also analyze the trajectory of disjoint-PolyNets trained with online SGD, confirming that aneural network trained with batch size 1 SGD can learn k -sparse parities within n O ( k ) iterations. Theorem 8 (SGD on disjoint-PolyNets) . Suppose we train a disjoint-PolyNet, initialized as above,with online SGD.
Revised: Theorem 8.

Original: Then there exists an adaptive learning rate schedule such that for any ϵ > 0 , withprobability .99, the error falls below  ϵ within ˜ O (cid:0) ( n ′ ) (2 k − 1) log(1 /ϵ ) (cid:1) steps.
Revised: For k ≥ 3 and ϵ > 0 , the fraction of the time it takes for the error to fall below 12 − ϵ is1 − ˜ O (cid:0) 1 ϵ 1 − 2 /k m k/ 2 − 1 (cid:1) of the running time required to achieve zero error.

Original: Extended versions of these theorems, along with proofs, can be found in Appendix B.3.
Revised: In other words, for sufficiently large m , almost no progress is visible in the accuracy for most of therunning time.

Original:  Hidden progress: discussion and additional experiments
Revised:  Hidden progress: discussion and additional experiments

Original: In this section, we advocate for sparse parities as an idealized testbed for understanding algorithmsand phenomena in modern deep learning.
Revised: In this section, we advocate for sparse parities as an idealized testbed for understanding algorithmsand phenomena in modern deep learning.

Original: These are accompanied by experimental vignettes whichare auxiliary to the core results from Section 3, with more systematic studies deferred to future work.
Revised: These are accompanied by experimental vignettes whichare auxiliary to the core results from Section 3, with more systematic studies deferred to future work.

Original: Details are given in Appendix D.
Revised: Details are given in Appendix D.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: A progress measure for parity.
Revised: A progress measure for parity.

Original: To begin, the black-box experiments in Section 3 suggest thatrandom search is the incorrect model of SGD’s behavior in this setting.
Revised: To begin, the black-box experiments in Section 3 suggest thatrandom search is the incorrect model of SGD’s behavior in this setting.

Original: Using the theoretical insight that amplifying a precise initial population gradient suffices for learning parities, we construct onepossible progress measure, which is a function of the sequence of weights w 0 , . . .
Revised: Using the theoretical insightthat amplifying a precise initial population gradient suffices for learning parities, we construct onepossible progress measure, which is a function of the sequence of weights w 0 , . . .

Original: w t ∈ R n so far:ρ ( w 0: t ) :
Revised: w t ∈ R n so far:ρ ( w 0: t ) :

Original: =
Revised: =

Original: ∥ w t − w 0 ∥ ∞ , using the fact that w t − w 0 is an estimate for the initial population gradientin the linearized setting.
Revised: ∥ w t − w 0 ∥ ∞ , using the fact that w t − w 0 is an estimate for the initial population gradientin the linearized setting.

Original: Figure 3 shows how gradual weight movement (and thus, progress) can behidden behind plateauing losses.
Revised: Figure 3 shows how gradual weight movement (and thus, progress) can behidden behind plateauing losses.

Original: Roles of overparameterization vs. oversampling .
Revised: Roles of overparameterization vs. oversampling .

Original: An interesting consequence of our analysis is thatit illuminates scaling behaviors with respect to a third fundamental resource parameter: model size ,which we study in terms of network width r .
Revised: An interesting consequence of our analysis is thatit illuminates scaling behaviors with respect to a third fundamental resource parameter: model size ,which we study in terms of network width r .

Original: If SGD operated by a “random search” mechanism, onewould expect width to provide a parallel speedup.
Revised: If SGD operated by a “random search” mechanism, onewould expect width to provide a parallel speedup.

Original: Instead, we SGD sequentially amplifies progress.
Revised: Instead, we SGD sequentially amplifies progress.

Original: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD does not give(1 /r) × speedups; more directly, in Figure 4 (left) , convergence times for sparse parities empiricallyplateau at large model sizes.
Revised: The sharp lower tails in Figure 2 (left) imply that running r identical copies of SGD does not give× 1 /r speedups; more directly, in Figure 4 (left) , convergence times for sparse parities empiricallyplateau at large model sizes.

Original: It is a significant challenge to understand the interactions between network depth and computation,and largely outside the scope of this work.
Revised: It is a significant challenge to understand the interactions between network depth and computation,and largely outside the scope of this work.

Original: However, in Appendix C.7, we provide a brief note onusing parities and polynomial-activation MLPs to construct a simple counterexample to the “deeponly works if shallow is good” principle of (48), demonstrating a case where a deep network can getnear-perfect accuracy even when greedy layerwise training (e.g. (13)) cannot beat trivial performance.
Revised: However, in Appendix C.3, we provide a brief note onusing parities and polynomial-activation MLPs to construct a simple counterexample to the “deeponly works if shallow is good” principle of Malach and Shalev-Shwartz (2019), demonstrating acase where a deep network can get near-perfect accuracy even when greedy layerwise training (e.g.(Belilovsky et al., 2019)) cannot beat trivial performance.

Original: Learning and grokking in the finite-sample multi-pass setting.
Revised: Learning and grokking in the finite-sample multi-pass setting.

Original: The main theoretical and empiricalresults in this work consider online learning algorithms which couple the resources of training timeand independent samples.
Revised: The main theoretical and empiricalresults in this work consider online learning algorithms which couple the resources of training timeand independent samples.

Original: However, due to the computational-statistical gap in parity learning, thesepositive results are suboptimal in terms of sample efficiency.
Revised: However, due to the computational-statistical gap in parity learning, thesepositive results are suboptimal in terms of sample efficiency.

Original: We find that minibatch SGD (withweight decay) can empirically solve sparse parities, even from a sample of size m ≪ n k .
Revised: We find that minibatch SGD (with weightdecay) can empirically solve sparse parities, even from a sample of size m ≪ n k .

Original: For smallvalues of m , we reliably observe the grokking phenomenon (55): overfitting for a long time, thena delayed phase transition for the generalization error; see the two center panels of Figure 4 (right) .
Revised: For small values ofm , we reliably observe the grokking phenomenon (Power et al., 2022): overfitting for a long time, thena delayed phase transition for the generalization error; see the two center panels of Figure 4 (right) .

Original:  Conclusion
Revised:  Conclusion

Original: This work puts forward parity learning as a stylized test case to explore some of the puzzling featuresof the role of computational (as opposed to statistical) resources in deep learning.
Revised: This work puts forward parity learning as a stylized test case to explore some of the puzzling featuresof the role of computational (as opposed to statistical) resources in deep learning.

Original: These includediscontinuous improvements (a.k.a.
Revised: These includediscontinuous improvements (a.k.a.

Original: emergent capabilities ), feature learning, and universality ofarchitectures.
Revised: emergent capabilities ), feature learning, and universality ofarchitectures.

Original: In particular, we show that deep learning on parities exhibits a phase transition behavior,that it is successfully learned by a variety of deep-net architectures, and that this success cannot beexplained as a “random exhaustive search”, nor through frameworks such as the neural tangent kernelor layer-by-layer learning.
Revised: In particular, we show that deep learning on parities exhibits a phase transition behavior,that it is successfully learned by a variety of deep-net architectures, and that this success cannot beexplained as a “random exhaustive search”, nor through frameworks such as the neural tangent kernelor layer-by-layer learning.

Original: However, there are more experimental and theoretical questions, even for this simplified case of paritylearning.
Revised: However, there are more experimental and theoretical questions, even for this simplified case of paritylearning.

Original: Our focus in this work was on the online learning case, where training time and samplesarise in tandem.
Revised: Our focus in this work was on the online learning case, where training time and samplesarise in tandem.

Original: However, we believe it would be instructive to investigate parity learning when threeresources of samples, time, and model size are scaled separately.
Revised: However, we believe it would be instructive to investigate parity learning when threeresources of samples, time, and model size are scaled separately.

Original: Some very preliminary findingsalong these lines are presented in Section 3.
Revised: Some very preliminary findingsalong these lines are presented in Section 3.

Original: Extending our theoretical results to the small-batch setting, as well as to more architectures, is anopen problem.
Revised: Extending our theoretical results to the small-batch setting, as well as to more architectures, is anopen problem.

Original: Resolving it would require a better understanding of anti-concentration (lower boundon deviation from mean) of Fourier coefficients, a phenomenon that is much less studied than theconcentration of these coefficients.
Revised: Resolving it would require a better understanding of anti-concentration (lower boundon deviation from mean) of Fourier coefficients, a phenomenon that is much less studied than theconcentration of these coefficients.

Original: We would also want to extend the analysis beyond parities totasks that are not aligned with the elementary basis such as low-rank tensor recovery.
Revised: We would also want to extend the analysis beyond parities totasks that are not aligned with the elementary basis such as low-rank tensor recovery.

Original: Another important follow-up direction is understanding the extent that these insights extend fromparity learning to real-world problems, as well as the extent into which non-synthetic tasks (in,e.g., natural language processing and program synthesis) embed within them parity-like subtasks ofexhaustive combinatorial search.
Revised: Another important follow-up direction is understanding the extent that these insights extend fromparity learning to real-world problems, as well as the extent into which non-synthetic tasks (in,e.g., natural language processing and program synthesis) embed within them parity-like subtasks ofexhaustive combinatorial search.

Original: Broader impact.
Revised: Broader impact.

Original: This work seeks to contribute to the foundational understanding of computationalscaling behaviors in deep learning.
Revised: This work seeks to contribute to the foundational understanding of computationalscaling behaviors in deep learning.

Original: Our theoretical and empirical analyses are in a heavily-idealized
Revised: Our theoretical and empirical analyses are in a heavily-idealizedsynthetic problem setting.

Revised: Hence, we see no direct societal impacts of these results in this study.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type
Revised: g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]