Original: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Revised: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Original: Abstract
Revised: Abstract

Original: There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times.
Revised: There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times.

Original: While there are some accounts of how these resources modulate statistical capacity, far less is known about their effect on the computational problem of model training.
Revised: While there are some accounts of how these resources modulate statistical capacity, far less is known about their effect on the computational problem of model training.

Original: This work conducts such an exploration through the lens of learning a k -sparse parity of n bits, a canonical discrete search problem which is statistically easy but computationally hard.
Revised: This work conducts such an exploration through the lens of learning a k -sparse parity of n bits, a canonical discrete search problem which is statistically easy but computationally hard.

Original: Empirically, we ﬁnd that a variety of neural networks successfully learn sparse parities, with discontinuous phase transitions in the training curves.
Revised: Empirically, we ﬁnd that a variety of neural networks successfully learn sparse parities, with discontinuous phase transitions in the training curves.

Original: On small instances, learning abruptly occurs at approximately n O ( k ) iterations; this nearly matches SQ lower bounds, despite the apparent lack of a sparse prior.
Revised: On small instances, learning abruptly occurs at approximately n O ( k ) iterations; this nearly matches SQ lower bounds, despite the apparent lack of a sparse prior.

Original: Our theoretical analysis shows that these observations are not explained by a Langevin-like mechanism, whereby SGD “stumbles in the dark” until it ﬁnds the hidden set of features (a natural algorithm which also runs in n O ( k ) time).
Revised: Our theoretical analysis shows that these observations are not explained by a Langevin-like mechanism, whereby SGD “stumbles in the dark” until it ﬁnds the hidden set of features (a natural algorithm which also runs in n O ( k ) time).

Original: Instead, we show that SGD gradually ampliﬁes the sparse solution via a
Revised: Instead, we show that SGD gradually ampliﬁes the sparse solution via a

Original: Fourier gap in the population gradient, making continual progress that is invisible to loss and error metrics.
Revised: Fourier gap in the population gradient, making continual progress that is invisible to loss and error metrics.

Original:  Introduction
Revised:  Introduction

Original: In deep learning, performance improvements are frequently observed upon simply scaling up resources (such as data, model size and training time).
Revised: In deep learning, performance improvements are frequently observed upon simply scaling up resources (such as data, model size and training time).

Original: While these improvements are often continuous in terms of these resources, some of the most surprising recent advances in the ﬁeld have been emergent capabilities : at a certain threshold, behavior changes qualitatively and discontinuously .
Revised: While these improvements are often continuous in terms of these resources, some of the most surprising recent advances in the ﬁeld have been emergent capabilities : at a certain threshold, behavior changes qualitatively and discontinuously .

Original: Through a statistical lens, it is well-understood that larger models, trained with more data, can ﬁt more complex and expressive functions.
Revised: Through a statistical lens, it is well-understood that larger models, trained with more data, can ﬁt more complex and expressive functions.

Original: However, far less is known about the analogous computational question: how does the scaling of these resources inﬂuence the success of gradient-based optimization?
Revised: However, far less is known about the analogous computational question: how does the scaling of these resources inﬂuence the success of gradient-based optimization?

Original: These phase transitions cannot be explained via statistical capacity alone: they can appear even when the amount of data remains ﬁxed, with only model size or training time increasing.
Revised: These phase transitions cannot be explained via statistical capacity alone: they can appear even when the amount of data remains ﬁxed, with only model size or training time increasing.

Original: A timely example is the emergence of reasoning and few-shot learning capabilities when scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022); Srivastava et al.
Revised: A timely example is the emergence of reasoning and few-shot learning capabilities when scaling up language models (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022); Srivastava et al.

Original: (2022) identify various tasks which language models are only able to solve if they are larger than a critical scale.
Revised: (2022) identify various tasks which language models are only able to solve if they are larger than a critical scale.

Original: Power et al.
Revised: Power et al.

Original: (2022) give examples of discontinuous improvements in population accuracy (“grokking”) when running time increases, while dataset and model sizes remain ﬁxed.
Revised: (2022) give examples of discontinuous improvements in population accuracy (“grokking”) when running time increases, while dataset and model sizes remain ﬁxed.

Original: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: In this work, we analyze the computational aspects of scaling in deep learning, in an elementary synthetic setting which already exhibits discontinuous improvements.
Revised: In this work, we analyze the computational aspects of scaling in deep learning, in an elementary synthetic setting which already exhibits discontinuous improvements.

Original: Speciﬁcally, we consider the supervised learning problem of learning a sparse parity : the label is the parity (XOR) of k ⌧ n bits in a random length- n binary string.
Revised: Speciﬁcally, we consider the supervised learning problem of learning a sparse parity : the label is the parity (XOR) of k ⌧ n bits in a random length- n binary string.

Original: This problem is computationally difﬁcult for a range of algorithms, including gradient-based (Kearns, 1998) and streaming (Kol et al., 2017) algorithms.
Revised: This problem is computationally difﬁcult for a range of algorithms, including gradient-based (Kearns, 1998) and streaming (Kol et al., 2017) algorithms.

Original: We focus on analyzing the resource measure of training time , and demonstrate that the loss curves for sparse parities display a phase transition across a variety of architectures and hyperparameters (see
Revised: We focus on analyzing the resource measure of training time , and demonstrate that the loss curves for sparse parities display a phase transition across a variety of architectures and hyperparameters (see

Original: Figure 1, left).
Revised: Figure 1, left).

Original: Strikingly, we observe that SGD ﬁnds the sparse subset (and hence, reaches 0 error) with a variety of activation functions and initialization schemes, even with no over-parameterization.
Revised: Strikingly, we observe that SGD ﬁnds the sparse subset (and hence, reaches 0 error) with a variety of activation functions and initialization schemes, even with no over-parameterization.

Original: A natural hypothesis to explain SGD’s success in learning parities, with no visible progress in error and loss for most of training, would be that it simply “stumbles in the dark”, performing random search for the unknown target (e.g. via stochastic gradient Langevin dynamics).
Revised: A natural hypothesis to explain SGD’s success in learning parities, with no visible progress in error and loss for most of training, would be that it simply “stumbles in the dark”, performing random search for the unknown target (e.g. via stochastic gradient Langevin dynamics).

Original: If that were the case, we might expect to observe a convergence time of 2 ⌦ ( n ) , like a naive search over parameters or subsets of indices.
Revised: If that were the case, we might expect to observe a convergence time of 2 ⌦ ( n ) , like a naive search over parameters or subsets of indices.

Original: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: the convergence time adapts to the sparsity parameter k , with a scaling of n O ( k ) on small instances.
Revised: However, Figure 1 (right) , already provides some evidence against this “random search” hypothesis: the convergence time adapts to the sparsity parameter k , with a scaling of n O ( k ) on small instances.

Original: Notably, such a convergence rate implies that SGD is closer to achieving the optimal computation time among a natural class of algorithms (namely, statistical query algorithms).
Revised: Notably, such a convergence rate implies that SGD is closer to achieving the optimal computation time among a natural class of algorithms (namely, statistical query algorithms).

Original: Through an extensive empirical analysis of the scaling behavior of a variety of models, as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.
Revised: Through an extensive empirical analysis of the scaling behavior of a variety of models, as well as theoretical analysis, we give strong evidence against the “stumbling in the dark” viewpoint.

Original: Instead, there is a hidden progress measure under which SGD is steadily improving.
Revised: Instead, there is a hidden progress measure under which SGD is steadily improving.

Original: Furthermore, and perhaps surprisingly, we show that SGD achieves a computational runtime much closer to the optimal SQ lower bound than simply doing (non-sparse) parameter search.
Revised: Furthermore, and perhaps surprisingly, we show that SGD achieves a computational runtime much closer to the optimal SQ lower bound than simply doing (non-sparse) parameter search.

Original: More generally, our investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources: we identify phase transitions when varying data, model size, and training time.
Revised: More generally, our investigations reveal a number of notable phenomena regarding the dependence of SGD’s performance on resources: we identify phase transitions when varying data, model size, and training time.

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: SGD learns sparse parities.
Revised: SGD learns sparse parities.

Original: It is known from SQ lower bounds that with a constant noise level, gradient descent on any architecture requires at least n ⌦ ( k ) computational steps to learn k -sparse n -dimensional parities (for background, see Appendix A).
Revised: It is known from SQ lower bounds that with a constant noise level, gradient descent on any architecture requires at least n ⌦ ( k ) computational steps to learn k -sparse n -dimensional parities (for background, see Appendix A).

Original: We ﬁrst show a wide variety of positive empirical results, in which neural networks successfully solve the parity problem in a number of iterations which scales near this computational limit:
Revised: We ﬁrst show a wide variety of positive empirical results, in which neural networks successfully solve the parity problem in a number of iterations which scales near this computational limit:

Original: Empirical Finding 1.
Revised: Empirical Finding 1.

Original: For all small instances ( n  30 , k  4 ) of the sparse parity problem, architectures A 2 { 2-layer MLPs, Transformers 1 , sinusoidal/oscillating neurons, PolyNets 2 } , initial- izations in { uniform, Gaussian, Bernoulli } , and batch sizes 1  B  1024 , SGD on A solves the ( n, k ) -sparse parity problem (w.p.  0 . 2 ) within at most c · n ↵ k steps, for small constants c, ↵ .
Revised: For all small instances ( n  30 , k  4 ) of the sparse parity problem, architectures A 2 { 2-layer MLPs, Transformers 1 , sinusoidal/oscillating neurons, PolyNets 2 } , initial- izations in { uniform, Gaussian, Bernoulli } , and batch sizes 1  B  1024 , SGD on A solves the ( n, k ) -sparse parity problem (w.p.  0 . 2 ) within at most c · n ↵ k steps, for small constants c, ↵ .

Original: With a smaller range of hyperparameters
Revised: With a smaller range of hyperparameters

Original: Theoretical analyses of sparse feature emergence.
Revised: Theoretical analyses of sparse feature emergence.

Original: Our empirical results suggest that, in a number of computational steps matching the SQ limit, SGD is able to solve the parity problem and identify the inﬂuential coordinates, without an explicit sparse prior.
Revised: Our empirical results suggest that, in a number of computational steps matching the SQ limit, SGD is able to solve the parity problem and identify the inﬂuential coordinates, without an explicit sparse prior.

Original: We give a theoretical analysis which validates this claim.
Revised: We give a theoretical analysis which validates this claim.

Original: Informal Theorem 2.
Revised: Informal Theorem 2.

Original: On 2-layer MLPs of width 2 ⇥ ( k ) , and with batch size n O ( k ) , SGD converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most 2 O ( k ) · poly(1 / ✏ ) iterations.
Revised: On 2-layer MLPs of width 2 ⇥ ( k ) , and with batch size n O ( k ) , SGD converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most 2 O ( k ) · poly(1 / ✏ ) iterations.

Original: We also present a stronger analysis for an idealized architecture (which we call the disjoint-PolyNet ), which allows for any batch size, and captures the phase transitions observed in the error curves.
Revised: We also present a stronger analysis for an idealized architecture (which we call the disjoint-PolyNet ), which allows for any batch size, and captures the phase transitions observed in the error curves.

Original: Informal Theorem 3.
Revised: Informal Theorem 3.

Original: On disjoint-PolyNets, SGD (with any batch size B  1 ) converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most n O ( k ) · log(1 / ✏ ) iterations.
Revised: On disjoint-PolyNets, SGD (with any batch size B  1 ) converges with high probability to a solution with at most ✏ error on the ( n, k ) -parity problem in at most n O ( k ) · log(1 / ✏ ) iterations.

Original: Continuous-time gradient ﬂow exhibits a phase transition: it spends a 1  o (1) fraction of its time before convergence with error  49% .
Revised: Continuous-time gradient ﬂow exhibits a phase transition: it spends a 1  o (1) fraction of its time before convergence with error  49% .

Original: Our theoretical and empirical results hold in non-overparameterized regimes (including with a widthsinusoidal neuron), in which no ﬁxed kernel, including the neural tangent kernel (NTK)
Revised: Our theoretical and empirical results hold in non-overparameterized regimes (including with a widthsinusoidal neuron), in which no ﬁxed kernel, including the neural tangent kernel (NTK)

Original: (Jacot et al., 2018), is sufﬁciently expressive to ﬁt all sparse parities with a large margin.
Revised: (Jacot et al., 2018), is sufﬁciently expressive to ﬁt all sparse parities with a large margin.

Original: Thus, our ﬁndings comprise an elementary example of combinatorial feature learning : SGD can only successfully converge by learning a low-width sparse representation.
Revised: Thus, our ﬁndings comprise an elementary example of combinatorial feature learning : SGD can only successfully converge by learning a low-width sparse representation.

Original: Further empirical explorations.
Revised: Further empirical explorations.

Original: Building upon our core positive results, we provide a wide variety of preliminary experiments, showing sparse parity learning to be a versatile testbed for understanding the challenges and surprises in solving combinatorial problems with neural networks.
Revised: Building upon our core positive results, we provide a wide variety of preliminary experiments, showing sparse parity learning to be a versatile testbed for understanding the challenges and surprises in solving combinatorial problems with neural networks.

Original: These include quantities which reveal the continual hidden progress behind uninformative training curves (as predicted by the theory), experiments at small sample sizes which exhibit grokking (Power et al., 2022), as well as an example where greedy layer-wise learning is impossible but end-to-end SGD can learn the layers jointly.
Revised: These include quantities which reveal the continual hidden progress behind uninformative training curves (as predicted by the theory), experiments at small sample sizes which exhibit grokking (Power et al., 2022), as well as an example where greedy layer-wise learning is impossible but end-to-end SGD can learn the layers jointly.

Original: 1.2 Related work
Revised: 1.2 Related work

Original: We present the most directly related work on feature learning, and learning parities with neural nets.
Revised: We present the most directly related work on feature learning, and learning parities with neural nets.

Original: A broader discussion can be found in Appendix A.3.
Revised: A broader discussion can be found in Appendix A.3.

Original: SGD and feature learning.
Revised: SGD and feature learning.

Original: Theoretical analysis of gradient descent on neural networks is notoriously hard, due to the non-convex nature of the optimization problem.
Revised: Theoretical analysis of gradient descent on neural networks is notoriously hard, due to the non-convex nature of the optimization problem.

Original: That said, it has been established that in some settings, the dynamics of GD keep the weights close to their initialization, thus behaving like convex optimization over the Neural Tangent Kernel (see, for example, (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2018)).
Revised: That said, it has been established that in some settings, the dynamics of GD keep the weights close to their initialization, thus behaving like convex optimization over the Neural Tangent Kernel (see, for example, (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2018)).

Original: In contrast, it has been shown that in various tasks, moving away from the ﬁxed features of the NTK is essential for the success of neural networks trained with GD (for example (Yehudai and Shamir, 2019; Allen-Zhu and Li, 2019; Wei et al., 2019) and the review in (Malach et al., 2021)).
Revised: In contrast, it has been shown that in various tasks, moving away from the ﬁxed features of the NTK is essential for the success of neural networks trained with GD (for example (Yehudai and Shamir, 2019; Allen-Zhu and Li, 2019; Wei et al., 2019) and the review in (Malach et al., 2021)).

Original: These results demonstrate that feature learning is an important part of the GD optimization process.
Revised: These results demonstrate that feature learning is an important part of the GD optimization process.

Original: Our work also focuses on a setting where feature learning is essential for the target task.
Revised: Our work also focuses on a setting where feature learning is essential for the target task.

Original: In our theoretical analysis, we show that the initial population gradient encodes the relevant features for the problem.
Revised: In our theoretical analysis, we show that the initial population gradient encodes the relevant features for the problem.

Original: The importance of the ﬁrst gradient step for feature learning has been recently studied in (Ba et al., 2022).
Revised: The importance of the ﬁrst gradient step for feature learning has been recently studied in (Ba et al., 2022).

Original: Learning parities with neural networks.
Revised: Learning parities with neural networks.

Original: The problem of learning parities using neural networks has been investigated in prior works from various perspectives.
Revised: The problem of learning parities using neural networks has been investigated in prior works from various perspectives.

Original: It has been demonstrated that parities are hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartz et al., 2017; Abbe and Sandon, 2020).
Revised: It has been demonstrated that parities are hard for gradient-based algorithms, using similar arguments as in the SQ analysis (Shalev-Shwartz et al., 2017; Abbe and Sandon, 2020).

Original: One possible approach for overcoming the computational hardness is to make favorable assumptions on the input distribution.
Revised: One possible approach for overcoming the computational hardness is to make favorable assumptions on the input distribution.

Original: Indeed, recent works show that under various assumptions on the input distribution, neural networks can be efﬁciently trained to learn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;
Revised: Indeed, recent works show that under various assumptions on the input distribution, neural networks can be efﬁciently trained to learn parities (XORs) (Daniely and Malach, 2020; Shi et al., 2021;

Original: Frei et al., 2022; Malach et al., 2021).
Revised: Frei et al., 2022; Malach et al., 2021).

Original: In contrast to these results, this work takes the approach of intentionally focusing on a hard benchmark task, without assuming that the distribution has some favorable (namely, non-uniform) structure.
Revised: In contrast to these results, this work takes the approach of intentionally focusing on a hard benchmark task, without assuming that the distribution has some favorable (namely, non-uniform) structure.

Original: This setting allows us to probe the performance of deep learning at a known computational limit.
Revised: This setting allows us to probe the performance of deep learning at a known computational limit.

Original: Notably, the work of Andoni et al.
Revised: Notably, the work of Andoni et al.

Original: (2014) provides analysis for learning polynomials (and in particular, parities) under the uniform distribution.
Revised: (2014) provides analysis for learning polynomials (and in particular, parities) under the uniform distribution.

Original: However, their main results require a network of size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical and empirical evidence for the success of smaller networks.
Revised: However, their main results require a network of size n O ( k ) (i.e., extremely overparameterized network), and provides only partial theoretical and empirical evidence for the success of smaller networks.

Original: Studying a related subject, some works have shown that neural networks display a spectral bias, learning to ﬁt low-frequency coefﬁcients before high-frequency ones (Rahaman et al., 2019; Cao et al., 2019).
Revised: Studying a related subject, some works have shown that neural networks display a spectral bias, learning to ﬁt low-frequency coefﬁcients before high-frequency ones (Rahaman et al., 2019; Cao et al., 2019).

Original:  Preliminaries
Revised:  Preliminaries

Original: We provide an expanded discussion of background and related work in Appendix A.
Revised: We provide an expanded discussion of background and related work in Appendix A.

Original: Sparse parities.
Revised: Sparse parities.

Original: For integer n  1 and non-empty set S ✓ [ n ] , the ( n, S ) - parity function  S : {± 1 } n ! {± 1 } is deﬁned as  S ( x ) =
Revised: For integer n  1 and non-empty set S ✓ [ n ] , the ( n, S ) - parity function  S : {± 1 } n ! {± 1 } is deﬁned as  S ( x ) =

Original: Q i 2 S x i .
Revised: Q i 2 S x i .

Original: We deﬁne the ( n, S ) - parity distribution D S as the joint distribution over ( x, y ) 3 where x is drawn from Unif( {± 1 } n ) , the uniform distribution over random length- n sign vectors, and y :=
Revised: We deﬁne the ( n, S ) - parity distribution D S as the joint distribution over ( x, y ) 3 where x is drawn from Unif( {± 1 } n ) , the uniform distribution over random length- n sign vectors, and y :=

Original:  S ( x ) is the product of the inputs at the indices given by the “relevant features” S (thus, ± 1 , depending on whether the number of relevant  1 inputs is even or odd).
Revised:  S ( x ) is the product of the inputs at the indices given by the “relevant features” S (thus, ± 1 , depending on whether the number of relevant  1 inputs is even or odd).

Original: We deﬁne the ( n, k ) - parity learning problem as the task of recovering the S using samples from D S , where S is chosen at random from  [ n ] k  .
Revised: We deﬁne the ( n, k ) - parity learning problem as the task of recovering the S using samples from D S , where S is chosen at random from  [ n ] k  .

Original: A key fact about parities is that they are orthogonal under the correlation inner product: for S 0 ✓ [ n ] ,
Revised: A key fact about parities is that they are orthogonal under the correlation inner product: for S 0 ✓ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: That is, a learner who guesses indices S 0 cannot use correlations (equivalently, the accuracy of the hypothesis  S 0 ) as feedback to reveal which indices in S 0 are correct, unless S 0 is exactly the correct subset.
Revised: That is, a learner who guesses indices S 0 cannot use correlations (equivalently, the accuracy of the hypothesis  S 0 ) as feedback to reveal which indices in S 0 are correct, unless S 0 is exactly the correct subset.

Original: This notion of indistinguishability leads to a computational lower bound in the statistical query (SQ) model (Kearns, 1998): ⌦ ( n k ) constant-noise queries are necessary, which is far greater than the statistical limit of ⇥ (log  nk  )
Revised: This notion of indistinguishability leads to a computational lower bound in the statistical query (SQ) model (Kearns, 1998): ⌦ ( n k ) constant-noise queries are necessary, which is far greater than the statistical limit of ⇥ (log  nk  )

Original: ⇡ k log n samples.
Revised: ⇡ k log n samples.

Original: The hardness of parity has been used to derive computational hardness results for other settings, like agnostically learning halfspaces (Klivans and Kothari, 2014) and MLPs (Goel et al., 2019).
Revised: The hardness of parity has been used to derive computational hardness results for other settings, like agnostically learning halfspaces (Klivans and Kothari, 2014) and MLPs (Goel et al., 2019).

Original: Beyond the restricted computational model of statistical queries, noiseless parities can be learned in poly( n ) time via Gaussian elimination.
Revised: Beyond the restricted computational model of statistical queries, noiseless parities can be learned in poly( n ) time via Gaussian elimination.

Original: However, learning sparse noisy parities, even at a very small noise level (i.e., o (1) or n   ), is believed to inherently require n ⌦ ( k ) computational steps.
Revised: However, learning sparse noisy parities, even at a very small noise level (i.e., o (1) or n   ), is believed to inherently require n ⌦ ( k ) computational steps.

Original: In all, learning sparse parities is a well-studied combinatorial problem which exempliﬁes the computational difﬁculty of learning a joint dependence on multiple relevant features.
Revised: In all, learning sparse parities is a well-studied combinatorial problem which exempliﬁes the computational difﬁculty of learning a joint dependence on multiple relevant features.

Original: Notation for neural networks and training.
Revised: Notation for neural networks and training.

Original: Our main results are presented in the online learning setting, with a stream of i.i.d.
Revised: Our main results are presented in the online learning setting, with a stream of i.i.d.

Original: batches of examples.
Revised: batches of examples.

Original: At each iteration t = 1 , . . .
Revised: At each iteration t = 1 , . . .

Original: , T , a learning algorithm
Revised: , T , a learning algorithm

Original: A receives a batch of B examples { ( x t,i , y t,i ) }
Revised: A receives a batch of B examples { ( x t,i , y t,i ) }

Original: Bi =1 drawn i.i.d.
Revised: Bi =1 drawn i.i.d.

Original: from D S , then outputs a classiﬁer b y t : {± 1 } n ! {± 1 } .
Revised: from D S , then outputs a classiﬁer b y t : {± 1 } n ! {± 1 } .

Original: We say that A solves the parity task in t steps (with error ✏ ) if
Revised: We say that A solves the parity task in t steps (with error ✏ ) if

Original: [Equation]
Revised: [Equation]

Original: We will focus on the case that b y t = sign( f ( x ; ✓ t )) for some parameters ✓ t in a continuous domain ⇥ and for a continuous function f : {± 1 } n ⇥ ⇥ !
Revised: We will focus on the case that b y t = sign( f ( x ; ✓ t )) for some parameters ✓ t in a continuous domain ⇥ and for a continuous function f : {± 1 } n ⇥ ⇥ !

Original: R 5 , updated with the ubiquitous online learning algorithm of gradient descent (GD), whose update rule is given by
Revised: R 5 , updated with the ubiquitous online learning algorithm of gradient descent (GD), whose update rule is given by

Original: [Equation]
Revised: [Equation]

Original: for a loss function ` : {± 1 } ⇥ R !
Revised: for a loss function ` : {± 1 } ⇥ R !

Original: R , learning rate schedule { ⌘ t } Tt =1 , and weight decay schedule {  t } Tt =1 6 .
Revised: R , learning rate schedule { ⌘ t } Tt =1 , and weight decay schedule {  t } Tt =1 6 .

Original: The initialization ✓ 0 is drawn randomly from a chosen distribution.
Revised: The initialization ✓ 0 is drawn randomly from a chosen distribution.

Original:  Empirical ﬁndings 3.1 SGD on neural networks learns sparse parities
Revised:  Empirical ﬁndings 3.1 SGD on neural networks learns sparse parities

Original: The central phenomenon of study in this work is the empirical observation that neural networks, with standard initialization and training, can solve the ( n, k ) -parity problem in a number of iterations scaling as n O ( k ) on small instances.
Revised: The central phenomenon of study in this work is the empirical observation that neural networks, with standard initialization and training, can solve the ( n, k ) -parity problem in a number of iterations scaling as n O ( k ) on small instances.

Original: We observed robust positive results for randomly-initialized
Revised: We observed robust positive results for randomly-initialized

Original: SGD on the following architectures, indexed by Roman numerals:
Revised: SGD on the following architectures, indexed by Roman numerals:

Original: • 2-layer MLPs: ReLU (  ( z ) =
Revised: • 2-layer MLPs: ReLU (  ( z ) =

Original: ( z ) + ) or polynomial (  ( z )
Revised: ( z ) + ) or polynomial (  ( z )

Original: =
Revised: =

Original: z k ) activation, in a wide variety of width regimes r  k .
Revised: z k ) activation, in a wide variety of width regimes r  k .

Original: Settings (i), (ii), (iii) (resp.
Revised: Settings (i), (ii), (iii) (resp.

Original: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU (resp.
Revised: (iv), (v), (vi)) use r = { 10 , 100 , 1000 } ReLU (resp.

Original: polynomial) activations.
Revised: polynomial) activations.

Original: We also consider r = k (exceptional settings (*i), (*ii) ), the minimum width for representing a k -wise parity for both activations.
Revised: We also consider r = k (exceptional settings (*i), (*ii) ), the minimum width for representing a k -wise parity for both activations.

Original: • 1-neuron networks: Next, we consider non-standard activation functions  which allow a one-neuron architecture f ( x ; w ) =
Revised: • 1-neuron networks: Next, we consider non-standard activation functions  which allow a one-neuron architecture f ( x ; w ) =

Original:  ( w > x ) to realize k -wise parities.
Revised:  ( w > x ) to realize k -wise parities.

Original: The constructions stem from letting w ⇤ =
Revised: The constructions stem from letting w ⇤ =

Original: P i
Revised: P i

Original: 2 S e i , and constructing  ( · ) to interpolate (the appropriate scaling of)
Revised: 2 S e i , and constructing  ( · ) to interpolate (the appropriate scaling of)

Original: k  w ⇤> xmod 2 with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).
Revised: k  w ⇤> xmod 2 with a piecewise linear k -zigzag activation (vii), or a degree- k polynomial (viii).

Original: Going a step further, a single 1 -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.
Revised: Going a step further, a single 1 -zigzag (ix) or sinusoidal (x) neuron can represent all k -wise parities.

Original: In settings (xi), (xii), (xiii), (xiv), we remove the second trainable layer (setting u = 1 ).
Revised: In settings (xi), (xii), (xiii), (xiv), we remove the second trainable layer (setting u = 1 ).

Original: We ﬁnd that wider architectures with these activations also train successfully.
Revised: We ﬁnd that wider architectures with these activations also train successfully.

Original: • Transformers: There is growing interest in using parity as a benchmark for combinatorial function learning, long-range dependency learning, and length generalization in Transformers (Lu et al., 2021;
Revised: • Transformers: There is growing interest in using parity as a benchmark for combinatorial function learning, long-range dependency learning, and length generalization in Transformers (Lu et al., 2021;

Original: Edelman et al., 2021; Hahn, 2020; Anil et al., 2022; Liu et al., 2022).
Revised: Edelman et al., 2021; Hahn, 2020; Anil et al., 2022; Liu et al., 2022).

Original: Motivated by these recent theoretical and empirical works, we consider a simpliﬁed specialization of the Transformer architecture to this sequence classiﬁcation problem.
Revised: Motivated by these recent theoretical and empirical works, we consider a simpliﬁed specialization of the Transformer architecture to this sequence classiﬁcation problem.

Original: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.
Revised: This is the less-robust setting (*iii); the architecture and optimizer are described in Appendix D.

Original: 3. • PolyNets: Our ﬁnal setting (xv) is the PolyNet, a slightly modiﬁed version of the parity machine architecture.
Revised: 3. • PolyNets: Our ﬁnal setting (xv) is the PolyNet, a slightly modiﬁed version of the parity machine architecture.

Original: Parity machines have been studied extensively in the statistical mechanics of ML literature (see the related work section) as well as in a line of work on ‘neural cryptography’
Revised: Parity machines have been studied extensively in the statistical mechanics of ML literature (see the related work section) as well as in a line of work on ‘neural cryptography’

Original: (Rosen-Zvi et al., 2002).
Revised: (Rosen-Zvi et al., 2002).

Original: A parity machine outputs the sign of the product of k linear functions of the input.
Revised: A parity machine outputs the sign of the product of k linear functions of the input.

Original: A PolyNet simply outputs the product itself.
Revised: A PolyNet simply outputs the product itself.

Original: Both architectures can clearly realize ksparse parities.
Revised: Both architectures can clearly realize ksparse parities.

Original: The PolyNet architecture was originally motivated by the search for an idealized setting where an end-to-end optimization trajectory analysis is tractable (see Section 4.1); we found in these experiments that this architecture trains very stably and sample-efﬁciently.
Revised: The PolyNet architecture was originally motivated by the search for an idealized setting where an end-to-end optimization trajectory analysis is tractable (see Section 4.1); we found in these experiments that this architecture trains very stably and sample-efﬁciently.

Original: Robust space of positive results.
Revised: Robust space of positive results.

Original: All of the networks listed above were observed to successfully learn sparse parities in a variety of settings.
Revised: All of the networks listed above were observed to successfully learn sparse parities in a variety of settings.

Original: We summarize our ﬁndings as follows: for all combinations of n 2 { 10 , 20 , 30 } , k 2 { 2 , 3 , 4 } , batch sizes B 2 { 1 , 2 , 4 , . .
Revised: We summarize our ﬁndings as follows: for all combinations of n 2 { 10 , 20 , 30 } , k 2 { 2 , 3 , 4 } , batch sizes B 2 { 1 , 2 , 4 , . .

Original: . , 1024 } , initializations { uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture conﬁgurations { (i) , (ii) , . .
Revised: . , 1024 } , initializations { uniform, Gaussian, Bernoulli } , loss functions { hinge, square, cross entropy } , and architecture conﬁgurations { (i) , (ii) , . .

Original: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batch of 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rate ⌘ 2 { 0 . 001 , 0 . 01 , 0 .
Revised: , (xv) } , SGD solved the parity problem (with 100% accuracy, validated on a batch of 2 13 samples) in at least 20% of 25 random trials, for at least one choice of learning rate ⌘ 2 { 0 . 001 , 0 . 01 , 0 .

Original: The models converged in t c  c · n ↵ k  10 5 steps, for small architecture-dependent constants c, ↵ (see Appendix C).
Revised: The models converged in t c  c · n ↵ k  10 5 steps, for small architecture-dependent constants c, ↵ (see Appendix C).

Original: Figure 1 (left) shows some representative training curves.
Revised: Figure 1 (left) shows some representative training curves.

Original: Less robust conﬁgurations.
Revised: Less robust conﬁgurations.

Original: Settings (*i) and (*ii), where the MLP just barely represents a k -sparse parity, and the Transformer setting (*iii), are less robust to small batch sizes.
Revised: Settings (*i) and (*ii), where the MLP just barely represents a k -sparse parity, and the Transformer setting (*iii), are less robust to small batch sizes.

Original: In these settings, the same positive results as above only held for sufﬁciently large batch sizes: B  16 .
Revised: In these settings, the same positive results as above only held for sufﬁciently large batch sizes: B  16 .

Original: Also, setting (*iii) used the Adam optimizer (which is standard for Transformers); see Appendix D.1.3 for details.
Revised: Also, setting (*iii) used the Adam optimizer (which is standard for Transformers); see Appendix D.1.3 for details.

Original: Phase transitions in training curves.
Revised: Phase transitions in training curves.

Original: For almost all of the architectures, we ﬁnd that that the training curves exhibit phase transitions in terms of running time (and thus, in the online learning setting, dataset size as well): long durations of seemingly no progress, followed by periods of rapid decrease in the validation error.
Revised: For almost all of the architectures, we ﬁnd that that the training curves exhibit phase transitions in terms of running time (and thus, in the online learning setting, dataset size as well): long durations of seemingly no progress, followed by periods of rapid decrease in the validation error.

Original: Strikingly, for architectures (v) and (vi), this plateau is absent: the error in the initial phase appears to decrease with a linear slope.
Revised: Strikingly, for architectures (v) and (vi), this plateau is absent: the error in the initial phase appears to decrease with a linear slope.

Original: See Appendix C.8 for more plots.
Revised: See Appendix C.8 for more plots.

Original: 3.2 Random search or hidden progress?
Revised: 3.2 Random search or hidden progress?

Original: The remainder of this paper seeks to answer the question: “By what mechanism does deep learning solve these emblematic computationally-hard optimization problems?”
Revised: The remainder of this paper seeks to answer the question: “By what mechanism does deep learning solve these emblematic computationally-hard optimization problems?”

Original: A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search, “bouncing around” the loss landscape in the absence of a useful gradient signal.
Revised: A natural hypothesis would be that SGD somehow implicitly performs Monte Carlo random search, “bouncing around” the loss landscape in the absence of a useful gradient signal.

Original: Upon closer inspection, several empirical observations clash with this hypothesis:
Revised: Upon closer inspection, several empirical observations clash with this hypothesis:

Original: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization, it is unclear how to account for the runtimes observed in experiments, which adapt to the sparsity k .
Revised: • Scaling of convergence times: Without an explicit sparsity prior in the architecture or initialization, it is unclear how to account for the runtimes observed in experiments, which adapt to the sparsity k .

Original: The initializations, which certainly do not prefer sparse functions 7 , are close to the correct solutions with probability 2  ⌦ ( n ) ⌧ n  k .
Revised: The initializations, which certainly do not prefer sparse functions 7 , are close to the correct solutions with probability 2  ⌦ ( n ) ⌧ n  k .

Original: • No early convergence: Over a large number of random trials, no copies of this randomized algorithm get “lucky” (i.e. solve the problem in signiﬁcantly fewer than the median number of iterations); see Figure 2 (left) .
Revised: • No early convergence: Over a large number of random trials, no copies of this randomized algorithm get “lucky” (i.e. solve the problem in signiﬁcantly fewer than the median number of iterations); see Figure 2 (left) .

Original: The success times of random exhaustive search would be distributed as Geom(1 /  nk  ) , whose probability mass is highest at t = 0 and decreases monotonically with t .
Revised: The success times of random exhaustive search would be distributed as Geom(1 /  nk  ) , whose probability mass is highest at t = 0 and decreases monotonically with t .

Original: • Sensitivity to initialization, not SGD samples: Running these training setups over multiple stochastic batches from a common initialization, we ﬁnd that loss curves and convergence times are highly correlated with the architecture’s random initialization, and are quite concentrated conditioned on initialization; see Figure 2 (center) .
Revised: • Sensitivity to initialization, not SGD samples: Running these training setups over multiple stochastic batches from a common initialization, we ﬁnd that loss curves and convergence times are highly correlated with the architecture’s random initialization, and are quite concentrated conditioned on initialization; see Figure 2 (center) .

Original: • Elbows in the scaling curves: For larger n , the power-law scaling ceases to hold: the exponent worsens (see Figure 2 (right) , as well as the discussion in Appendix C.2).
Revised: • Elbows in the scaling curves: For larger n , the power-law scaling ceases to hold: the exponent worsens (see Figure 2 (right) , as well as the discussion in Appendix C.2).

Original: This would not be true for random exhaustive search.
Revised: This would not be true for random exhaustive search.

Original: Even these observations, which do not probe the internal state of the algorithm, suggest that exhaustive search is an insufﬁcient picture of the training dynamics, and a different mechanism is at play.
Revised: Even these observations, which do not probe the internal state of the algorithm, suggest that exhaustive search is an insufﬁcient picture of the training dynamics, and a different mechanism is at play.

Original:  Theoretical analyses 4.1 Provable emergence of the parity indices in high-precision gradients
Revised:  Theoretical analyses 4.1 Provable emergence of the parity indices in high-precision gradients

Original: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.
Revised: We now provide a theoretical account for the success of SGD in solving the ( n, k ) -parity problem.

Original: Our main theoretical observation is that, in many cases, the population gradient of the weights at initialization contains enough “information” for solving the parity problem.
Revised: Our main theoretical observation is that, in many cases, the population gradient of the weights at initialization contains enough “information” for solving the parity problem.

Original: That is, given an accurate enough estimate of the initial gradient (by e.g. computing the gradient over a large enough batch size), the relevant subset S can be found.
Revised: That is, given an accurate enough estimate of the initial gradient (by e.g. computing the gradient over a large enough batch size), the relevant subset S can be found.

Original: As a warm-up example, consider training a single ReLU neuron b y ( x ; w ) =
Revised: As a warm-up example, consider training a single ReLU neuron b y ( x ; w ) =

Original: ( w > x ) + with the correlation loss ` ( y, b y ) =
Revised: ( w > x ) + with the correlation loss ` ( y, b y ) =

Original:  y b y over D S , from an all-ones initialization w =
Revised:  y b y over D S , from an all-ones initialization w =

Original: 1] 2 R n .
Revised: 1] 2 R n .

Original: While a single neuron cannot express the parity, we observe that the correct subset can be extracted from the population gradient at initialization:
Revised: While a single neuron cannot express the parity, we observe that the correct subset can be extracted from the population gradient at initialization:

Original: [Equation]
Revised: [Equation]

Original: The key insight is that each coordinate in the above expression is a correlation between a parity and the function x 7!
Revised: The key insight is that each coordinate in the above expression is a correlation between a parity and the function x 7!

Original:  [
Revised:  [

Original: P i x i  0] , and thus a Fourier coefﬁcient of this Boolean function.
Revised: P i x i  0] , and thus a Fourier coefﬁcient of this Boolean function.

Original: At each relevant coordinate ( i 2 S ), the population gradient is the order-( k  1 )
Revised: At each relevant coordinate ( i 2 S ), the population gradient is the order-( k  1 )

Original: Fourier coefﬁcient
Revised: Fourier coefﬁcient

Original: S \ { i } ; for the irrelevant features ( i 62 S ), it is instead the order- ( k + 1) coefﬁcient S [ { i } .
Revised: S \ { i } ; for the irrelevant features ( i 62 S ), it is instead the order- ( k + 1) coefﬁcient S [ { i } .

Original: All we require is a detectable gap between these quantities.
Revised: All we require is a detectable gap between these quantities.

Original: Formally, letting f ( x ; w ) =
Revised: Formally, letting f ( x ; w ) =

Original:  ( w > x ) , letting b f ( S ) :
Revised:  ( w > x ) , letting b f ( S ) :

Original: = E [ f ( x )  S ( x )] denote the Fourier coefﬁcient of f at S , we isolate the desired property:
Revised: = E [ f ( x )  S ( x )] denote the Fourier coefﬁcient of f at S , we isolate the desired property:

Original: Deﬁnition 1 (Fourier gap) .
Revised: Deﬁnition 1 (Fourier gap) .

Original: For a function f : {± 1 } n !
Revised: For a function f : {± 1 } n !

Original: R and S ✓ [ n ] of size k , we say that f has a  - Fourier gap at S if, for every ( k  1) -element subset S  ⇢ S and ( k + 1) -element superset
Revised: R and S ✓ [ n ] of size k , we say that f has a  - Fourier gap at S if, for every ( k  1) -element subset S  ⇢ S and ( k + 1) -element superset

Original: S +  S , it holds that |
Revised: S +  S , it holds that |

Original: [Equation]
Revised: [Equation]

Original: For the all-ones initialization, observe [
Revised: For the all-ones initialization, observe [

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: is just an afﬁne transformation of the majority function of x , for which a Fourier gap can be established, with  = ⇥ ( n  ( k  1) / 2 ) .
Revised: is just an afﬁne transformation of the majority function of x , for which a Fourier gap can be established, with  = ⇥ ( n  ( k  1) / 2 ) .

Original: This arises from closed-form formulas for the Fourier spectrum of majority (see Lemma 2 in
Revised: This arises from closed-form formulas for the Fourier spectrum of majority (see Lemma 2 in

Original: Appendix B.1), a landmark result from the harmonic analysis of Boolean functions (Titsworth, 1962;
Revised: Appendix B.1), a landmark result from the harmonic analysis of Boolean functions (Titsworth, 1962;

Original: O’Donnell, 2014).
Revised: O’Donnell, 2014).

Original: Thus, the coordinates in S can be recovered from
Revised: Thus, the coordinates in S can be recovered from

Original: [Equation]
Revised: [Equation]

Original: samples; see Proposition 9 in Appendix B.2 for a formal argument.
Revised: samples; see Proposition 9 in Appendix B.2 for a formal argument.

Original: Carefully extending this insight, we obtain an end-to-end convergence result for ReLU-activation
Revised: Carefully extending this insight, we obtain an end-to-end convergence result for ReLU-activation

Original: MLP networks with a particular symmetric choice of ± 1 initialization, trained with the hinge loss:
Revised: MLP networks with a particular symmetric choice of ± 1 initialization, trained with the hinge loss:

Original: Theorem 4 (SGD on MLPs learns sparse parities) .
Revised: Theorem 4 (SGD on MLPs learns sparse parities) .

Original: Let ✏ 2 (0 , 1) .
Revised: Let ✏ 2 (0 , 1) .

Original: Let k  2 an even integer, and let n = ⌦ ( k 4 log( nk/ ✏ )) be an odd integer.
Revised: Let k  2 an even integer, and let n = ⌦ ( k 4 log( nk/ ✏ )) be an odd integer.

Original: Then, there exist a random initialization scheme, ⌘ t , and  t such that for every S ✓ [ n ] of size k , SGD on a ReLU MLP of width r = ⌦ (2 k k log( k/ ✏ )) , with batch size B = ⌦ ( n k log( n/ ✏ )) on D S with the hinge loss, outputs a network f ( x ; ✓ t ) with expectedloss E [ ` ( f ( x ; ✓ t ) , y )]  ✏ in at most O ( k 3 r 2 n/ ✏ 2 ) iterations.
Revised: Then, there exist a random initialization scheme, ⌘ t , and  t such that for every S ✓ [ n ] of size k , SGD on a ReLU MLP of width r = ⌦ (2 k k log( k/ ✏ )) , with batch size B = ⌦ ( n k log( n/ ✏ )) on D S with the hinge loss, outputs a network f ( x ; ✓ t ) with expectedloss E [ ` ( f ( x ; ✓ t ) , y )]  ✏ in at most O ( k 3 r 2 n/ ✏ 2 ) iterations.

Original: This does not capture the full range of settings in which we empirically observe successful convergence.
Revised: This does not capture the full range of settings in which we empirically observe successful convergence.

Original: First, it requires a sign vector initialization, while we observe convergence with other random initialization schemes (namely, uniform and Gaussian).
Revised: First, it requires a sign vector initialization, while we observe convergence with other random initialization schemes (namely, uniform and Gaussian).

Original: Second, it requires the batch size to scale with n ⌦ ( k ) 9 , while we also obtain positive results when B is small (even B = 1 ).
Revised: Second, it requires the batch size to scale with n ⌦ ( k ) 9 , while we also obtain positive results when B is small (even B = 1 ).

Original: Analogous statements for these cases (as well as other activations and losses) would require Fourier gaps for population gradient functions other than majority; lower bounds on the degree- ( k  1) coefﬁcients ( “Fourier anti-concentration” ) are particularly elusive in the literature, and we leave it as an open challenge to establish them in more general settings.
Revised: Analogous statements for these cases (as well as other activations and losses) would require Fourier gaps for population gradient functions other than majority; lower bounds on the degree- ( k  1) coefﬁcients ( “Fourier anti-concentration” ) are particularly elusive in the literature, and we leave it as an open challenge to establish them in more general settings.

Original: We provide preliminary empirics in Appendix C.1, suggesting that the Fourier gaps in our empirical settings are sufﬁciently large.
Revised: We provide preliminary empirics in Appendix C.1, suggesting that the Fourier gaps in our empirical settings are sufﬁciently large.

Original: Low width necessitates feature learning.
Revised: Low width necessitates feature learning.

Original: We note that in the low-width (non-overparameterized) regimes considered in this work, no ﬁxed kernel (including the neural tangent kernel (Jacot et al., 2018), whose dimensionality is the network’s parameter count) can solve the sparse parity problem.
Revised: We note that in the low-width (non-overparameterized) regimes considered in this work, no ﬁxed kernel (including the neural tangent kernel (Jacot et al., 2018), whose dimensionality is the network’s parameter count) can solve the sparse parity problem.

Original: The following is a consequence of results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022):
Revised: The following is a consequence of results in (Kamath et al., 2020; Malach and Shalev-Shwartz, 2022):

Original: Theorem 5 (Low-width NTK cannot ﬁt all parities) .
Revised: Theorem 5 (Low-width NTK cannot ﬁt all parities) .

Original: Let : {± 1 } n !
Revised: Let : {± 1 } n !

Original: R D be any D -dimensional embedding with sup
Revised: R D be any D -dimensional embedding with sup

Original: [Equation]
Revised: [Equation]

Original: Let R, ✏ > 0 , and let ` denote the 0-1 loss or hinge loss.
Revised: Let R, ✏ > 0 , and let ` denote the 0-1 loss or hinge loss.

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: , then there exists some S ✓ [ n ] of size k such that
Revised: , then there exists some S ✓ [ n ] of size k such that

Original: [Equation]
Revised: [Equation]

Original: Thus, our low-width results lie outside the NTK regime, which requires far larger models (size n ⌦ ( k ) ) to express parities.
Revised: Thus, our low-width results lie outside the NTK regime, which requires far larger models (size n ⌦ ( k ) ) to express parities.

Original: However, we note that better sample complexity bounds are possible in the NTK regime, with an algorithm more similar to standard SGD (see (Telgarsky, 2022) and Appendix A.3).
Revised: However, we note that better sample complexity bounds are possible in the NTK regime, with an algorithm more similar to standard SGD (see (Telgarsky, 2022) and Appendix A.3).

Original: 4.2 Disjoint-PolyNet: exact trajectory analysis for an idealized architecture
Revised: 4.2 Disjoint-PolyNet: exact trajectory analysis for an idealized architecture

Original: In this section, we present an architecture (a version of PolyNets (xv)) which empirically exhibits similar behavior to MLPs and bypasses the difﬁculty of analyzing Fourier gaps.
Revised: In this section, we present an architecture (a version of PolyNets (xv)) which empirically exhibits similar behavior to MLPs and bypasses the difﬁculty of analyzing Fourier gaps.

Original: The disjointPolyNet takes a product over k linear functions of an equal-sized 11 partition P 1 , . .
Revised: The disjointPolyNet takes a product over k linear functions of an equal-sized 11 partition P 1 , . .

Original: , P k of the input coordinates: f ( x ; w 1: k ) :=
Revised: , P k of the input coordinates: f ( x ; w 1: k ) :=

Original: Q ki =1 h w i , x P i i .
Revised: Q ki =1 h w i , x P i i .

Original: As noted in the Section 1.2, this is equivalent to a tree parity machine, with real-valued (instead of ± 1 ) outputs.
Revised: As noted in the Section 1.2, this is equivalent to a tree parity machine, with real-valued (instead of ± 1 ) outputs.

Original: This architecture also requires us to assume that the set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to each disjoint partition, that is, for all
Revised: This architecture also requires us to assume that the set S of size k in the ( n, k ) -parity problem is selected such that exactly one index belongs to each disjoint partition, that is, for all

Original: i 2 [ k ] ,
Revised: i 2 [ k ] ,

Original: S \ P i = 1 .
Revised: S \ P i = 1 .

Original: We refer to this problem as the ( n, k ) -disjoint parity problem.
Revised: We refer to this problem as the ( n, k ) -disjoint parity problem.

Original: Note that there are still ( n 0 )
Revised: Note that there are still ( n 0 )

Original: k
Revised: k

Original: = ( n/k ) k different possibilities for set S under this restriction.
Revised: = ( n/k ) k different possibilities for set S under this restriction.

Original: For ﬁxed k , these represent a constant fraction of the
Revised: For ﬁxed k , these represent a constant fraction of the

Original: [Equation]
Revised: [Equation]

Original: Stirling’s approximation) possibilities for S in the general non-disjoint case.
Revised: Stirling’s approximation) possibilities for S in the general non-disjoint case.

Original: Consider training a disjoint-PolyNet w.r.t.
Revised: Consider training a disjoint-PolyNet w.r.t.

Original: the correlation loss.
Revised: the correlation loss.

Original: Without loss of generality, assume that each relevant coordinate in S is the ﬁrst element P i .
Revised: Without loss of generality, assume that each relevant coordinate in S is the ﬁrst element P i .

Original: Then, the population gradient is non-zero only at indices
Revised: Then, the population gradient is non-zero only at indices

Original: i 2 S :
Revised: i 2 S :

Original: [Equation]
Revised: [Equation]

Original: This allows us to analyze the gradient ﬂow dynamics of the disjoint-PolyNet, without needing to establish Fourier gaps.
Revised: This allows us to analyze the gradient ﬂow dynamics of the disjoint-PolyNet, without needing to establish Fourier gaps.

Original: For each i 2 [ k ] , in this section we treat w i as a function from R  0 !
Revised: For each i 2 [ k ] , in this section we treat w i as a function from R  0 !

Original: R nwhich satisﬁes the following differential equation: ˙
Revised: R nwhich satisﬁes the following differential equation: ˙

Original: w i =
Revised: w i =

Original:  g
Revised:  g

Original: i ( w 1: k ( t )) .
Revised: i ( w 1: k ( t )) .

Original: For clarity of exposition, assume all-ones initialization.
Revised: For clarity of exposition, assume all-ones initialization.

Original: Then, all of the relevant weights { w i, 1 : i 2 [ k ] } follow the same trajectory.
Revised: Then, all of the relevant weights { w i, 1 : i 2 [ k ] } follow the same trajectory.

Original: By analyzing the resulting differential equations, we can formally exhibit “phase transition”-like behavior in the fully deterministic gradient ﬂow setting.
Revised: By analyzing the resulting differential equations, we can formally exhibit “phase transition”-like behavior in the fully deterministic gradient ﬂow setting.

Original: Theorem 6 (Loss plateau for gradient ﬂow on disjoint-PolyNets) .
Revised: Theorem 6 (Loss plateau for gradient ﬂow on disjoint-PolyNets) .

Original: Suppose k  3 .
Revised: Suppose k  3 .

Original: Let T ( ✏ ) denote the smallest time at which the error is at most ✏ .
Revised: Let T ( ✏ ) denote the smallest time at which the error is at most ✏ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Informally, the network takes much longer to reach better trivial accuracy than takes to trivial the trajectory of disjoint-PolyNets trained online size
Revised: Informally, the network takes much longer to reach better trivial accuracy than takes to trivial the trajectory of disjoint-PolyNets trained online size

Original: Theorem 7 (SGD on disjoint-PolyNets learns disjoint parities) .
Revised: Theorem 7 (SGD on disjoint-PolyNets learns disjoint parities) .

Original: Suppose we train a disjoint-PolyNet, initialized as above, with online SGD.
Revised: Suppose we train a disjoint-PolyNet, initialized as above, with online SGD.

Original: Then there exists an adaptive learning rate schedule such that for any ✏ > 0 , with probability 0.99, the error falls below ✏ within ˜ O  ( n 0 ) (2 k  1) log(1 / ✏ )  steps.
Revised: Then there exists an adaptive learning rate schedule such that for any ✏ > 0 , with probability 0.99, the error falls below ✏ within ˜ O  ( n 0 ) (2 k  1) log(1 / ✏ )  steps.

Original: Extended versions of these theorems, along with proofs, can be found in Appendix B.3.
Revised: Extended versions of these theorems, along with proofs, can be found in Appendix B.3.

Original:  Hidden progress: discussion and additional experiments
Revised:  Hidden progress: discussion and additional experiments

Original: So far, we have shown that sparse parity learning provides an idealized setting in which neural networks successfully learn sparse combinatorial features, with a mechanism of continual progress hiding behind discontinuous training curves.
Revised: So far, we have shown that sparse parity learning provides an idealized setting in which neural networks successfully learn sparse combinatorial features, with a mechanism of continual progress hiding behind discontinuous training curves.

Original: In this section, we outline preliminary explorations on a broader range of interesting phenomena which arise in this setting.
Revised: In this section, we outline preliminary explorations on a broader range of interesting phenomena which arise in this setting.

Original: Details are provided in
Revised: Details are provided in

Original: Appendix C, while more systematic investigations are deferred to future work.
Revised: Appendix C, while more systematic investigations are deferred to future work.

Original: Hidden progress measures for learning parities.
Revised: Hidden progress measures for learning parities.

Original: The theoretical and (black-box) empirical results suggest that SGD does not learn parities via the memoryless process of random exhaustive search.
Revised: The theoretical and (black-box) empirical results suggest that SGD does not learn parities via the memoryless process of random exhaustive search.

Original: This suggests the existence of progress measures : scalar quantities which are functions of the training algorithm’s state (i.e. the model weights w t ) and are predictive of the time to successful convergence.
Revised: This suggests the existence of progress measures : scalar quantities which are functions of the training algorithm’s state (i.e. the model weights w t ) and are predictive of the time to successful convergence.

Original: We provide some white-box investigations which further support the hypothesis of hidden progress, by examining the gradual improvement in quantities other than the training loss.
Revised: We provide some white-box investigations which further support the hypothesis of hidden progress, by examining the gradual improvement in quantities other than the training loss.

Original: In Appendix C.1, we directly plot the Fourier gaps of the population gradient, as a function of t , ﬁnding that they are large (within a small constant factor of those of majority) in practice.
Revised: In Appendix C.1, we directly plot the Fourier gaps of the population gradient, as a function of t , ﬁnding that they are large (within a small constant factor of those of majority) in practice.

Original: In Figure 3 and Appendix C.3, we examine the weight movement norm ⇢ ( w 0: t ) :
Revised: In Figure 3 and Appendix C.3, we examine the weight movement norm ⇢ ( w 0: t ) :

Original: =
Revised: =

Original: k w t  w 0 k 1 to reveal hidden progress, motivated by the fact that w t  w 0 is a linearized estimate for the initial population gradient.
Revised: k w t  w 0 k 1 to reveal hidden progress, motivated by the fact that w t  w 0 is a linearized estimate for the initial population gradient.

Original: Roles of overparameterization vs. oversampling .
Revised: Roles of overparameterization vs. oversampling .

Original: An interesting consequence of our analysis is that it illuminates scaling behaviors with respect to a third fundamental resource parameter: model size , which we study in terms of network width r .
Revised: An interesting consequence of our analysis is that it illuminates scaling behaviors with respect to a third fundamental resource parameter: model size , which we study in terms of network width r .

Original: If SGD operated by a “random search” mechanism, one would expect width to provide a parallel speedup.
Revised: If SGD operated by a “random search” mechanism, one would expect width to provide a parallel speedup.

Original: Instead, we ﬁnd that SGD sequentially ampliﬁes progress.
Revised: Instead, we ﬁnd that SGD sequentially ampliﬁes progress.

Original: The sharp lower tails in Figure 2 (left) imply that running r identical copies of
Revised: The sharp lower tails in Figure 2 (left) imply that running r identical copies of

Original: SGD does not give (1 /r ) ⇥ speedups; more directly, in Appendix C.4 (previewed in Figure 4 (left) ), we ﬁnd that convergence times for sparse parities empirically plateau at large model sizes.
Revised: SGD does not give (1 /r ) ⇥ speedups; more directly, in Appendix C.4 (previewed in Figure 4 (left) ), we ﬁnd that convergence times for sparse parities empirically plateau at large model sizes.

Original: Emergence of grokking in the ﬁnite-sample (multi-pass) setting.
Revised: Emergence of grokking in the ﬁnite-sample (multi-pass) setting.

Original: Our main results are presented in the online learning setting (fresh minibatches from D S at each iteration).
Revised: Our main results are presented in the online learning setting (fresh minibatches from D S at each iteration).

Original: While this mitigates the confounding factor of overﬁtting, it couples the resources of training time and independent samples in a suboptimal way, due to the computational-statistical gap for parity learning.
Revised: While this mitigates the confounding factor of overﬁtting, it couples the resources of training time and independent samples in a suboptimal way, due to the computational-statistical gap for parity learning.

Original: In Appendix C.5, we ﬁnd empirically that minibatch SGD (with weight decay) can learn sparse parities, even with smaller sample sizes m ⌧ n k .
Revised: In Appendix C.5, we ﬁnd empirically that minibatch SGD (with weight decay) can learn sparse parities, even with smaller sample sizes m ⌧ n k .

Original: We reliably observe the grokking phenomenon (Power et al., 2022): an initial overﬁtting phase, then a delayed phase transition in the generalization error; see the two center panels of Figure 4 (right) .
Revised: We reliably observe the grokking phenomenon (Power et al., 2022): an initial overﬁtting phase, then a delayed phase transition in the generalization error; see the two center panels of Figure 4 (right) .

Original: These results complement and corroborate the ﬁndings of Nanda and Lieberum (2022), who analyze the hidden progress of Transformers trained on arithmetic tasks (a setting which also exhibits grokking).
Revised: These results complement and corroborate the ﬁndings of Nanda and Lieberum (2022), who analyze the hidden progress of Transformers trained on arithmetic tasks (a setting which also exhibits grokking).

Original: Deeper networks.
Revised: Deeper networks.

Original: It is a signiﬁcant challenge (and generally outside the scope of this work) to understand the interactions between network depth and computational/statistical efﬁciency.
Revised: It is a signiﬁcant challenge (and generally outside the scope of this work) to understand the interactions between network depth and computational/statistical efﬁciency.

Original: In
Revised: In

Original: Appendix C.7, we show that learning parities with deeper polynomial-activation MLPs comprises a simple counterexample to the “deep only works if shallow is good” principle of Malach and
Revised: Appendix C.7, we show that learning parities with deeper polynomial-activation MLPs comprises a simple counterexample to the “deep only works if shallow is good” principle of Malach and

Original: Shalev-Shwartz (2019): a deep network can get near-perfect accuracy, even when greedy layer-wise training (e.g. (Belilovsky et al., 2019)) cannot beat trivial performance.
Revised: Shalev-Shwartz (2019): a deep network can get near-perfect accuracy, even when greedy layer-wise training (e.g. (Belilovsky et al., 2019)) cannot beat trivial performance.

Original: By providing positive theory and empirics which elude these simpliﬁed explanations of SGD, we hope to point the way to a more complete understanding of learning dynamics in the challenging cases where no apparent progress is made for extended periods of time.
Revised: By providing positive theory and empirics which elude these simpliﬁed explanations of SGD, we hope to point the way to a more complete understanding of learning dynamics in the challenging cases where no apparent progress is made for extended periods of time.

Original:  Conclusion
Revised:  Conclusion

Original: This work puts forward sparse parity learning as an elementary test case to explore the puzzling features of the role of computational (as opposed to statistical) resources in deep learning.
Revised: This work puts forward sparse parity learning as an elementary test case to explore the puzzling features of the role of computational (as opposed to statistical) resources in deep learning.

Original: In particular, we have shown that a variety of neural architectures solve this combinatorial search problem, with a number of computational steps scaling near the sparsity-dependent SQ lower bound; furthermore, the loss curves exhibit phase transitions . This success cannot be explained by a model of random exhaustive search, nor through simpliﬁed frameworks such as the neural tangent kernel or layer-by-layer learning.
Revised: In particular, we have shown that a variety of neural architectures solve this combinatorial search problem, with a number of computational steps nearly matching the sparsity-dependent SQ lower bound. Furthermore, we have shown that despite abrupt phase transitions in the loss and accuracy curves, SGD works by gradually amplifying the sparse features “under the hood”.

Original: Even in this simple setting, there are several open experimental and theoretical questions.
Revised: Even in this simple setting, there are several open experimental and theoretical questions.

Original: Our focus in this work was on the online learning case, where training time and samples arise in tandem.
Revised: This work largely focuses on the online learning case, which couples training iterations with fresh i.i.d.

Original:  We believe it would be instructive to investigate parity learning when  three resources of samples, time, and model size are scaled separately.
Revised: samples. We believe it would be instructive to investigate parity learning when the three resources of samples, time, and model size are scaled separately.

Original: Some very preliminary ﬁndings along these lines are presented in Section 3.
Revised: Some very preliminary ﬁndings along these lines are presented in Section 3.

Original: Extending our theoretical results to the small-batch setting, as well as to more architectures, is an open problem.
Revised: It is an open problem to extend our theoretical results to the small-batch setting, as well as to the full range of architectures and losses in our experiments.

Original: Resolving it would require a better understanding of  anti-concentration  of  Fourier coefﬁcients, a phenomenon that is much less studied than the  concentration of these coefﬁcients.
Revised: Resolving these questions would require a better understanding of the anti-concentration behavior of Boolean Fourier coefﬁcients, which is much less studied than the analogous concentration phenomena.

Original: Another important follow-up direction is understanding the extent to which these insights extend from parity learning to more complex (including real-world) combinatorial problem settings, as well as the extent to which non-synthetic tasks (in, e.g., natural language processing and program synthesis) embed within them parity-like subtasks of exhaustive combinatorial search.
Revised: Another important follow-up direction is understanding the extent to which these insights extend from parity learning to more complex (including real-world) combinatorial problem settings, as well as the extent to which non-synthetic tasks (in, e.g., natural language processing and program synthesis) embed within them parity-like subtasks of exhaustive combinatorial search.

Original: We hope that our results will lead to further progress towards understanding and improving the optimization dynamics behind the recent slew of dramatic empirical successes of deep learning in these types of domains.
Revised: We hope that our results will lead to further progress towards understanding and improving the optimization dynamics behind the recent slew of dramatic empirical successes of deep learning in these types of domains.

Original: Broader impact.
Revised: Broader impact.

Original: This work seeks to contribute to the foundational understanding of computational scaling behaviors in deep learning.
Revised: This work seeks to contribute to the foundational understanding of computational scaling behaviors in deep learning.

Original: Our theoretical and empirical analyses are in a heavily-idealized synthetic problem setting.
Revised: Our theoretical and empirical analyses are in a heavily-idealized synthetic problem setting.

Original: Hence, we see no direct societal impacts of the results in this study.
Revised: Hence, we see no direct societal impacts of the results in this study.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: We would like to thank Lenka Zdeborová for providing us with references to the statistical physics literature on phase transitions in the learning curves of neural networks, and
Revised: We would like to thank Lenka Zdeborová for providing us with references to the statistical physics literature on phase transitions in the learning curves of neural networks, and

Original: Matus Telgarsky for bringing to our attention the better sample complexity guarantees of 2-sparse parity learning in the NTK regime.
Revised: Matus Telgarsky for bringing to our attention the better sample complexity guarantees of 2-sparse parity learning in the NTK regime.

Original: Sham Kakade acknowledges funding from the Ofﬁce of Naval
Revised: Sham Kakade acknowledges funding from the Ofﬁce of Naval

Original: Research under award N00014-22-1-2377.
Revised: Research under award N00014-22-1-2377.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]