Original: L IMITATIONS OF THE NTK FOR U NDERSTANDING G ENERALIZATION IN D EEP L EARNING Anonymous authors
Revised: L IMITATIONS OF THE NTK FOR U NDERSTANDING G ENERALIZATION IN D EEP L EARNING Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The “Neural Tangent Kernel” (NTK) (Jacot et al., 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks.
Revised: The “Neural Tangent Kernel” (NTK) (Jacot et al., 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks.

Original: In this work, we study NTKs through the lens of scaling laws and demonstrate that they fall short of explaining important aspects of neural network generalization.
Revised: In this work, we study NTKs through the lens of scaling laws and demonstrate that they fall short of explaining important aspects of neural network generalization.

Original: In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization.
Revised: In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization.

Original: This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy.
Revised: This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy.

Original: Further, we show that even if the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling.
Revised: Further, we show that even if the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling.

Original: Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after a few epochs of training.
Revised: Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after a few epochs of training.

Original: Altogether, our work establishes concrete limitations of the NTK approach in understanding generalization of real networks on natural datasets.
Revised: Altogether, our work establishes concrete limitations of the NTK approach in understanding generalization of real networks on natural datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The seminal work of Jacot et al Jacot et al.
Revised: The seminal work of Jacot et al Jacot et al.

Original: (2018) introduced the “Neural Tangent Kernel” (NTK) as the limit of neural networks with widths approaching infinity.
Revised: (2018) introduced the “Neural Tangent Kernel” (NTK) as the limit of neural networks with widths approaching infinity.

Original: Since this limit holds provably under certain initializations, and kernels are more amenable to analysis than neural networks, the NTK promises to be a useful reduction to understand deep learning.
Revised: Since this limit holds provably under certain initializations, and kernels are more amenable to analysis than neural networks, the NTK promises to be a useful reduction to understand deep learning.

Original: Thus, it has initiated a rich research program to use the NTK to explain various behaviors of neural networks, such as convergence to global minima (Du et al., 2018; 2019), good generalization performance (Allen-Zhu et al., 2018;
Revised: Thus, it has initiated a rich research program to use the NTK to explain various behaviors of neural networks, such as convergence to global minima (Du et al., 2018; 2019), good generalization performance (Allen-Zhu et al., 2018;

Original: Arora et al., 2019a), implicit bias of networks (Tancik et al., 2020) as well as neural scaling laws (Bahri et al., 2021a).
Revised: Arora et al., 2019a), implicit bias of networks (Tancik et al., 2020) as well as neural scaling laws (Bahri et al., 2021).

Original: In addition to the infinite NTK, the emperical NTK — the kernel with features that are gradients of a finite-width neural network— can be a useful object to study, since it is an approximation to both the true neural network and the infinite NTK.
Revised: In addition to the infinite NTK, the emperical NTK — the kernel with features that are gradients of a finite-width neural network— can be a useful object to study, since it is an approximation to both the true neural network and the infinite NTK.

Original: This has also been studied extensively as a tool to understand deep learning (Fort et al., 2020; Long, 2021; Paccolat et al., 2021; Ortiz-Jim´enez et al., 2021).
Revised: This has also been studied extensively as a tool to understand deep learning (Fort et al., 2020; Long, 2021; Paccolat et al., 2021; Ortiz-Jim´enez et al., 2021).

Original: In this work, we probe the upper limits of this research program: we want to understand the extent to which understanding NTKs (empirical and infinite) can teach us about the success of neural networks.
Revised: In this work, we probe the upper limits of this research program: we want to understand the extent to which understanding NTKs (empirical and infinite) can teach us about the success of neural networks.

Original: We study this question under the lens of scaling Kaplan et al. ; Rosenfeld et al. (2019)—how performance improves as a function of samples and as a function of time— since the scaling is an important “signature” of the mechanisms underlying any learning algorithm.
Revised: We study this question under the lens of scaling (Kaplan et al., 2020; Rosenfeld et al., 2019)—how performance improves as a function of samples and as a function of time— since the scaling is an important “signature” of the mechanisms underlying any learning algorithm.

Original: We thus compare the scaling of real networks to the scaling of NTKs in the following ways.
Revised: We thus compare the scaling of real networks to the scaling of NTKs in the following ways.

Original: Data scaling of initial kernel (Section 3) : We show that both the infinite and empirical NTK (at initialization) can have worse data scaling exponents than neural networks, in realistic settings (Figure 1).
Revised: Data scaling of initial kernel (Section 3) : We show that both the infinite and empirical NTK (at initialization) can have worse data scaling exponents than neural networks, in realistic settings (Figure 1).

Original: We find that this is robust to various important hyperparameter changes such as learning rate (in the range used in practice), batchsize and optimization method.
Revised: We find that this is robust to various important hyperparameter changes such as learning rate (in the range used in practice), batchsize and optimization method.

Original: Width scaling of initial kernel (Section 3): Since neural networks provably converge to the NTK at infinite width, we investigate why the scaling behavior differs at finite width.
Revised: Width scaling of initial kernel (Section 3): Since neural networks provably converge to the NTK at infinite width, we investigate why the scaling behavior differs at finite width.

Original: We show (Figure 2(b), 2(c)) realistic settings where as the width of the neural network increases
Revised: We show (Figure 2(b), 2(c)) realistic settings where as the width of the neural network increases

Original: to very large values, the test performance of the network gets worse and approaches the performance of the infinite NTK, unlike existing results in literature which suggest that over-parameterization is always good.
Revised: to very large values, the test performance of the network gets worse and approaches the performance of the infinite NTK, unlike existing results in literature which suggest that increasing the width in the over-parameterized regime is always good.

Original: This also raises new questions about scaling of neural networks with width, in particular the “variance-limited” neural scaling regimes (Bahri et al., 2021a).
Revised: This also raises new questions about scaling of neural networks with width, in particular the “variance-limited” neural scaling regimes (Bahri et al., 2021).

Original: Data scaling of after-kernel (Section 4): We consider the after-kernel Long (2021) i e. Empirical NTK extracted after training to completion on a fixed number of samples.
Revised: Data scaling of after-kernel (Section 4): We consider the after-kernel (Long, 2021) i.e. empirical NTK extracted after training to completion on a fixed number of samples.

Original: We show
Revised: We show

Original: Figure 1(B), 
Revised: (Figure 1(B), 4(

Original: b)) that the after-kernel continues to improve as we increase the training dataset size.
Revised: b that the after-kernel continues to improve as we increase the training dataset size.

Original: On the other hand, we find (Figure 
Revised: On the other hand, we find (Figure 4(

Original: c)) that the scaling exponent of the afterkernel, extracted after training on a fixed number of samples remains worse than that of the corresponding neural network.
Revised: c) that the scaling exponent of the afterkernel, extracted after training on a fixed number of samples remains worse than that of the corresponding neural network.

Original: Time scaling (Section 5): We show
Revised: Time scaling (Section 5): We show

Original: (Figure 1(C)
Revised: (Figure 1(C)

Original: a)) realistic settings where the empirical NTK continues to improve uniformly throughout most of the training.
Revised: a)) realistic settings where the empirical NTK continues to improve uniformly throughout most of the training.

Original: This is in contrast
Revised: This is in contrast

Original: with
Revised: with

Original: prior workFort et al. ; Ortiz -Jim ´
Revised: prior work

Original:  enez et al. Atanasov et al. Long (
Revised: (Fort et al., 2020; Ortiz-Jim ´ enez et al., 2021; Atanasov et al., 2021; Long, 2021)

Original: which suggests that theempirical
Revised: which suggests that the

Original:  NTK changes rapidly in 
Revised: empirical NTK changes rapidly in the

Original: the beginning
Revised: beginning

Original: of the training followed by a slowing of this change
Revised: of the training followed by a slowing of this change.

Original: We demonstrate these phenomena occur in certain settings which are based on real, non-synthetic data, and modern architectures (for e.g.: for datasets CIFAR-10 and SVHN and convolutional networks).
Revised: We demonstrate these phenomena occur in certain settings which are based on real, non-synthetic data, and modern architectures (for e.g.: for datasets CIFAR-10 and SVHN and convolutional networks).

Original: While we do not claim that these phenomena manifest for all possible datasets and architectures, we believe that our examples highlight important limitations to the use of NTK to understand the test performance of neural networks.
Revised: While we do not claim that these phenomena manifest for all possible datasets and architectures, we believe that our examples highlight important limitations to the use of NTK to understand the test performance of neural networks.

Original: Formalizing the set of distributions or architectures for which these phenomenon occur is an important direction for future theoretical research.
Revised: Formalizing the set of distributions or architectures for which these phenomenon occur is an important direction for future theoretical research.

Original: 1. C OMPARISON TO P RIOR W ORK ON NTK G ENERALIZATION
Revised: 1. C OMPARISON TO P RIOR W ORK ON NTK G ENERALIZATION

Original: Our main focus is to understand feature learning occurring due to finite width.
Revised: Our main focus is to understand feature learning occurring due to finite width.

Original: To do this, we make the following deliberate choices in all of our experiments: a) We use the NTK parameterization, this makes sure that infinite width networks will be equivalent to kernels b)
Revised: To do this, we make the following deliberate choices in all of our experiments: a) We use the NTK parameterization, this makes sure that infinite width networks will be equivalent to kernels b)

Original: We use the same optimization setup for the neural network, Empirical NTK and Infinite NTK, this makes sure that as width tends to infinity all 3 models will have the same limit.
Revised: We use the same optimization setup for the neural network, empirical NTK and Infinite NTK, this makes sure that as width tends to infinity all 3 models will have the same limit.

Original: We make sure that our comparisons are robust by c) using scaling laws to compare these models and d) doing various hyperparameter ablations (Figure 3).
Revised: We make sure that our comparisons are robust by c) using scaling laws to compare these models and d) doing various hyperparameter ablations (Figure 3).

Original: Below we describe several lines of related works and how our work differs from them.
Revised: Below we describe several lines of related works and how our work differs from them.

Original: Small initialization and representation learning at infinite width.
Revised: Small initialization and representation learning at infinite width.

Original: Infinite widths neural networks in the NTK and standard initialization are equivalent to kernels Jacot et al. ; Yang & Hu (2021).
Revised: Infinite widths neural networks in the NTK and standard initialization are equivalent to kernels (Jacot et al., 2018; Yang & Hu, 2021).

Original: On the other hand it has been shown Yang & Hu (2021); Sirignano & Spiliopoulos (2019); Nguyen & Pham (2020); Ara´ujo et al. ; Fang et al. that with small initialization feature learning is possible at infinite width.
Revised: On the other hand it has been shown (Yang & Hu, 2021; Sirignano & Spiliopoulos, 2019; Nguyen & Pham, 2020; Ara´ujo et al., 2019; Fang et al., 2020) that with small initialization feature learning is possible at infinite width.

Original: The feature learning displayed in our experiments is not due to small initialization as we initialize our networks in the NTK parameterization.
Revised: The feature learning displayed in our experiments is not due to small initialization as we initialize our networks in the NTK parameterization.

Original: This was a deliberate choice as we are interested in feature learning occurring due to finite width as this is the kind of feature learning displayed by empirical neural networks (which usually do not have a small initialization).
Revised: This was a deliberate choice as we are interested in feature learning occurring due to finite width as this is the kind of feature learning displayed by empirical neural networks (which usually do not have a small initialization).

Original: Data Scaling for NTKs and neural networks.
Revised: Data Scaling for NTKs and neural networks.

Original: Scaling laws have been empirically shown Kaplan et al.
Revised: Scaling laws have been empirically shown Kaplan et al.

Original: ; Rosenfeld et al.
Revised: ; Rosenfeld et al.

Original: for neural networks and have been theoretically proven Bordelon et al.
Revised: for neural networks and have been theoretically proven Bordelon et al.

Original: Canatar et al.
Revised: Canatar et al.

Original: ; Bordelon & Pehlevan (2022) for NTKs under natural assumptions.
Revised: ; Bordelon & Pehlevan (2022) for NTKs under natural assumptions.

Original: Comparison between the scaling laws for neural network and empirical NTKs has been previous looked at by Paccolatet. al. Paccolat et al.
Revised: Comparison between the scaling laws for neural network and empirical NTKs has been previous looked at by Paccolat et al.

Original: (2021) and Guillermo et.

Original: al. Ortiz-Jim´enez et al.
Revised: and Ortiz-Jim´enez et al.

Original: (2021) and both find that neural networks have better scaling than empirical NTK at initialization.
Revised: (2021) and both find that neural networks have better scaling than empirical NTK at initialization.

Original: Both of these papers do not compare to infinite NTKs which leaves open the possibility that neural networks and infinite width NTKs behave the same wrt their scaling constants.
Revised: Both of these papers do not compare to infinite NTKs which leaves open the possibility that neural networks and infinite width NTKs behave the same wrt their scaling constants.

Original: Pointwise comparisons of neural networks and corresponding infinite NTKs has also been studied extensively in the literature Arora et al.
Revised: Pointwise comparisons of neural networks and corresponding infinite NTKs has also been studied extensively in the literature (Arora et al., 2019b;

Original: (2019b) ; Lee et al. Simon et al.
Revised: Lee et al., 2020; Simon et al., 2021)

Original: but the results have been divided.
Revised: but the results have been divided.

Original: As discussed earlier we focus on comparing scaling laws.
Revised: As discussed earlier we focus on comparing scaling laws.

Original: We argue that scaling laws, instead of point-wise comparisons, are the appropriate tool to compare neural networks and NTKs.
Revised: We argue that scaling laws, instead of point-wise comparisons, are the appropriate tool to compare neural networks and NTKs.

Original: Practically, pointwise comparisons between any two models can be fraught with issues as the ordering can flip depending on dataset size, as well as the specific choice of hyperparameters.
Revised: Practically, pointwise comparisons between any two models can be fraught with issues as the ordering can flip depending on dataset size, as well as the specific choice of hyperparameters.

Original: On the other hand, scaling exponents have been found to be more robust to the choice of hyperparameters (Bansal et al., 2022; Kaplan et al., 2020).
Revised: On the other hand, scaling exponents have been found to be more robust to the choice of hyperparameters (Bansal et al., 2022; Kaplan et al., 2020).

Original: More importantly, the claim that NTK can capture ”most” of the performance of the neural network can be subjective, specially when we are comparing small error or loss values.
Revised: More importantly, the claim that NTK can capture ”most” of the performance of the neural network can be subjective, specially when we are comparing small error or loss values.

Original: We show that when we look closely at the scaling exponents of these objects instead, we find major differences.
Revised: We show that when we look closely at the scaling exponents of these objects instead, we find major differences.

Original: Theoretical studied effects of finite width with respect to the NTK regime.
Revised: Theoretical studied effects of finite width with respect to the NTK regime.

Original: Finite width corrections to the NTK theory have been studied by Andreassen & Dyer (2020); Roberts et al.
Revised: Finite width corrections to the NTK theory have been studied by Andreassen & Dyer (2020); Roberts et al.

Original: Bahri et al. (2021b).
Revised: Bahri et al. (2021).

Original: While these results do not need infinite widths they still require much higher than practically used widths particularly for the training sizes used in practice.
Revised: While these results do not need infinite widths they still require much higher than practically used widths particularly for the training sizes used in practice.

Original: These papers either consider a) the finite width corrections of empirical NTK or b) they consider the change in NTK but predict that the higher order analogues of empirical NTK remain constant.
Revised: These papers either consider a) the finite width corrections of empirical NTK or b) they consider the change in NTK but predict that the higher order analogues of empirical NTK remain constant.

Original: For a) we show that the empirical NTK is very far from the performance of finite width neural networks.
Revised: For a) we show that the empirical NTK is very far from the performance of finite width neural networks.

Original: Regarding b), in
Revised: Regarding b), in

Original: Appendix D we show that the higher order analogues of empirical NTK change significantly.
Revised: Appendix D we show that the higher order analogues of empirical NTK change significantly.

Original: After Kernel and Time Dynamics We discuss these in detail in Section 4 and 5.
Revised: After Kernel and Time Dynamics We discuss these in detail in Section 4 and 5.

Original: We describe other related works in Appendix G.
Revised: We describe other related works in Appendix G.

Original:  E XPERIMENTAL M ETHODOLOGY
Revised:  E XPERIMENTAL M ETHODOLOGY

Original: Here we describe the common methodology used in our experiments.
Revised: Here we describe the common methodology used in our experiments.

Original: The core object we want to understand is the data-scaling law of real neural networks— that is, what is its asymptotic performance as a function of the number of train samples?
Revised: The core object we want to understand is the data-scaling law of real neural networks— that is, what is its asymptotic performance as a function of the number of train samples?

Original: Concretely, in this work we restrict to classification problems, where we measure performance in terms of test classification error.
Revised: Concretely, in this work we restrict to classification problems, where we measure performance in terms of test classification error.

Original: For a given classification algorithm, let L ( n ) be its learning curve : its expected test error as a function of number of samples n .
Revised: For a given classification algorithm, let L ( n ) be its learning curve : its expected test error as a function of number of samples n .

Original: In practice, many neural networks exhibit power-law decay in their learning curves, Kaplan et al.
Revised: In practice, many neural networks exhibit power-law decay in their learning curves (Kaplan et al., 2020).

Original: In such settings, we have L ( n ) ∼ αn β
Revised: In such settings, we have L ( n ) ∼ αn β

Original: and we are interested primarily in the scaling exponent β , which determines the asymptotic rate of convergence.
Revised: and we are interested primarily in the scaling exponent β , which determines the asymptotic rate of convergence.

Original: Empirical and Infinite NTK Let f ( w, x ) be a neural network with w representing the weights and x an input.
Revised: Empirical and Infinite NTK Let f ( w, x ) be a neural network with w representing the weights and x an input.

Original: By Taylor expansion around w 0 we have:
Revised: By Taylor expansion around w 0 we have:

Original: [Equation]
Revised: [Equation]

Original: Empirical NTK of the neural network around weights w 0 refers to the model g 1 ( w, x )
Revised: Empirical NTK of the neural network around weights w 0 refers to the model g 1 ( w, x )

Original: = ∇ w f ( w, x ) | w 0 ( w − w 0 ) .
Revised: = ∇ w f ( w, x ) | w 0 ( w − w 0 ) .

Original: Note that this is not the same as linearizing the network as we omit the f ( w 0 , x ) term.
Revised: Note that this is not the same as linearizing the network as we omit the f ( w 0 , x ) term.

Original: Empirical NTK is a linear model with respect to the weights w .
Revised: Empirical NTK is a linear model with respect to the weights w .

Original: Infinite NTK refers to the limit of the empirical NTK of the network around initial weights as width tends to infinity.
Revised: Infinite NTK refers to the limit of the empirical NTK of the network around initial weights as width tends to infinity.

Original: For a given learning problem and given neural network architecture NN, we want to understand its data-scaling law L NN ( n ) .
Revised: For a given learning problem and given neural network architecture NN, we want to understand its data-scaling law L NN ( n ) .

Original: We consider the infinite NTK of the NN and the empirical NTK of the
Revised: We consider the infinite NTK of the NN and the empirical NTK of the

Original: NN at initialization and their corresponding learning curves, L NTK ( n ) and L ENTK ( n ) .
Revised: NN at initialization and their corresponding learning curves, L NTK ( n ) and L ENTK ( n ) .

Original: Now we ask: is the scaling-exponent of L NN always close to the scaling-exponent of either L ENTK or L NTK , in realistic settings?
Revised: Now we ask: is the scaling-exponent of L NN always close to the scaling-exponent of either L ENTK or L NTK , in realistic settings?

Original: That is, how well does the NTK approximation capture the generalization of real networks, on natural distributions?
Revised: That is, how well does the NTK approximation capture the generalization of real networks, on natural distributions?

Original: Recall that this question is especially interesting because the three objects involved (Neural Network,
Revised: Recall that this question is especially interesting because the three objects involved (Neural Network,

Original: NTK, and ENTK) all become provably equivalent in the appropriate width → ∞ limit.
Revised: NTK, and ENTK) all become provably equivalent in the appropriate width → ∞ limit.

Original: Thus, at infinite-width we know their scaling laws must be the equivalent.
Revised: Thus, at infinite-width we know their scaling laws must be the equivalent.

Original: The question is then, how far are we from this limit in practice?
Revised: The question is then, how far are we from this limit in practice?

Original: Are the widths used in practice large enough for their scalingbehavior to be captured by the infinite-width limit?
Revised: Are the widths used in practice large enough for their scalingbehavior to be captured by the infinite-width limit?

Original: To probe these questions, we empirically study scaling laws of these methods on image-classification problems.
Revised: To probe these questions, we empirically study scaling laws of these methods on image-classification problems.

Original: Remark on comparisons.
Revised: Remark on comparisons.

Original: We intentionally only compare a neural network to its corresponding
Revised: We intentionally only compare a neural network to its corresponding

Original: NTK , and not to other kernels.
Revised: NTK , and not to other kernels.

Original: Our motivation not address the question of “can (some) kernel perform as well as as a given neural network?”
Revised: Our motivation not address the question of “can (some) kernel perform as well as as a given neural network?”

Original: — indeed, there may be some better kernel to consider than the NTK.
Revised: — indeed, there may be some better kernel to consider than the NTK.

Original: However, our goal is to study the specific kernels given by the NTK approximation, in correspondence with real networks.
Revised: However, our goal is to study the specific kernels given by the NTK approximation, in correspondence with real networks.

Original: Datasets.
Revised: Datasets.

Original: We use the following datasets:
Revised: We use the following datasets:

Original: A 2 class subset (dog, horse) ofthe CIFAR-5m Nakkiran et al. (2021) dataset, as a binary classification problem, which we denote CIFAR-5m-bin .
Revised: A 2 class subset (dog, horse) of CIFAR-5m (Nakkiran et al., 2021) dataset, as a binary classification problem, which we denote CIFAR-5m-bin .

Original: This is a dataset of synthetic but realistic 32 × 32 RGB images similar to CIFAR-10,which were generated using a generative model.
Revised: This is a dataset of synthetic but realistic 32 × 32 RGB images similar to CIFAR-10, generated using a generative model.

Original: A binary classification task on the SVHN dataset Netzer et al. (2011) with the labels being the parity of the digit, denoted by SVHN-parity .
Revised: A binary classification task on the SVHN dataset (Netzer et al., 2011) with the labels being the parity of the digit, denoted by SVHN-parity .

Original: For the training data we use a balanced random subset of ’train’ and ’extra’ partitions, for test data we use the ’test’ partition.
Revised: For the training data we use a balanced random subset of ’train’ and ’extra’ partitions, for test data we use the ’test’ partition.

Original: We focus on the CIFAR-5m-bin experiments in the main body.
Revised: We focus on the CIFAR-5m-bin experiments in the main body.

Original: Corresponding SVHN-parity experiments can be found in Appendix F.
Revised: Corresponding SVHN-parity experiments can be found in Appendix F.

Original: We use these particular datasets because we need datasets with a large number of samples in order to measure data-scaling, and CIFAR-5m-bin and the SVHN dataset both have ≥ 600k samples.
Revised: We use these particular datasets because we need datasets with a large number of samples in order to measure data-scaling, and CIFAR-5m-bin and the SVHN dataset both have ≥ 600k samples.

Original: We chose to consider binary tasks as this makes the kernel experiments computationally feasible.
Revised: We chose to consider binary tasks as this makes the kernel experiments computationally feasible.

Original: Although there are other datasets with similar sample sizes (e.g. ImageNet), the datasets we use have the advantage that they are low-resolution and an easier task— thus, scaling-law experiments are far more computationally feasible.
Revised: Although there are other datasets with similar sample sizes (e.g. ImageNet), the datasets we use have the advantage that they are low-resolution and an easier task— thus, scaling-law experiments are far more computationally feasible.

Original: We also do some experiments on a synthetic dataset in Appendix E.
Revised: We also do some experiments on a synthetic dataset in Appendix E.

Original: Architectures.
Revised: Architectures.

Original: We use the following base architectures: Myrtle CNN Page (2018); Shankar et al. (2020) for the CIFAR-5m-bin task and a 5 layer CNN with 64 channels for the SVHN-parity task.
Revised: We use the following base architectures: Myrtle CNN (Page, 2018; Shankar et al., 2020) for the CIFAR-5m-bin task and a 5 layer CNN with 64 channels for the SVHN-parity task.

Original: We consider various width scaling for these networks: For the Myrtle CNN we vary the width from 16 to 1024 and from 16 to 4096 for the 5 layer CNN.
Revised: We consider various width scaling for these networks: For the Myrtle CNN we vary the width from 16 to 1024 and from 16 to 4096 for the 5 layer CNN.

Original: See Appendix B for more details.
Revised: See Appendix B for more details.

Original: Experimental Details.
Revised: Experimental Details.

Original: We describe some subtleties in the experimental setup.
Revised: We describe some subtleties in the experimental setup.

Original: We use NTK parameterization Jacot et al. (2018) for both the neural network and the kernels as this is the parameterization used in proving the equivalence of neural network and NTK at infinite width.
Revised: We use NTK parameterization (Jacot et al., 2018) for both the neural network and the kernels as this is the parameterization used in proving the equivalence of neural network and NTK at infinite width.

Original: We train with
Revised: We train with

Original: MSE loss and ± 1 labels.
Revised: MSE loss and ± 1 labels.

Revised: We use test error as the metric for all the plots except in Appendix I where we recreate some of the most important plots for test loss.

Original: All of our networks are in the overparameterized regime i.e. are able to reach 0 train error.
Revised: All of our networks are in the overparameterized regime i.e. are able to reach 0 train error.

Original: To preserve the correspondence between the neural networks, empirical NTKs and infinite NTKs we train all of them with SGD with momentum with the same hyperparameters.
Revised: To preserve the correspondence between the neural networks, empirical NTKs and infinite NTKs we train all of them with SGD with momentum with the same hyperparameters.

Original: This also ensures that in all experiments neural networks will be trained below the critical learning rate , i.e. the learning rate at which training of the empirical and infinite NTK can converge (See also Appendix B.3).
Revised: This also ensures that in all experiments neural networks will be trained below the critical learning rate , i.e. the learning rate at which training of the empirical and infinite NTK can converge (See also Appendix B.3).

Original: Training for the empirical NTK is done by linearizing the initialized neural neural network using Novak et al.
Revised: Training for the empirical NTK is done by linearizing the initialized neural neural network using Novak et al.

Original: (2020) library while for infinite NTK we directly use SGD with momentum on the linear system given by the infinite NTK and the labels.
Revised: (2020) library while for infinite NTK we directly use SGD with momentum on the linear system given by the infinite NTK and the labels.

Original: We describe further experimental details for each individual experiment in Appendix B.
Revised: We describe further experimental details for each individual experiment in Appendix B.

Original:  D ATA S CALING L AWS OF N EURAL N ETWORKS AND NTK S IN THE O VERPARAMETERIZED R EGIME
Revised:  D ATA S CALING L AWS OF N EURAL N ETWORKS AND NTK S IN THE O VERPARAMETERIZED R EGIME

Original: In this section we compare the data-scaling laws of neural networks to their corresponding emperical
Revised: In this section we compare the data-scaling laws of neural networks to their corresponding emperical

Original: NTKs and infinite NTKs.
Revised: NTKs and infinite NTKs.

Original: Our main claim is the following.
Revised: Our main claim is the following.

Original: Claim 3.1.
Revised: Claim 3.1.

Original: There exists a natural setting of task and network architecture such that the neural network trained with SGD has a better scaling constant than its corresponding infinite and empirical
Revised: There exists a natural setting of task and network architecture such that the neural network trained with SGD has a better scaling constant than its corresponding infinite and empirical

Original: NTK at initialization.
Revised: NTK at initialization.

Original: Further, this gap in scaling continues to hold over a wide range of widths and learning rates used in practice.
Revised: Further, this gap in scaling continues to hold over a wide range of widths and learning rates used in practice.

Original: The above claim can be interpreted as stating that there exists natural settings where the regime in which real neural networks are trained is meaningfully separated from the NTK regime, and real neural networks have a better scaling law.
Revised: The above claim can be interpreted as stating that there exists natural settings where the regime in which real neural networks are trained is meaningfully separated from the NTK regime, and real neural networks have a better scaling law.

Original: In Figure 1 (A), we train a Myrtle CNN Page (2018); Shankar et al. (2020), its empirical NTK at initialization, and its infinite NTK on the CIFAR-5m-bin task.
Revised: In Figure 1 (A), we train a Myrtle CNN (Page, 2018; Shankar et al., 2020), its empirical NTK at initialization, and its infinite NTK on the CIFAR-5m-bin task.

Original: In each case, we train to fit the train set with SGD and optimal early stopping.
Revised: In each case, we train to fit the train set with SGD and optimal early stopping.

Original: We then numerically fit scaling laws, and find, scaling-exponents β of: .185 (empirical NTK), .213 (infinite NTK), .291 (neural network).
Revised: We then numerically fit scaling laws, and find, scaling-exponents β of: .185 (empirical NTK), .213 (infinite NTK), .291 (neural network).

Original: Thus, in this image-classification setting, the real neural network significantly outperforms its corresponding
Revised: Thus, in this image-classification setting, the real neural network significantly outperforms its corresponding

Original: NTKs with respect to data-scaling.
Revised: NTKs with respect to data-scaling.

Original: We show the statistical significance of this result in Appendix A.
Revised: We show the statistical significance of this result in Appendix A.

Original: See the Appendix B for full experimental details.
Revised: See the Appendix B for full experimental details.

Original: We now investigate how robust this result is to changes in the width of the architecture and optimizer, within realistic bounds.
Revised: We now investigate how robust this result is to changes in the width of the architecture and optimizer, within realistic bounds.

Original: [Figure]
Revised: [Figure]

Original: Figure 2:
Revised: Figure 2:

Original: On the Effect of width.
Revised: On the Effect of width.

Original: In Figure 2(a) we plot data scaling laws of the Myrtle-CNN at small (16) and large (1024) widths and its the infinite NTK.
Revised: In Figure 2(a) we plot data scaling laws of the Myrtle-CNN at small (16) and large (1024) widths and its the infinite NTK.

Original: We observe that both finite widths have similar scaling constant which is better than that of the infinite NTK.
Revised: We observe that both finite widths have similar scaling constant which is better than that of the infinite NTK.

Original: In Figure 2(b) we plot the performance of Myrtle-CNN and its Empirical NTK for a fixed training size while varying width.
Revised: In Figure 2(b) we plot the performance of Myrtle-CNN and its empirical NTK for a fixed training size while varying width.

Original: In
Revised: In

Original: Figure 2(c)
Revised: Figure 2(c)

Original: we do the same for a 5-layer CNN and the SVHN-parity task.
Revised: we do the same for a 5-layer CNN and the SVHN-parity task.

Original: Both Figure 2(b) and 2(c) we observe that (a) the Empirical NTK performance continues to improve with width, moving towards the infinite NTK performance while (b) neural network performance improves initially and then starts to deteriorate towards the infinite NTK performance.
Revised: Both Figure 2(b) and 2(c) we observe that (a) the empirical NTK performance continues to improve with width, moving towards the infinite NTK performance while (b) neural network performance improves initially and then starts to deteriorate towards the infinite NTK performance.

Original: Error bars represent estimated standard deviation.
Revised: Error bars represent estimated standard deviation.

Original: See Appendix B for more details.
Revised: See Appendix B for more details.

Original: Effect of Width.
Revised: Effect of Width.

Original: We explore the effect of width.
Revised: We explore the effect of width.

Original: In Figure 2(a) we train neural networks with widths much smaller (16) and much larger (1024) than the width (64) used in Figure 1(A).
Revised: In Figure 2(a) we train neural networks with widths much smaller (16) and much larger (1024) than the width (64) used in Figure 1(A).

Original: We find that these networks behaved similarly with respect to their scaling constants (.276 and .
Revised: We find that these networks behaved similarly with respect to their scaling constants (.276 and .

Original: respectively)
Revised: respectively)

Original: and performed better than the infinite width NTK (scaling constant: .213), confirming that real neural networks are far from the NTK regime .
Revised: and performed better than the infinite width NTK (scaling constant: .213), confirming that real neural networks are far from the NTK regime .

Original: However, we know that in the truly infinite width limit, all these methods will perform identically.
Revised: However, we know that in the truly infinite width limit, all these methods will perform identically.

Original: Moreover, as mentioned in Section 2 we are careful to ensure this limit is preserved by our optimization and initialization setup.
Revised: Moreover, as mentioned in Section 2 we are careful to ensure this limit is preserved by our optimization and initialization setup.

Original: This implies that at some point, increasing width of the real network will start to hurt performance— although it may be computationally infeasible to observe such large widths.
Revised: This implies that at some point, increasing width of the real network will start to hurt performance— although it may be computationally infeasible to observe such large widths.

Original: To explore the width-dependency, in Figure 2(b) we plot the expected performance of Empirical NTK and Neural network as we increase the width , using a fixed training size of 4000.
Revised: To explore the width-dependency, in Figure 2(b) we plot the expected performance of empirical NTK and Neural network as we increase the width , using a fixed training size of 4000.

Original: Here we see that (a) the Empirical NTK at initialization continues to improve with larger width and approaches the infinite NTK’s error from above, while (b) the neural network improves initially and then starts to deteriorate and approach the infinite NTK’s error from below.
Revised: Here we see that (a) the empirical NTK at initialization continues to improve with larger width and approaches the infinite NTK’s error from above, while (b) the neural network improves initially and then starts to deteriorate and approach the infinite NTK’s error from below.

Original: In Figure 2(c) we repeat the experiment for the SVHN-parity task.
Revised: In Figure 2(c) we repeat the experiment for the SVHN-parity task.

Original: In this setup it was computationally feasible to try out much larger width (upto 4096) with a smaller training size of 1000.
Revised: In this setup it was computationally feasible to try out much larger width (upto 4096) with a smaller training size of 1000.

Original: Hence in this experiment as the width increases, we can observe stronger deterioration of the performance of neural network, towards the infinite NTK performance.
Revised: Hence in this experiment as the width increases, we can observe stronger deterioration of the performance of neural network, towards the infinite NTK performance.

Original: Together these results suggest that “intermediate” widths (not too large, not too small) are important for the performance of overparameterized neural networks, and any explanatory theory must be consistent with this.
Revised: Together these results suggest that “intermediate” widths (not too large, not too small) are important for the performance of overparameterized neural networks, and any explanatory theory must be consistent with this.

Original: Effect of Learning Rate.
Revised: Effect of Learning Rate.

Original: We now study how robust our results are to changes in the learning rate, within practically used bounds.
Revised: We now study how robust our results are to changes in the learning rate, within practically used bounds.

Original: Note that changing the learning rate only affects the neural network training, and does not affect any of their corresponding NTKs.
Revised: Note that changing the learning rate only affects the neural network training, and does not affect any of their corresponding NTKs.

Original: In Figure 3(a) we train networks in the same setting as Figure 2, but with varying learning rates.
Revised: In Figure 3(a) we train networks in the same setting as Figure 2, but with varying learning rates.

Original: We find that after moderate modifications of the the learning rate the neural network still has a better scaling law than infinite and empirical
Revised: We find that after moderate modifications of the the learning rate the neural network still has a better scaling law than infinite and empirical

Original: NTK at initialization, suggesting that practically used learning rates (for practically used widths) are far from the NTK regime.
Revised: NTK at initialization, suggesting that practically used learning rates (for practically used widths) are far from the NTK regime.

Original: The scaling constants are .
Revised: The scaling constants are .

Original: 213 for the 3x higher learning rate, 10x lower learning rate and the infinite neural network.
Revised: 213 for the 3x higher learning rate, 10x lower learning rate and the infinite neural network.

Original: We discuss the effects of more drastic changes (1000x) in the learning rate in Appendix C.
Revised: We discuss the effects of more drastic changes (1000x) in the learning rate in Appendix C.

Original: [Figure]
Revised: [Figure]

Original: Other Changes in Optimization.
Revised: Other Changes in Optimization.

Original: We now study whether our results hold under other changes to optimization parameters.
Revised: We now study whether our results hold under other changes to optimization parameters.

Original: In Figure 3(b), 3(c), 3(d), we see the effect of doing GD instead of SGD, effect of training without momentum and of using the final test error instead of doing optimal early stopping respectively.
Revised: In Figure 3(b), 3(c), 3(d), we see the effect of doing GD instead of SGD, effect of training without momentum and of using the final test error instead of doing optimal early stopping respectively.

Original: We see that in all of these cases, while there is some change in the scaling laws, the neural network scaling constant is still always better than the one for infinite NTK.
Revised: We see that in all of these cases, while there is some change in the scaling laws, the neural network scaling constant is still always better than the one for infinite NTK.

Original: The scaling constants for neural networks in Figure 3(b), 3(c), 3(d) are .294, .310 and .292 respectively.
Revised: The scaling constants for neural networks in Figure 3(b), 3(c), 3(d) are .294, .310 and .292 respectively.

Original: The scaling constant for the infinite NTK is .213 in Figure 3(b), 3(c) and .219 for Figure 3(d).
Revised: The scaling constant for the infinite NTK is .213 in Figure 3(b), 3(c) and .219 for Figure 3(d).

Original: This suggests that these optimization factors (within commonly used values) are not the fundamental reason behind the improved scaling laws of neural networks.
Revised: This suggests that these optimization factors (within commonly used values) are not the fundamental reason behind the improved scaling laws of neural networks.

Original: Various extensions to the NTK regime have been proposed Roberts et al. ; Andreassen &
Revised: Various extensions to the NTK regime have been proposed (Roberts et al., 2021; Andreassen &

Original: Dyer (2020) in the literature which allow for the change in empirical NTK but posit that higher order analogues of the NTK remain constant.
Revised: Dyer, 2020) in the literature which allow for the change in empirical NTK but posit that higher order analogues of the NTK remain constant.

Original: This would predict that higher order analogues of the empirical NTK at initialization would be sufficient to match the performance of neural networks.
Revised: This would predict that higher order analogues of the empirical NTK at initialization would be sufficient to match the performance of neural networks.

Original: In
Revised: In

Original: Appendix D we show that this is not the case suggesting that these theories may also not be sufficient to explain the performance of practical neural networks.
Revised: Appendix D we show that this is not the case suggesting that these theories may also not be sufficient to explain the performance of practical neural networks.

Original: Discussion and Future Questions.
Revised: Discussion and Future Questions.

Original: The equivalence between neural networks and corresponding
Revised: The equivalence between neural networks and corresponding

Original: NTKs applies when width ≫ train-size.
Revised: NTKs applies when width ≫ train-size.

Original: On the other hand nearly all overparameterized networks and natural tasks fall in the regime of width ≪ train-size (though width is still large enough to fit the dataset).
Revised: On the other hand nearly all overparameterized networks and natural tasks fall in the regime of width ≪ train-size (though width is still large enough to fit the dataset).

Original: The results of this section — showing separations between neural networks in the latter regime and NTKs lead to following concrete question on the gap between theory and practice which could guide future work.
Revised: The results of this section — showing separations between neural networks in the latter regime and NTKs lead to following concrete question on the gap between theory and practice which could guide future work.

Original: Question 3.1.
Revised: Question 3.1.

Original: How can we understand the the behavior of overparameterized networks in the width ≪ train-size regime?
Revised: How can we understand the the behavior of overparameterized networks in the width ≪ train-size regime?

Original:  E XPLORATION OF A FTER -K ERNEL WRT D ATASET SIZE
Revised:  E XPLORATION OF A FTER -K ERNEL WRT D ATASET SIZE

Original: In the previous section, we studied the empirical NTK when linearized around weights at initialization .
Revised: In the previous section, we studied the empirical NTK when linearized around weights at initialization .

Original: In this section we will study the behaviour of Empirical NTK when linearized around the weights obtained at the end of training .
Revised: In this section we will study the behaviour of empirical NTK when linearized around the weights obtained at the end of training .

Original: This is known as the after-kernel for empirical NTK, in the terminology of Long (2021).
Revised: This is known as the after-kernel for empirical NTK, in the terminology of Long (2021).

Original: We will show, in the more precise sense defined below, that (1) the after-kernel continues to improves with dataset size, and thus (2) no fixed-time after-kernel is sufficient to capture the data scaling law of its corresponding neural network.
Revised: We will show, in the more precise sense defined below, that (1) the after-kernel continues to improves with dataset size, and thus (2) no fixed-time after-kernel is sufficient to capture the data scaling law of its corresponding neural network.

Original: Formally, denote the after-kernel from the neural network trained on m samples as K m .
Revised: Formally, denote the after-kernel from the neural network trained on m samples as K m .

Original: We will denote the accuracy of K m when fit on n samples as K m ( n ) .
Revised: We will denote the accuracy of K m when fit on n samples as K m ( n ) .

Original: Here, the n samples are a subset of the original m samples.
Revised: Here, the n samples are a subset of the original m samples.

Original: When we use fresh n samples to fit we use the notation K Fm ( n ) .
Revised: When we use fresh n samples to fit we use the notation K Fm ( n ) .

Original: We study the after-kernel as improved performance of neural networks over NTKs has been attributed OrtizJim´enez et al. ; Atanasov et al. (2021) to the adaptation of the empirical NTK of the neural network to the task.
Revised: We study the after-kernel as improved performance of neural networks over NTKs has been attributed (OrtizJim´enez et al., 2021; Atanasov et al., 2021) to the adaptation of the empirical NTK of the neural network to the task.

Original: Concretely, prior works Long (2021); Paccolat et al. (2021) have shown that this explanation is complete in the following sense: The behaviour of K n ( n ) is similar to that of the neural network fit on n samples.
Revised: Concretely, prior works (Long, 2021; Paccolat et al., 2021) have shown that this explanation is complete in the following sense: The behaviour of K n ( n ) is similar to that of the neural network fit on n samples.

Original: In other words, when we fit an after-kernel obtained from training on n samples to the same n samples we get an accuracy very close 1 to that of the neural network fit on the same n samples.
Revised: In other words, when we fit an after-kernel obtained from training on n samples to the same n samples we get an accuracy very close 1 to that of the neural network fit on the same n samples.

Original: We verify this for our setup in Figure 4(a).
Revised: We verify this for our setup in Figure 4(a).

Original: This tells us that the following two factors are sufficient to explain the behaviour of neural networks fit on n samples: (1) Change in empirical NTK from empirical NTK around initial weights to the after-kernel due to training on n samples.
Revised: This tells us that the following two factors are sufficient to explain the behaviour of neural networks fit on n samples: (1) Change in empirical NTK from empirical NTK around initial weights to the after-kernel due to training on n samples.

Original: (2) Fitting the after-kernel on n samples.
Revised: (2) Fitting the after-kernel on n samples.

Original: What this does not tell us is how these two improvements scale with training size n .
Revised: What this does not tell us is how these two improvements scale with training size n .

Original: In particular, we know that K 0 ( n ) i.e. the empirical NTK at initialization fit on n samples does not match the neural network trained on n samples on the other hand K n ( n ) does.
Revised: In particular, we know that K 0 ( n ) i.e. the empirical NTK at initialization fit on n samples does not match the neural network trained on n samples on the other hand K n ( n ) does.

Original: This raises the following natural question: How data dependent does the kernel need to be to recover the performance of the neural network?
Revised: This raises the following natural question: How data dependent does the kernel need to be to recover the performance of the neural network?

Original: For example, it is possible that for some sample size m 0 and all m ≥ m 0 , the after-kernel
Revised: For example, it is possible that for some sample size m 0 and all m ≥ m 0 , the after-kernel

Original: K m is roughly constant, and has same scaling law as the neural network itself.
Revised: K m is roughly constant, and has same scaling law as the neural network itself.

Original: We find that this is not the case– the after kenel continuously improves with dataset size m .
Revised: We find that this is not the case– the after kenel continuously improves with dataset size m .

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: After-Kernel continues to improve with dataset size.
Revised: After-Kernel continues to improve with dataset size.

Original: In Figure 4(b) we plot K m (500) versus m for our base Myrtle CNN from Figure 1.
Revised: In Figure 4(b) we plot K m (500) versus m for our base Myrtle CNN from Figure 1.

Original: We observe that K m (500) improves as m goes fromto 1024 k showing that the after-kernel keeps improving with larger dataset sizes.
Revised: We observe that K m (500) improves as m goes fromto 1024 k showing that the after-kernel keeps improving with larger dataset sizes.

Original: Corresponding
Revised: Corresponding

Original: SVHN-parity experiments can be found in Appendix F.
Revised: SVHN-parity experiments can be found in Appendix F.

Original: Fixed after-kernel is not sufficient to capture neural network data scaling.
Revised: Fixed after-kernel is not sufficient to capture neural network data scaling.

Original: In Figure 4(b) we plot the data scaling curves for the base Myrtle-CNN, its empirical NTK at initialization, K 16 k , K 64 k with scaling constants .
Revised: In Figure 4(b) we plot the data scaling curves for the base Myrtle-CNN, its empirical NTK at initialization, K 16 k , K 64 k with scaling constants .

Original: 097 respectively.
Revised: 097 respectively.

Original: We find that the neural network has the best scaling constant.
Revised: We find that the neural network has the best scaling constant.

Original: This shows that the scaling of the after-kernel with training size is an important component of neural network scaling laws as even the after-kernel learnt with 64k samples (on the simple CIFAR-5m-bin task) is not sufficient to explain the data scaling of neural networks.
Revised: This shows that the scaling of the after-kernel with training size is an important component of neural network scaling laws as even the after-kernel learnt with 64k samples (on the simple CIFAR-5m-bin task) is not sufficient to explain the data scaling of neural networks.

Original: We also see that K 64 k has better performance than K 16 k , another evidence towards the fact that after-kernel improves with dataset size.
Revised: We also see that K 64 k has better performance than K 16 k , another evidence towards the fact that after-kernel improves with dataset size.

Original:  T IME D YNAMICS
Revised:  T IME D YNAMICS

Original: In the previous section, we saw that the change in the empirical NTK from initialization to the end of training (the after-kernel ) is sufficient to explain the improved performance of neural networks.
Revised: In the previous section, we saw that the change in the empirical NTK from initialization to the end of training (the after-kernel ) is sufficient to explain the improved performance of neural networks.

Original: Thus the empirical NTK must have evolved throughout training, and in this section we take a closer
Revised: Thus the empirical NTK must have evolved throughout training, and in this section we take a closer

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: look at this evolution.
Revised: look at this evolution.

Original: Our main focus in this section is to investigate the following informal proposal in the literature Fort et al. (2020); Long (2021) about how the empirical NTK evolves:
Revised: Our main focus in this section is to investigate the following informal proposal in the literature (Fort et al., 2020; Long, 2021) about how the empirical NTK evolves:

Original: Hypothesis 5.1 ((Informal, from Fort et al.
Revised: Hypothesis 5.1 ((Informal, from Fort et al.

Original: (2020); Long (2021)) .
Revised: (2020); Long (2021)) .

Original: The empirical NTK evolves rapidly in the beginning of training ( < 5 epochs), but then undergoes a “phase transition” into a slower regime.
Revised: The empirical NTK evolves rapidly in the beginning of training ( < 5 epochs), but then undergoes a “phase transition” into a slower regime.

Original: One way to interpret the above hypothesis is that there is both a qualitative and quantitative difference in the empirical NTK between the “early phase” of training (the first few epochs) and the later stage of training.
Revised: One way to interpret the above hypothesis is that there is both a qualitative and quantitative difference in the empirical NTK between the “early phase” of training (the first few epochs) and the later stage of training.

Original: This is called a “phase transition” in the literature, in analogy to physics, where systems undergo discontinuities between two regimes with quantitatively different dynamics.
Revised: This is called a “phase transition” in the literature, in analogy to physics, where systems undergo discontinuities between two regimes with quantitatively different dynamics.

Original: In this section we will give evidence that suggests, contrary to prior work, that there is no such “phrase transition”.
Revised: In this section we will give evidence that suggests, contrary to prior work, that there is no such “phase transition”.

Original: We show that if empirical NTK performance is measured at the appropriate scale, performance appears to continuously improve throughout training (from early to late stages), at approximately the same “rate.”
Revised: We show that if empirical NTK performance is measured at the appropriate scale, performance appears to continuously improve throughout training (from early to late stages), at approximately the same “rate.”

Original: Our experiments are in fact compatible with the experiments in prior work (e.g. Fort et al.
Revised: Our experiments are in fact compatible with the experiments in prior work (e.g. Fort et al.

Original: (2020)): we simply observe that if performance and time are measured on a log-log scale (as is appropriate for measuring multi-scale dynamics), then the NTK is seen to improve continuously throughout most of the training.
Revised: (2020)): we simply observe that if performance and time are measured on a log-log scale (as is appropriate for measuring multi-scale dynamics), then the NTK is seen to improve continuously throughout most of the training.

Original: 5. E XPERIMENTS
Revised: 5. E XPERIMENTS

Original: Setup.
Revised: Setup.

Original: We now describe the setup more formally.
Revised: We now describe the setup more formally.

Original: Let K t refer to the empirical NTK (as described in Section 2) extracted at time t in the training, where we measure time in terms of number of
Revised: Let K t refer to the empirical NTK (as described in Section 2) extracted at time t in the training, where we measure time in terms of number of

Original: SGD batches seen in optimization.
Revised: SGD batches seen in optimization.

Original: K tfit denotes the model K t fit to the whole training data.
Revised: K tfit denotes the model K t fit to the whole training data.

Original: Prior works Fort et al. Long (2021) have used the slope of  curve of test error of K tfit versus t to decide if the kernel is changing rapidly or not.
Revised: Prior works (Fort et al., 2020; Long, 2021) have used the slope of the curve of test error of K tfit versus t to decide if the kernel is changing rapidly or not.

Original: We will do the same with one crucial difference: We will measure this slope on a log-log plot instead of directly plotting test error and time.
Revised: We will do the same with one crucial difference: We will measure this slope on a log-log plot instead of directly plotting test error and time.

Original: We do this as empirically scaling laws with respect to time (or tokens processed) have been observed Kaplan et al. (2020) for natural language tasks in neural networks and formally proved for kernels Velikanov & Yarotsky (2021) for natural tasks.
Revised: We do this as empirically scaling laws with respect to time (or tokens processed) have been observed (Kaplan et al., 2020) for natural language tasks in neural networks and formally proved for kernels (Velikanov & Yarotsky, 2021) for natural tasks.

Original: These results suggest the need for log-log plots to observe qualitative phase transitions in training dynamics.
Revised: These results suggest the need for log-log plots to observe qualitative phase transitions in training dynamics.

Original: Results.
Revised: Results.

Original: Our main claim is that the test error of K tfit as a function of time t is approximately linear on a log-log scale, throughout the course of training.
Revised: Our main claim is that the test error of K tfit as a function of time t is approximately linear on a log-log scale, throughout the course of training.

Original: Recall that K tfit is the model obtained by extracting the empirical NTK after t batches of training the real neural network.
Revised: Recall that K tfit is the model obtained by extracting the empirical NTK after t batches of training the real neural network.

Original: In Figure 5(a) we compare the test error of the base Myrtle-CNN at time t , test error of empirical
Revised: In Figure 5(a) we compare the test error of the base Myrtle-CNN at time t , test error of empirical

Original: NTK at initialization at time t and K tfit when trained on 64 k samples with the same hyperparameters as Figure 1.
Revised: NTK at initialization at time t and K tfit when trained on 64 k samples with the same hyperparameters as Figure 1.

Original: Since we want to probe Hypothesis 5.1, which is about the beginning of training, we plot these quantities until train error reaches < 5% (which requires 32 epochs in our experiments).
Revised: Since we want to probe Hypothesis 5.1, which is about the beginning of training, we plot these quantities until train error reaches < 5% (which requires 32 epochs in our experiments).

Original: This should be sufficient to cover any reasonable definition of “beginning of training”.
Revised: This should be sufficient to cover any reasonable definition of “beginning of training”.

Original: Observe that in Figure 5(a) we do not observe a “phase transition” after which the improvement in kernel test error (in red) slows down.
Revised: Observe that in Figure 5(a) we do not observe a “phase transition” after which the improvement in kernel test error (in red) slows down.

Original: In fact, we observe that the kernel starts out being essentially constant and then starts and continues to improve uniformly.
Revised: In fact, we observe that the kernel starts out being essentially constant and then starts and continues to improve uniformly.

Original: We instead observe the following two regimes: (1.)
Revised: We instead observe the following two regimes: (1.)

Original: In the first regime (before the dashed vertical line) the empirical NTK at initialization and the neural network have very similar behaviour, and
Revised: In the first regime (before the dashed vertical line) the empirical NTK at initialization and the neural network have very similar behaviour, and

Original: K tfit is nearly constant.
Revised: K tfit is nearly constant.

Original: This only lasts for around 140 batches ≈ 0.5 epochs 2 .
Revised: This only lasts for around 140 batches ≈ 0.5 epochs 2 .

Original: In the next regime (after the dashed line) the empirical NTK at initialization and the neural network diverge.
Revised: In the next regime (after the dashed line) the empirical NTK at initialization and the neural network diverge.

Original: As they diverge, the extracted kernel K tfit also starts to improve with a constant slope, and this improvement continues uniformly until the terminal stage of training.
Revised: As they diverge, the extracted kernel K tfit also starts to improve with a constant slope, and this improvement continues uniformly until the terminal stage of training.

Original: Importantly, the kernel K tfit does not transition into a “slower phase” of learning at any point 3 in our experiments.
Revised: Importantly, the kernel K tfit does not transition into a “slower phase” of learning at any point 3 in our experiments.

Original: Corresponding SVHN experiments can be found in Appendix F.
Revised: Corresponding SVHN experiments can be found in Appendix F.

Original: Next, we measure the performance of K t in terms of its data scaling law .
Revised: Next, we measure the performance of K t in terms of its data scaling law .

Original: For to computational limitations (since measuring data-scaling is expensive), we can only measure the scaling law for several selected values of time t , instead of every batch (as in Figure 5(a)).
Revised: Due to computational limitations (since measuring data-scaling is expensive), we can only measure the scaling law for several selected values of time t , instead of every batch (as in Figure 5(a)).

Original: In Figure 5(b) we plot datascaling of K t for t = [-
-] ( Empirical NTK at initialization ) , 3308 (10 epochs) , 11861 (32 epochs), in the same setup as Figure 5(a).
Revised: In Figure 5(b) we plot data-scaling of K t for t = [-
-]0 ( empirical NTK at initialization ) , 3308 (10 epochs) , 11861 (32 epochs), in the same setup as Figure 5(a).

Original: We also plot the data scaling of the base Myrtle-CNN with the same hyperparameters.
Revised: We also plot the data scaling of the base Myrtle-CNN with the same hyperparameters.

Original: As in Figure 4(c) of Section 4 we again observe that the neural network has the best scaling law, outperforming any of the extracted kernels.
Revised: As in Figure 4(c) of Section 4 we again observe that the neural network has the best scaling law, outperforming any of the extracted kernels.

Original: This shows that representations learnt after any constant time t of training are not sufficient to explain the data scaling of neural networks.
Revised: This shows that representations learnt after any constant time t of training are not sufficient to explain the data scaling of neural networks.

Original: Rather, these representations improve throughout training, and the entire course of training must be considered to recover the correct scaling law.
Revised: Rather, these representations improve throughout training, and the entire course of training must be considered to recover the correct scaling law.