Original: Feature Learning and Random Features in Standard Finite-Width Convolutional Neural Networks: An Empirical Study
Revised: Feature Learning and Random Features in Standard Finite-Width Convolutional Neural Networks: An Empirical Study

Original: Abstract
Revised: Abstract

Original: The Neural Tangent Kernel is an important milestone in the ongoing eﬀort to build a theory for deep learning.
Revised: The Neural Tangent Kernel is an important milestone in the ongoing effort to build a theory for deep learning.

Original: Its prediction that suﬃciently wide neural networks behave as kernel methods, or equivalently as random feature models arising from linearized networks, has been conﬁrmed empirically for certain wide architectures.
Revised: Its prediction that sufficiently wide neural networks behave as kernel methods, or equivalently as random feature models arising from linearized networks, has been confirmed empirically for certain wide architectures.

Original: In this paper, we compare the performance of two common ﬁnite-width convolutional neural networks, LeNet and AlexNet, to their linearizations on common benchmark datasets like MNIST and modiﬁed versions of it, CIFARand an ImageNet subset.
Revised: In this paper, we compare the performance of two common finite-width convolutional neural networks, LeNet and AlexNet, to their linearizations on common benchmark datasets like MNIST and modified versions of it, CIFARand an ImageNet subset.

Original: We demonstrate empirically that ﬁnite-width neural networks, generally, greatly outperform the ﬁnite-width linearization of these architectures.
Revised: We demonstrate empirically that finite-width neural networks, generally, greatly outperform the finite-width linearization of these architectures.

Original: When increasing the problem diﬃculty of the classiﬁcation task, we observe a larger gap which is in line with common intuition that ﬁnite-width neural networks perform feature learning which ﬁnite-width linearizations cannot.
Revised: When increasing the problem difficulty of the classification task, we observe a larger gap which is in line with common intuition that finite-width neural networks perform feature learning which finite-width linearizations cannot.

Original: At the same time, ﬁnite-width linearizations improve dramatically with width, approaching the behavior of the wider standard networks which in turn perform slightly better than their standard width counterparts.
Revised: At the same time, finite-width linearizations improve dramatically with width, approaching the behavior of the wider standard networks which in turn perform slightly better than their standard width counterparts.

Original: Therefore, it appears that feature learning for non-wide standard networks is important but becomes less signiﬁcant with increasing width.
Revised: Therefore, it appears that feature learning for non-wide standard networks is important but becomes less significant with increasing width.

Original: We furthermore identify cases where both standard and linearized networks match in performance, in agreement with NTK theory, and a case where a wide linearization outperforms its standard width counterpart.
Revised: We furthermore identify cases where both standard and linearized networks match in performance, in agreement with NTK theory, and a case where a wide linearization outperforms its standard width counterpart.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: The Neural Tangent Kernel (NTK)
Revised: The Neural Tangent Kernel (NTK)

Original: [Jacot et al., 2018] is a seminal contribution to the study of deep neural networks which extended important insights about the connection of Gaussian processes and neural networks [Neal, 1996,
Revised: [Jacot et al., 2018] is a seminal contribution to the study of deep neural networks which extended important insights about the connection of Gaussian processes and neural networks [Neal, 1996,

Original: Williams, 1996, de G. Matthews et al., 2018, Lee et al., 2018, Garriga-Alonso et al., 2019].
Revised: Williams, 1996, de G. Matthews et al., 2018, Lee et al., 2018, Garriga-Alonso et al., 2019].

Original: Ever since, subsequent investigations have reﬁned our view on the NTK with results suggesting both its validity and insuﬃciency as an explanation for the performance of practical ﬁnite-width neural networks, and the focus of investigation has moved to the network parameterization and architecture diﬀerences and their relationship to the NTK
Revised: Ever since, subsequent investigations have refined our view on the NTK with results suggesting both its validity as well as insufficiency as an explanation for the performance of practical finite-width neural networks, and the focus of investigation has moved to the network parameterization and architecture differences and their relationship to the NTK

Original: [Chizat et al., 2019, Lee et al., 2019, Chen et al., 2020, Hanin and Nica, 2020, Xiao et al., 2020, Seleznova and Kutyniok, 2022].
Revised: [Chizat et al., 2019, Lee et al., 2019, Chen et al., 2020, Hanin and Nica, 2020, Xiao et al., 2020, Seleznova and Kutyniok, 2022].

Original: The NTK framework has inspired work in many directions like inﬁnite ensembles of trees [Kanoh and Sugiyama, 2022], federated learning [Huang et al., 2021] and thus continues to stimulate various advances in deep learning theory.
Revised: The NTK framework has inspired work in many directions like infinite ensembles of trees [Kanoh and Sugiyama, 2022], federated learning [Huang et al., 2021] and thus continues to stimulate various advances in deep learning theory.

Original: Jacot et al.
Revised: Jacot et al.

Original: [2018] proved that  modeling neural network training under gradient ﬂow, i.e. full batch gradient descent of inﬁnitesimal step size, the training trajectory 𝑓 ( 𝑤, 𝑥 ) satisﬁes an ordinary diﬀerential equation (ODE) involving the ﬁnite-width Neural Tangent Kernel
Revised: [2018] proved that when modeling neural network training under gradient flow, i.e. full batch gradient descent of infinitesimal step size, the training trajectory 𝑓 ( 𝑤, 𝑥 ) satisfies an ordinary differential equation (ODE) involving the finite-width Neural Tangent Kernel

Original: [Equation]
Revised: [Equation]

Original: for weights 𝑤 ∈ ℝ 𝑝 and inputs 𝑥, 𝑥 ′ ∈
Revised: for weights 𝑤 ∈ ℝ 𝑝 and inputs 𝑥, 𝑥 ′ ∈

Original: ℝ 𝑑 .
Revised: ℝ 𝑑 .

Original: The form of this kernel depends on the network architecture and the current time-dependent weights 𝑤 as well as a particular initialization.
Revised: The form of this kernel depends on the network architecture and the current time-dependent weights 𝑤 as well as a particular initialization.

Original: In this NTK parameterization , they showed that  scaling the learning rate per layer in an appropriate way and letting the width tend to inﬁnity, this kernel converges to the inﬁnite-width NTK ̂ Θ which is independent of the weights and stays constant during training, greatly simplifying the
Revised: In this NTK parameterization , they showed that when scaling the learning rate per layer in an appropriate way and letting the width tend to infinity, the kernel converges to the infinite-width NTK ̂ Θ which is independent of the weights and stays constant during training, greatly simplifying the

Original: ODE in this limit.
Revised: ODE in this limit.

Original: They showed for the 𝐿 2 loss that the predictor at convergence is precisely what a kernel regression using the inﬁnite-width NTK would produce.
Revised: They showed for the 𝐿 2 loss that the predictor at convergence is precisely what a kernel regression using the infinite-width NTK would produce.

Original: Importantly, the inﬁnite-width NTK depends only on the architecture of the network; it is not learned and thus data-independent.
Revised: Importantly, the infinite-width NTK depends only on the architecture of the network; it is not learned and thus data-independent.

Original: The formalism was extended from fully-connected to other architectures including convolutional networks [Arora et al., 2019, Yang, 2019], recurrent neural networks [Alemohammad et al., 2020], residual networks [Huang et al., 2020] and more general architectures [Yang, 2020].
Revised: The formalism was extended from fully-connected to other architectures including convolutional networks [Arora et al., 2019, Yang, 2019], recurrent neural networks [Alemohammad et al., 2020], residual networks [Huang et al., 2020] and more general architectures [Yang, 2020].

Original: This result can be understood as the convergence of wide networks to random feature models [Chizat et al., 2019].
Revised: This result can be understood as the convergence of wide networks to random feature models [Chizat et al., 2019].

Original: Let 𝑓 ∶ ℝ 𝑑
Revised: Let 𝑓 ∶ ℝ 𝑑

Original: →
Revised: →

Original: ℝ
Revised: ℝ

Original: 𝐿 be the function give by a network parameterised with weights 𝑤 ∈ ℝ 𝑝 and input 𝑥 ∈ ℝ 𝑑 and let 𝑓 𝑙 be the output in component 𝑙 = 1 , … , 𝐿 , with 𝐿 being typically the number of classes in a classiﬁcation task.
Revised: 𝐿 be the function given by a network parameterized by weights 𝑤 ∈ ℝ 𝑝 with input 𝑥 ∈ ℝ 𝑑 and let 𝑓 𝑙 be the output in component 𝑙 = 1 , … , 𝐿 , with 𝐿 being typically the number of classes in a classification task.

Original: For 𝑤 suﬃciently close to the random initial weights 𝑤 0 and 𝑢 =
Revised: For 𝑤 sufficiently close to the random initial weights 𝑤 0 and 𝑢 =

Original: 𝑤 − 𝑤 0 , the ﬁrst order Taylor expansion in the weights 𝑓 [Equation] ( 𝑤, 
Revised: 𝑤 − 𝑤 0 , the first order Taylor expansion in the weights 𝑓 𝑙 ( 𝑤, 𝑥 ) ≈ 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) + ∇ 𝑤 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) ⋅ 𝑢 =∶ 𝑓 𝑙 lin ( 𝑢, 𝑥 ) (2) is an accurate approximation.

Original: [Equation] [Equation] ) is an accurate approximation. The right-hand side 𝑓 lin ( 𝑢, 𝑥 ) is a random feature model with weights 𝑢 ∈ ℝ 𝑝 and the feature mapping 𝜙 ( 𝑥 )
Revised: The right-hand side 𝑓 lin ( 𝑢, 𝑥 ) is a random feature model with weights 𝑢 ∈ ℝ 𝑝 and the feature mapping 𝜙 ( 𝑥 )

Original: ∈
Revised: ∈

Original: ℝ 𝐿 × 𝑝 is given by the gradients 𝜙 𝑙 ( 𝑥 )
Revised: ℝ 𝐿 × 𝑝 is given by the gradients 𝜙 𝑙 ( 𝑥 )

Original: = ∇ 𝑤 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) with respect to the weights at initialization 𝑤 0 .
Revised: = ∇ 𝑤 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) with respect to the weights at initialization 𝑤 0 .

Original: If approximationEq. (2) holds, then also the gradients ∇ 𝑤 𝑓 ( 𝑤, 𝑥 ) and ∇ 𝑢 𝑓 lin ( 𝑢, 𝑥 ) of the two models will be close.
Revised: If approximation (2) holds, then also the gradients ∇ 𝑤 𝑓 ( 𝑤, 𝑥 ) and ∇ 𝑢 𝑓 lin ( 𝑢, 𝑥 ) of the two models will be close.

Original: When training these models with some form of gradient descent and suﬃciently small step size for a suﬃciently small number of steps, then the training trajectories will stay close, as long as the weight vectors remain in the region around 𝑢 = 0 or 𝑤 = 𝑤 0 , respectively.
Revised: When training these models with some form of gradient descent and sufficiently small step size for a sufficiently small number of steps, then the training trajectories will stay close, as long as the weight vectors remain in the region around 𝑢 = 0 or 𝑤 = 𝑤 0 , respectively.

Original: Using an 𝐿 2 loss in over-parameterized models, one can expect both models to converge to zero loss [Du et al., 2019].
Revised: Using an 𝐿 2 loss in over-parameterized models, one can expect both models to converge to zero loss [Du et al., 2019].

Original: If convergence occurs before leaving this region, then the models – whether trained with early stopping or until convergence – will predict a similar function.
Revised: If convergence occurs before leaving this region, then the models – whether trained with early stopping or until convergence – will predict a similar function.

Original: For the inﬁnite-width case,
Revised: For the infinite-width case,

Original: Lee et al.
Revised: Lee et al.

Original: [2019] proved the convergence of 𝑓 ( 𝑤, 𝑥 ) and 𝑓 lin ( 𝑢, 𝑥 )  in distribution to the same Gaussian distribution.
Revised: [2019] proved that 𝑓 ( 𝑤, 𝑥 ) and 𝑓 lin ( 𝑢, 𝑥 ) converge in distribution to the same Gaussian distribution.

Original: Furthermore, the NTK result can be proved by showing that for very wide neural networks the models 𝑓 ( 𝑤, 𝑥 ) and 𝑓 lin ( 𝑢, 𝑥 ) reach zero loss and thus stop evolving before leaving the region where the approximation in Eq.
Revised: Furthermore, the NTK result can be proved by showing that for very wide neural networks the models 𝑓 ( 𝑤, 𝑥 ) and 𝑓 lin ( 𝑢, 𝑥 ) reach zero loss and thus stop evolving before leaving the region where the approximation in Eq.

Original: (2) is accurate
Revised: (2) is accurate

Original: [Chizat et al., 2019, Lee et al., 2019], known as lazy training .
Revised: [Chizat et al., 2019, Lee et al., 2019], known as lazy training .

Original: The linearized model 𝑓 lin does not learn a representation but uses the random representation ∇ 𝑤 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) which is ﬁxed by the initial weights 𝑤 0 and remains unchanged throughout training.
Revised: The linearized model 𝑓 lin does not learn a representation but uses the random representation ∇ 𝑤 𝑓 𝑙 ( 𝑤 0 , 𝑥 ) which is fixed by the initial weights 𝑤 0 and remains unchanged throughout training.

Original: More in line with Gaussian processes and random feature models [Rahimi and Recht, 2007]
Revised: More in line with Gaussian processes and random feature models [Rahimi and Recht, 2007]

Original: but at odds with general intuition on deep learning, NTK theory predicts that, at large widths, a network and its linearization behave similarly and no signiﬁcant feature learning takes place.
Revised: but at odds with general intuition on deep learning, NTK theory predicts that, at large widths, a network and its linearization behave similarly and no significant feature learning takes place.

Original: This seems to imply that – even for standard neural networks – learning a good representation might become decreasingly relevant with increasing over-parameterization.
Revised: This seems to imply that – even for standard neural networks – learning a good representation might become decreasingly relevant with increasing over-parameterization.

Original: In this regard, we study standard convolutional neural networks (CNNs) and their respective linearizations  given by 
Revised: Motivated by this conjecture, we study standard convolutional neural networks (CNNs) and their respective linearizations (at initialization) given by Eq.

Original: Eq.
Revised: We complement previous work in that direction (see Sec. 2) and extend these results for more standard architectures in more common classification tasks.

Original: We complement previous work in that direction by a thorough study of two standard CNNs, LeNet [LeCun et al., 1998] and AlexNet [Krizhevsky et al., 2012], for increasingly diﬃcult classiﬁcation tasks at diﬀerent widths.
Revised: In particular, we perform a thorough study of two standard CNNs, LeNet [LeCun et al., 1998] and AlexNet [Krizhevsky et al., 2012], for increasingly difficult classification tasks at different widths.

Original: We observe test accuracy gaps between these networks, in line with the idea thatthe standard neural networks perform feature learning while their linearizations do not.
Revised: We observe test accuracy gaps between these networks, in line with the idea that standard neural networks perform feature learning while their linearizations do not.

Original: For the wider networks the generalization gap closes, in line with NKT theory, supporting the picture summarized in Fig. 
Revised: For the wider networks the generalization gap closes, in line with NTK theory, supporting the picture summarized in Fig. 1.

Original: However, we also observe low training accuracy for the linearized networks.
Revised: However, we also observe low training accuracy for the linearized networks.

Original: We investigate numerical issues which can explain reduced training performance in the linearizations and consider a simpliﬁed binary classiﬁcation setting in which we can solve the linear system in Eq.
Revised: We investigate numerical issues which can explain reduced training performance in the linearizations and consider a simplified binary classification setting in which we can solve the linear system in Eq

Original: 2 with a standard solver achieving 100% training accuracy, but observe that this generally causes even worse test accuracy for the linearized models
Revised: (2) with a standard solver achieving 100% training accuracy, but observe that this generally causes even worse test accuracy for the linearized models.

Original: In this work, we make the following contributions:
Revised: In this work, we make the following contributions

Original:  • We show that for (nearly) all considered widths, there is a prominent performance gap between the standard and linearized LeNet and AlexNet and this gap increases when the classiﬁcation task increases in diﬃculty.
Revised: : • We show that for (nearly) all considered widths, there is a prominent performance gap between the standard and linearized LeNet and AlexNet and this gap increases when the classification task increases in difficulty.

Original: This is shown for MNIST, CIFAR-10, and a subset of ImageNet.
Revised: This is shown for MNIST, CIFAR-10, and a subset of ImageNet.

Original: We believe this gap exhibits the importance of feature learning for non-wide standard networks.
Revised: We believe this gap exhibits the importance of feature learning for non-wide standard networks.

Original: • We present further instances where wide linearized networks perform as well as the standard network and cases where linearized wide networks outperform their standard width counterpart.
Revised: • We present further instances where wide linearized networks perform as well as the standard network and cases where linearized wide networks outperform their standard width counterpart.

Original: • As for wider networks the generalization gap closes, in line with NTK theory, we raise the question if this means that the non-wide and wide standard network generalize due to a very diﬀerent mechanism: feature learning for non-wide networks and eﬀectively employing unlearned random features at larger widths.
Revised: • As for wider networks the generalization gap closes, in line with NTK theory, we raise the question if this means that the non-wide and wide standard network generalize due to a very different mechanism: feature learning for non-wide networks and effectively employing unlearned random features at larger widths.

Original: • We extend the discussion in previous work of numerical aspects of training the non-wide linearized models by considering the eﬀective rank of the kernel.
Revised: • We extend the discussion in previous work of numerical aspects of training the non-wide linearized models by considering the effective rank of the kernel.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: The original motivation and prevailing appeal of (ﬁnite) deep neural networks is that they are powerful methods to extract statistics and learn features leading to strong performance for down-stream tasks (regime I in Fig.
Revised: The original motivation and prevailing appeal of (finite) deep neural networks is that they are powerful methods to extract statistics and learn features leading to strong performance for down-stream tasks (regime I in Fig.

Original: [Lee et al., 2009, Alekseev and Bobe, 2019].
Revised: [Lee et al., 2009, Alekseev and Bobe, 2019].

Original: The behavior of neural networks in the highly over-parameterized regime has been extensively studied, too, suggesting minor weight changes from initialization during training (regime II)
Revised: The behavior of neural networks in the highly over-parameterized regime has been extensively studied, too, suggesting minor weight changes from initialization during training (regime II)

Original: [Du et al., 2019].
Revised: [Du et al., 2019, Allen-Zhu et al., 2019, Zou et al., 2020].

Original: In the NTK literature, typically, the inﬁnite-width limit for ﬁnite-depth neural networks is considered (connection betweenregimes II and IV). In contrast, Deep Equilibrium
Revised: In the NTK literature, typically, the infinite-width limit for finitedepth neural networks is considered (connection between

Original: Models consider the inﬁnite-depth limit at ﬁnite width [Bai et al., 2019].

Original: Hanin and Nica [2020] study the NTK for both inﬁnitely wide and deep ReLU networks, showing particular data-dependent features of the resulting NTK in these limits.

Original: Width
Revised: Width

Original: Feature Learning s t a nd a r d n e t .
Revised: Feature Learning s t a nd a r d n e t .

Original: I : learned features, strong performance
Revised: I : learned features, strong performance

Original: II : approaching lazy training, performance improvement li n ea r i za ti on
Revised: II : approaching lazy training, performance improvement li n ea r i za ti on

Original: III : random features, weak performance
Revised: III : random features, weak performance

Original: IV : random features, lazy training, strong performance improvement 
Revised: IV : random features, lazy training, strong performance improvement regimes II and IV).

Revised: In contrast, Deep Equilibrium Models consider the infinite-depth limit at finite width [Bai et al., 2019].

Revised: Hanin and Nica [2020] study the NTK for both infinitely wide and deep ReLU networks, showing particular data-dependent features of the resulting NTK in these limits.

Original: Focusing on the ﬁnite-depth case, there are several studies which compare the ﬁnite-width NTK Θ or inﬁnite-width
Revised: Focusing on the finite-depth case, there are several studies which compare the finite-width NTK Θ or infinite-width

Original: NTK ̂ Θ to their standard network (regimes I and IV) and provide, to some extent, diverging results.
Revised: NTK ̂ Θ to their standard network (regimes I and IV) and provide, to some extent, diverging results.

Original: The original work by Jacot et al.
Revised: The original work by Jacot et al.

Original: [2018] gives experimental results for small synthetic datasets, as well as fully-connected networks trained on MNIST of widths 𝑛 = 10 2 , 10 3 , and 10 4 , showing good agreement with the inﬁnite-width NTK for the widest network.
Revised: [2018] gives experimental results for small synthetic datasets, as well as fully-connected networks trained on MNIST of widths 𝑛 = 10 2 , 10 3 , and 10 4 , showing good agreement with the infinite-width NTK for the widest network.

Original: Lee et al.
Revised: Lee et al.

Original: [2019] extend the original work and show good agreement for small synthetic datasets (of size ≤ 256 ) and for a two hidden layer fully-connected network trained with SGD on MNIST.
Revised: [2019] extend the original work and show good agreement for small synthetic datasets (of size ≤ 256 ) and for a two hidden layer fully-connected network trained with SGD on MNIST.

Original: Most interestingly, a wide ResNet[Zagoruyko and Komodakis, 2016] trained on CIFAR-10 shows similar behavior, though the non-linearized model appears to have been trained only to below 80% training accuracy, and in the test accuracy a gap seems to develop towards the end of training (see Fig.
Revised: Most interestingly, a wide ResNet trained on CIFAR-10 shows similar behavior, though the non-linearized model appears to have been trained only to below 80% training accuracy, and in the test accuracy a gap seems to develop towards the end of training (see Fig.

Original: 7 in their paper).
Revised: 7 in their paper).

Original: In contrast to that, in Chizat et al.
Revised: In contrast to that, in Chizat et al.

Original: [2019] VGG-11[Simonyan and Zisserman, 2015] and ResNet-18 [He et al., 2016] trained on CIFAR-10 and widened with a scaling factor 𝛼 for tuning the models into the non-linearized and linearized regimes, demonstrated large gaps in test accuracy.
Revised: [2019] VGG-11 and ResNet-18 – trained on CIFARand widened with a scaling factor 𝛼 for tuning the models into the non-linearized and linearized regimes – exhibit large gaps in test accuracy.

Original: They highlight that the decreased training performance of the linearization is due to bad conditioning and eﬀectively low rank of the associated kernel matrix.
Revised: They highlight that the decreased training performance of the linearization is due to bad conditioning and effectively low rank of the associated kernel matrix.

Original: In their extension to CNNs, Arora et al.
Revised: In their extension to CNNs, Arora et al.

Original: [2019] compared CNNs with two to 20 convolutional layers combined with fully-connected or global average pooling output layers to the derived inﬁnite-width convolutional NTK (CNTK), observing large gaps in test accuracy on CIFAR-10.
Revised: [2019] compare CNNs with two to 20 convolutional layers combined with fully-connected or global average pooling output layers to the derived infinite-width convolutional NTK (CNTK), observing large gaps in test accuracy on CIFAR-10.

Original: Our work is most closely related to Lee et al.
Revised: Our work is most closely related to Lee et al.

Original: [2020] and
Revised: [2020] and

Original: Geiger et al.
Revised: Geiger et al.

Original: In Lee et al.
Revised: In Lee et al.

Original: [2020] an extensive empirical study of neural networks, their linearizations and the inﬁnite-width NTK as well as the Neural Network Gaussian process (NNGP) kernel [Lee et al., 2018, de G. Matthews et al., 2018] is conducted.
Revised: [2020] an extensive empirical study of neural networks, their linearizations and the infinite-width NTK as well as the Neural Network Gaussian process (NNGP) kernel [Lee et al., 2018, de G. Matthews et al., 2018] is conducted.

Original: For architectures built from fullyconnected and  convolutional layers, they show cases where NTK can both outperform but also underperform their corresponding networks on CIFAR-10.
Revised: For fully-connected and simple convolutional architectures, they show cases where NTK can both outperform but also underperform their corresponding networks on CIFAR-10.

Original: Importantly, they study the relevance of regularization of the kernels and identify bad conditioning of the kernel as a reason for decreased performance.
Revised: Importantly, they study the relevance of regularization of the kernels and identify bad conditioning of the kernel as a reason for decreased performance.

Original: In line with results by Wei et al.
Revised: In line with results by Wei et al.

Original: [2020], they showed that 𝐿 2 regularization (like weight decay) of the kernel is required for good performance in practice, although this breaks the inﬁnite-width correspondence to kernel methods.
Revised: [2020], they showed that 𝐿regularization (like weight decay) of the kernel is required for good performance in practice, although this breaks the infinite-width correspondence to kernel methods.

Original: In contrast to their work, we focus on two more standard but also more extensive CNNs where we increase the widths of the standard and linearized networks explicitly and study their properties with a focus on feature learning.
Revised: In contrast to their work, we focus on two more standard but also more extensive CNNs where we increase the widths of the standard and linearized networks explicitly and study their properties with a focus on feature learning.

Original: In that regard, our work diﬀers from Geiger et al.
Revised: In that regard, our work differs from Geiger et al.

Original: [2020] who also study lazy training and feature learning but for fullyconnected networks of depth three to ﬁve and CNNs with four convolutional layers and in the framework of Chizat et al.
Revised: [2020] who also study lazy training and feature learning but for fully-connected networks of depth three to five and CNNs with four convolutional layers and in the framework of Chizat et al.

Original: [2019] with a scaling factor 𝛼 controlling the lazy training regime.
Revised: [2019] with a scaling factor 𝛼 controlling the lazy training regime.

Original: Another related line of research was conducted by Ortiz-Jiménez et al.
Revised: Another related line of research was conducted by OrtizJiménez et al.

Original: [2021] which study linearizations with respect to task complexity deﬁned on the basis of the NTK eigenfunctions as targets.
Revised: [2021] which study linearizations with respect to task complexity defined on the basis of the NTK eigenfunctions as targets.

Original: In their evaluation on CIFAR-10, they show that linearization performance can rank learning complexity and show that neural networks  not always outperform their kernel approximations.
Revised: In their evaluation on CIFAR-10, they show that linearization performance can rank learning complexity and show that neural networks do not always outperform their kernel approximations.

Original: Other relevant work includes Seleznova and Kutyniok [2022] which investigates the ordered and chaotic phase phenomena of vanishing and exploding gradients in the context of NTK theory, providing guarantees when the NTK is illconditioned (ordered phase) or well-conditioned (chaotic phase and at the border between the two phases).
Revised: Other relevant work includes Seleznova and Kutyniok [2022] which investigates the ordered and chaotic phase phenomena of vanishing and exploding gradients in the context of NTK theory, providing guarantees when the NTK is illconditioned (ordered phase) or well-conditioned (chaotic phase and at the border between the two phases).

Revised: Furthermore, Yang and Hu [2021] note that standard and NTK parameterizations do not lead to representations that learn features in the infinite-width limit and propose an alternative parameterization enabling feature learning in this limit.

Original:  METHOD
Revised:  METHOD

Original: We examine two standard ReLU CNNs, LeNet and AlexNet, trained for classiﬁcation tasks of increasing diﬃculty.
Revised: We examine two standard ReLU CNNs, LeNet and AlexNet, trained for classification tasks of increasing difficulty.

Original: One task is digit recognition in MNIST [LeCun et al., 1998] and modiﬁed versions which include random translations of the otherwise centered digits.
Revised: One task is digit recognition in MNIST [LeCun et al., 1998] and modified versions which include random translations of the otherwise centered digits.

Original: In addition, we train on CIFAR-
Revised: In addition, we train on CIFAR-

Original: [Krizhevsky et al., 2009], and a subset of ImageNet [Russakovsky et al., 2015] which contains ten diﬀerent snake classes (see Supp.
Revised: [Krizhevsky et al., 2009], and a subset of ImageNet [Russakovsky et al., 2015] which contains ten different snake classes (see Supp.

Original: Sec. A.1), whereby we deliberately chose similar classes to form a challenging classiﬁcation task.
Revised: Sec. A.1), whereby we deliberately chose similar classes to form a challenging classification task.

Original: In this setup, we study the performance of the standard network and its linearization 𝑓 lin (see Eq.
Revised: In this setup, we study the performance of the standard net- work and its linearization 𝑓 lin (see Eq.

Original: and the eﬀect of increasing the width of the networks, thereby investigating the relationships between regimes I and III as well as III and IV in Fig.
Revised: and the effect of increasing the width of the networks, thereby investigating the relationships between regimes I and III as well as III and IV in Fig.

Original: This is done by multiplying the number of channels in each convolutional layer and all widths of fully-connected layers with a common factor.
Revised: This is done by multiplying the number of channels in each convolutional layer and all widths of fullyconnected layers by a common factor.

Original: Due to GPU memory limitations, we are able to train LeNet and LinLeNet up to width factors of 60 and for AlexNet and LinAlexNet up to width factors of 4.
Revised: Due to GPU memory limitations, we are able to train LeNet and LinLeNet up to width factors of 60 and for AlexNet and LinAlexNet up to width factors of 4.

Original: As the number of parameters increase quadratically in the width, and standard width LeNet and
Revised: As the number of parameters increase quadratically in the width, and standard width LeNet and

Original: AlexNet have about 60 𝑘 and 60 𝑚 parameters, we were hence able to train networks of up to 216 𝑚 and 960 𝑚 parameters, respectively.
Revised: AlexNet have about 60 k and 60 𝑚 parameters, we were hence able to train networks of up to 216 𝑚 and 960 𝑚 parameters, respectively.

Original: Our implementation makes use of PyTorch’s
Revised: Our implementation makes use of PyTorch’s

Original: [Paszke et al., 2019] standard modules for deﬁning and training neural networks with our own custom-made modiﬁcations for linearization of the architectures.
Revised: [Paszke et al., 2019] standard modules for defining and training neural networks with our own custom-made modifications for linearization of the architectures.

Original: For LeNet, we adapt the original LeNet-5 architecture [LeCun et al., 1998] to use maxpooling and ReLU activations.
Revised: For LeNet, we adapt the original LeNet-5 architecture [LeCun et al., 1998] to use maxpooling and ReLU activations.

Original: For AlexNet, we use the PyTorch implementation [Krizhevsky, 2014] with 10 outputs rather than 1000 (see below).
Revised: For AlexNet, we use the PyTorch implementation [Krizhevsky, 2014] with 10 outputs rather than 1000 (see below).

Original: Despite training for classiﬁcation, we use the 𝐿 2 loss with one-hot encoded target vectors.
Revised: Despite training for classification, we use the 𝐿 2 loss with one-hot encoded target vectors.

Original: Firstly, with standard cross-entropy loss the networks never converge to exactly zero loss, so the networks must at some point leave the region where the approximation in Eq.
Revised: Firstly, with standard cross-entropy loss the networks never converge to exactly zero loss, so the networks must at some point leave the region where the approximation in Eq.

Original: (2) is valid, causing some ambiguity in the heuristic.
Revised: (2) is valid, causing some ambiguity in the heuristic.

Original: Secondly, the 𝐿 2 loss allows for an easier and more eﬃcient implementation of the training of the linearized models.
Revised: Secondly, the 𝐿 2 loss allows for an easier and more efficient implementation of the training of the linearized models.

Original: We furthermore do not make use of dropout, since it is not clear to us how to model it in the NTK framework (see however Novak et al.
Revised: We furthermore do not make use of dropout, since it is not clear to us how to model it in the NTK framework (see however Novak et al.

Original: We ﬁnd that that after optimizing hyperparameters, we can train LeNet and AlexNet to similar train and test performance as with cross-entropy loss without dropout (see Supp.
Revised: We find that after optimizing hyperparameters, we can train LeNet and AlexNet to similar train and test performance as with cross-entropy loss without dropout (see Supp.

Original: Sec. A.1).
Revised: Sec. A.1).

Original: We predict the class whose one-hot vector is closest to the output vector, which is equivalent to predicting the argmax of the output layer.
Revised: We predict the class whose one-hot vector is closest to the output vector, which is equivalent to predicting the argmax of the output layer.

Original: We train 𝑓 lin ( 𝑢, 𝑥 ) with SGD in the standard way by optimizing 𝑢 with gradient updates obtained by
Revised: We train 𝑓 lin ( 𝑢, 𝑥 ) with SGD in the standard way by optimizing 𝑢 with gradient updates obtained by

Original: [Equation]
Revised: [Equation]

Original: Computing the gradients of the linear model with 𝐿 outputs requires computing 𝐿 gradients of the original network per data point, and thus 𝐿 backward passes, which is computationally intensive if 𝐿 is large.
Revised: Computing the gradients of the linear model with 𝐿 outputs requires computing 𝐿 gradients of the original network per data point, and thus 𝐿 backward passes, which is computationally intensive if 𝐿 is large.

Original: We therefore train (Lin)AlexNet on the snakes subset of ImageNet consisting of 𝐿 = 10 classes, while we can use full MNIST and CIFARfor (Lin)LeNet.
Revised: We therefore train (Lin)AlexNet on the snakes subset of ImageNet consisting of 𝐿 = 10 classes, while we can use full MNIST and CIFARfor (Lin)LeNet.

Original: As our goal is to stay as close as possible to standard neural network training practices, we use SGD with weight decay and momentum.
Revised: As our goal is to stay as close as possible to standard neural network training practices, we use SGD with weight decay and momentum.

Original: In addition, we use the standard PyTorch weight initialization, which is a variant of Kaiming initialization [He et al., 2015], rather than the NTK parameterization used in the NTK proofs.
Revised: In addition, we use the standard PyTorch weight initialization, which is a variant of Kaiming initialization [He et al., 2015], rather than the NTK parameterization used in the NTK proofs.

Original:  EXPERIMENTAL RESULTS
Revised:  EXPERIMENTAL RESULTS

Original: In the experiments, ﬁve independent reruns of the speciﬁed networks for 100 epochs and batch size 32 were performed unless stated otherwise.
Revised: In the experiments, five independent reruns of the specified networks for 100 epochs and batch size 32 were performed unless stated otherwise.

Original: Hyperparameter search was conducted for each network architecture and its linearization at all widths separately, for learning rates including {1 , 0 . 1 , 0 . 01 , 0 . 001} and weight decay values including 5 × {10 −4 , 10 −5 , 10 −6 , 10 −7 , 10 −8 } .
Revised: Hyperparameter search was conducted for each network architecture and its linearization at all widths separately, for learning rates including {1 , 0 . 1 , 0 . 01 , 0 . 001} and weight decay values including 5 × {10 −4 , 10 −5 , 10 −6 , 10 −7 , 10 −8 } .

Original: The momentum parameter was set to the default value of 0 .
Revised: The momentum parameter was set to the default value of 0 .

Original: For each rerun, a diﬀerent ﬁxed random seed was used to ensure that both the standard and linearized models at a particular width are initialized exactly the same and receive the same mini-batches during training.
Revised: For each rerun, a different fixed random seed was used to ensure that both the standard and linearized models at a particular width are initialized exactly the same and receive the same mini-batches during training.

Original: For experiments involving CIFAR-10 and the snakes dataset, the learning rate was decreased by a factor 10 every 30 epochs.
Revised: For experiments involving CIFAR-10 and the snakes dataset, the learning rate was decreased by a factor 10 every 30 epochs.

Original: Otherwise, we follow the standard preprocessing for standardizing the input images and standard resizing (256 pixels) and center cropping (224 pixels) for
Revised: Otherwise, we follow the standard preprocessing for standardizing the input images and standard resizing (256 pixels) and center cropping (224 pixels) for

Original: ImageNet images.
Revised: ImageNet images.

Original: Computations were conducted on Nvidia
Revised: Computations were conducted on Nvidia

Original: GeForce Titan X Pascal and Tesla V100 GPUs.
Revised: GeForce Titan X Pascal and Tesla V100 GPUs.

Original: For experiments involving LinAlexNet ×3 and LinAlexNet ×4 , we used an Nvidia Quadro RTX 8000 with 48 GB memory due to the increased memory requirement.
Revised: For experiments involving LinAlexNet ×3 and LinAlexNet ×4 , we used an Nvidia Quadro RTX 8000 with 48 GB memory due to the increased memory requirement.

Original: All displayed results are obtained with single precision.
Revised: All displayed results are obtained with single precision.

Original: We also carried out all experiments in section 4.1 with double precision, but did not observe any striking diﬀerences.
Revised: We also carried out all experiments in section 4.1 with double precision, but did not observe any striking differences.

Original: 4.1 CLASSIFICATION WITH INCREASING FEATURE LEARNING REQUIREMENT
Revised: 4.1 CLASSIFICATION WITH INCREASING FEATURE LEARNING REQUIREMENT

Original: LeNet trained on MNIST and CIFAR-10:
Revised: LeNet trained on MNIST and CIFAR-10:

Original: For LeNet with about 60 𝑘 parameters, we used width factors ranging from 1 to 60 .
Revised: For LeNet with about 60 k parameters, we used width factors ranging from 1 to 60 .

Original: In all experiments involving MNIST and
Revised: In all experiments involving MNIST and

Original: CIFAR-10, a learning rate of 0 . 1 and weight decay of 5×10 − led to overall best test accuracies.Table A
Revised: CIFAR-10, a learning rate of 0 . 1 and weight decay of 5×10 − led to overall best test accuracies.

Original: The results for MNIST are presented in Fig.
Revised: The results for MNIST are presented in Fig.

Original: For the standard width, a substantial diﬀerence of 4 .
Revised: For the stan- dard width, a substantial difference of 4 .

Original: 67 percentage points in (mean) test error between LeNet and LinLeNet is observed.
Revised: 67 percentage points in (mean) test error between LeNet and LinLeNet is observed.

Original: While LeNet does not gain appreciably from increasing the width, LinLeNet does, and the gap shrinks to 0 .
Revised: While LeNet does not gain appreciably from increasing the width, LinLeNet does, and the gap shrinks to 0 .

Original: 48 percentage points for width factor 60 .
Revised: 48 percentage points for width factor 60 .

Original: Similarly, though not close in a path-wise sense, the statistics of trajectories of output values become more alike with increasing width (shown in Supp.
Revised: Similarly, though not close in a path-wise sense, the statistics of trajectories of output values become more alike with increasing width (shown in Supp.

Original: Fig.
Revised: Fig.

Original: A.9), indicating a more similar behavior of training dynamics of the linearized and standard models at large width factors.
Revised: A.2), indicating a more similar behavior of training dynamics of the linearized and standard models at large width factors.

Original: For factor 1 the linearized model outperforms a logistic regression on normalized MNIST pixels only by a small margin, which achieves about 93% train and 92% test accuracy.
Revised: For factor 1 the linearized model outperforms a logistic regression on normalized MNIST pixels only by a small margin, which achieves about 93% train and 92% test accuracy.

Original: The low training accuracy of the linearized models is investigated in more detail in Sec.
Revised: The low training accuracy of the linearized models is investigated in more detail in Sec.

Original: 4.2 and 4.3.
Revised: 4.2 and 4.3.

Original: When increasing the problem diﬃculty by randomly translating the digits horizontally and vertically, larger gaps in test (and train) accuracy are observed which also decrease with width.
Revised: When increasing the problem difficulty by randomly translating the digits horizontally and vertically, larger gaps in test (and train) accuracy are observed which also decrease with width.

Original: For factors 1 and 60 , we observe 27 .
Revised: For factors 1 and 60 , we observe 27 .

Original: 65 and 3 .percentage points diﬀerence in test error.
Revised: 65 and 3 .percentage points difference in test error.

Original: The full results are illustrated in Fig.
Revised: The full results are illustrated in Fig.

Original: 3 for translations up to 7 pixels (i.e. up to a quarter of the image size).
Revised: 3 for translations up to 7 pixels (i.e. up to a quarter of the image size).

Original: When training on the more challenging CIFAR-10 dataset even larger gaps are observed, as shown in Fig.
Revised: When training on the more challenging CIFAR-10 dataset even larger gaps are observed, as shown in Fig.

Original: For the standard width, a diﬀerence of 20 .
Revised: For the standard width, a difference of 20 .

Original: 22 percentage points in test error between LeNet and LinLeNet is observed.
Revised: 22 percentage points in test error between LeNet and LinLeNet is observed.

Original: This shrinks to a smaller but still appreciable gap of 13 .
Revised: This shrinks to a smaller but still appreciable gap of 13 .

Original: 17 percentage points at width factor 60 .
Revised: 17 percentage points at width factor 60 .

Original: Interestingly, LinLeNet × outperforms standard width LeNet × 1 in both training and test error (gray dashed line).
Revised: Interestingly, LinLeNet × outperforms standard width LeNet × 1 in both training and test error (gray dashed line).

Original: AlexNet trained on snakes dataset: For AlexNet with about 60 𝑚 parameters, width factors 1, 2, 3, and 4 were used and the networks were trained on the ten-class snakes subset of ImageNet (see Supp.
Revised: AlexNet trained on snakes dataset: For AlexNet with about 60 m parameters, width factors 1, 2, 3, and 4 were used and the networks were trained on the ten-class snakes subset of ImageNet (see Supp.

Original: Sec. A.1).
Revised: Sec. A.1).

Original: For the linearized networks, a learning rate of 1 and weight decay of 5 × 10 − provided the best test errors.
Revised: For the linearized networks, a learning rate of 1 and weight decay of 5 × 10 − provided the best test errors.

Original: For the standard networks, however, a learning rate of 0 . 1 and weight decay of 5 × 10 − lead to best test performance.
Revised: For the standard networks, however, a learning rate of 0 . 1 and weight decay of 5 × 10 − lead to best test performance.

Original: In addition, we trained the linearized networks with these hyperparameters settings, too, for comparison.
Revised: In addition, we trained the linearized networks with these hyperparameters settings, too, for comparison

Original: Figure 5 summarizes the ﬁndings, which fall in line with the observed trend for LeNet but give even larger gaps in test error.
Revised: Figure 5 summarizes the findings, which fall in line with the observed trend for LeNet but give even larger gaps in test error.

Original: Trained with the same hyperparameters as their non-linearized counterparts, the gaps in test error between standard AlexNet and LinAlexNet are more than 20 percentage points at all considered widths.
Revised: Trained with the same hyperparameters as their non-linearized counterparts, the gaps in test error between standard AlexNet and LinAlexNet are more than 20 percentage points at all considered widths.

Original: For the optimal hyperparameters in the linearized setting, the generalization gap shrinks only slightly to 20 .
Revised: For the optimal hyperparameters in the linearized setting, the generalization gap shrinks only slightly to 20 .

Original: 4 and 18 .
Revised: 4 and 18 .

Original: 56 percentage points for widths 1 and 4 .
Revised: 56 percentage points for widths 1 and 4 .

Original: While increasing the width has little impact on train and test error of AlexNet, for LinAlexNet the test error shows a slight decrease and the training error a strong decrease with width.
Revised: While increasing the width has little impact on train and test error of AlexNet, for LinAlexNet the test error shows a slight decrease and the training error a strong decrease with width.

Original: These results show that, at standard width or small width expansion factors, the random feature models given by the linearized networks perform poorly compared to their standard network counterpart or the random feature models of wider linearized networks.
Revised: These results show that, at standard width or small width expansion factors, the random feature models given by the linearized networks perform poorly compared to their standard network counterpart or the random feature models of wider linearized networks.

Original: With increasing problem diﬃculty, the increasing gap between linearized and standard 
Revised: With increasing problem difficulty, the increasing gap between linearized and standard LeNet and

Original: LeNet and AlexNet suggest that at standard widths significant feature learning is taking place in the standard (nonlinearized) model.
Revised: AlexNet suggests that at standard widths significant feature learning is taking place in the standard (non-linearized) model.

Original: But with increased overparameterization, these gaps indeed shrink as predicted by NTK theory.
Revised: But with increased over-parameterization, these gaps indeed shrink as predicted by NTK theory.

Original: The way the gap shrinks is through a dramatic improvement in performance of the linearized networks with width, while standard networks are less aﬀected in their performance by width.
Revised: The way the gap shrinks is through a dramatic improvement in performance of the linearized networks with width, while standard networks are less affected in their performance by width.

Original: However, as theory proves that the wide standard trained networks behave as random feature models,  the small improvements in accuracy of standard networks with width might be hiding a signiﬁcant transition is the underling reason for their good performance, namely from feature learning for the non-wide networks to utilizing non-learned random features that apparently provide a good inductive bias for the tasks at hand for the wider networks (both linearized and non-linearized).
Revised: However, as theory proves that the wide standard trained networks behave as random feature models, we hypothesize that the small improvements in accuracy of standard networks with width might be hiding a significant transition is the underlying reason for their good performance, namely from feature learning for the non-wide networks to utilizing non-learned random features that apparently provide a good inductive bias for the tasks at hand for the wider networks (both linearized and non-linearized).

Original: 4.2 NUMERICAL ASPECTS
Revised: 4.2 NUMERICAL ASPECTS

Original: The low training accuracy of the non-wide linearized models in the previous experiments raise the question of whether they are well-trained at all.
Revised: The low training accuracy of the non-wide linearized models in the previous experiments raise the question of whether they are well-trained at all.

Original: Fitting the linearized model with 𝑚 data points 𝑥 ∈
Revised: Fitting the linearized model with 𝑚 data points 𝑥 ∈

Original: ℝ 𝑑 is eﬀectively solving the linear system
Revised: ℝ 𝑑 is effectively solving the linear system

Original: [Equation]
Revised: [Equation]

Original: for weights 𝑢 ∈ ℝ 𝑝 and target 𝑦 ∈ ℝ 𝐿 , where 𝑝 is the number of parameters of the original model and the rows of the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑥 ) are the gradients of each output of the network at data point 𝑥 .
Revised: for weights 𝑢 ∈ ℝ 𝑝 and target 𝑦 ∈ ℝ 𝐿 , where 𝑝 is the number of parameters of the original model and the rows of the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑥 ) are the gradients of each output of the network at data point 𝑥 .

Original: With 𝑋 ∈ ℝ 𝑚 × 𝑑 , the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) ∈ ℝ 𝑛 × 𝑝 has 𝑛 = 10 × 𝑚 rows since one must ﬁt each of the 𝐿 = 10 outputs for each data point.
Revised: With 𝑋 ∈ ℝ 𝑚 × 𝑑 , the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) ∈ ℝ 𝑛 × 𝑝 has 𝑛 = 10 × 𝑚 rows since one must fit each of the 𝐿 = 10 outputs for each data point.

Original: LeNet at width factors 1 and 2 has roughly 𝑝 1 = 60 𝑘 and 𝑝 2 = 240 𝑘 parameters, respectively.
Revised: LeNet at width factors 1 and 2 has roughly 𝑝 1 = 60 k and 𝑝 2 = 240 k parameters, respectively.

Original: Thus, the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) cannot have full rank when ﬁtting a dataset of size 𝑚 = [-
 𝑘-] (CIFAR-10) or 𝑚 = 60 𝑘 (MNIST), i.e. 𝑝 1 , 𝑝 2 < 10× 𝑚 , making it impossible to ﬁt arbitrary targets.
Revised: Thus, the matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) cannot have full rank when fitting a dataset of size 𝑚 = [-
 𝑘-]50 k (CIFAR-10) or 𝑚 = 60 k (MNIST), i.e. 𝑝 1 , 𝑝 2 < 10× 𝑚 , making it impossible to fit arbitrary targets.

Original: Moreover, even for wider networks it appears that matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) remains eﬀectively of low rank.
Revised: Moreover, even for wider networks it appears that matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) remains effectively of low rank.

Revised: in ⋆ ) and snakes datasets, respectively, with full rank 6000 .

Original: We quantify this by computing the eﬀective rank [Roy and
Revised: We quantify this by computing the effective rank [Roy and

Original: Vetterli, 2007] which takes the distribution of singular values into consideration and can be viewed as the exponential entropy of normalized singular values (see Supp.
Revised: Vetterli, 2007] which takes the distribution of singular values into consideration and can be viewed as the exponential entropy of normalized singular values (see Supp.

Original: Sec.
Revised: Sec.

Original: A.3).
Revised: A.3).

Original: For computational reasons, we consider 𝑚 =data samples and the corresponding 6000 × 6000 kernel matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 )∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 )
Revised: For computational reasons, we consider 𝑚 =data samples and the corresponding 6000 × 6000 kernel matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 )∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 )

Original: ⊤ for each width factor.
Revised: ⊤ for each width factor.

Original: For
Revised: For

Original: LinLeNet and considering MNIST samples, these eﬀective ranks are much lower than the number of rows, i.e. 6000 , and increases with width factor asis illustrated in Fig.
Revised: LinLeNet and considering MNIST samples, these effective ranks are much lower than the number of rows, i.e. 6000 , and increase with width factor as illustrated in Fig.

Original: 6 (left).
Revised: 6 (left).

Original: A similar but less pronounced improvement in eﬀective rank with width is obtained for MNIST samples with additional random translation of up to 7 pixels.
Revised: A similar but less pronounced improvement in effective rank with width is obtained for MNIST samples with additional random translation of up to 7 pixels.

Original: Although AlexNet × with about 60 𝑚 parameters is well in the overparameterized regime for 𝑚 = 13 𝑘 datapoints and thus 𝑛 = 130 𝑘 rows in matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) , we still observe large gaps in training accuracy.
Revised: Although AlexNet × with about 60 m parameters is well in the over-parameterized regime for 𝑚 = 13 k datapoints and thus 𝑛 = 130 k rows in matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) , we still observe large gaps in training accuracy.

Original: As for LinLeNet, we show for 600 examples in Fig.
Revised: As for LinLeNet, we show for 600 examples in Fig.

Original: 6 (right) that the eﬀective ranks at all widths is signiﬁcantly lower than the number of rows of ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) and increase with width (marginally).
Revised: 6 (right) that the effective ranks at all widths are significantly lower than the number of rows of ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) and increase with width (marginally).

Original: In Fig.
Revised: In Fig.

Original: 7, the distribution of singular values 𝜎 of the kernel matrix is shown (see Supp.
Revised: 7, the distribution of singular values 𝜎 of the kernel matrix is shown (see Supp.

Original: Fig.
Revised: Fig.

Original: A.10 for LinAlexNet).
Revised: A.3 for LinAlexNet).

Original: We observe that increasing the width eﬀectively increases the smallest (non-vanishing) singular values of matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) and generally leads to a lower condition number (i.e. ratio 𝜎 max ∕ 𝜎 min ), for this matrix, thereby improving numerical properties.
Revised: We observe that increasing the width effectively increases the smallest (non-vanishing) singular values of matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) and generally leads to a lower condition number (i.e. ratio 𝜎 max ∕ 𝜎 min ), for this matrix, thereby improving numerical properties.

Original: We suspect that, in order to perfectly ﬁt the training data, one needs to ﬁt 𝑢 also in a subspace with very small singular values, making it diﬃcult to achieve close to 100% train accuracy with SGD with non-inﬁnitesimal step sizes. 
Revised: We suspect that, in order to perfectly fit the training data, one needs to fit 𝑢 also in a subspace with very small singular values, making it difficult to achieve close to 100% train accuracy with SGD with non-infinitesimal step sizes. 7 pixels. See Supp. Fig.

Revised: A.3 for LinAlexNet.

Original: 4.3 BINARY CLASSIFICATION ON MNIST
Revised: 4.3 BINARY CLASSIFICATION ON MNIST

Original: In order to study these numerical aspects in more detail, we take a closer look at the solution of the linear system in
Revised: In order to study these numerical aspects in more detail, we take a closer look at the solution of the linear system in

Original: Eq.
Revised: Eq.

Original: In particular, we examine if the multiclass setting might be the cause for numerical stability issues due to having multiple outputs (the diﬀerent classes) for a single input, potentially leading to e.g. collinearity of rows in the matrix.
Revised: In particular, we examine if the multiclass setting might be the cause for numerical stability issues due to having multiple outputs (the different classes) for a single input, potentially leading to e.g. collinearity of rows in the matrix.

Original: Therefore, we consider binary classiﬁcation with one output and train to classify a digit as 0 or not 0.
Revised: Therefore, we consider binary classification with one output and train to classify a digit as 0 or not 0.

Original: Qualitatively similar results were obtained for other target classes.
Revised: Qualitatively similar results were obtained for other target classes.

Original: In the following, we solve the one-vs-rest classiﬁcation task with the same least-squares objective in three ways: by training
Revised: In the following, we solve the one-vs-rest classification task with the same least-squares objective in three ways: by training

Original: LeNet with SGD, by training LinLeNet with SGD, and by using a standard solver for linear systems.
Revised: LeNet with SGD, by training LinLeNet with SGD, and by using a standard solver for linear systems.

Original: The presented results are obtained from single runs of the respective model with a ﬁxed random seed.
Revised: The presented results are obtained from single runs of the respective model with a fixed random seed.

Original: Solving the linear system with SGD: The training is performed in the same manner as before, but with a learning rate of 0.01 and for 200 epochs.
Revised: Solving the linear system with SGD: The training is performed in the same manner as before, but with a learning rate of 0.01 and for 200 epochs.

Original: Tables 1 and 2 summarize the binary classiﬁcation results with target class 0 for standard
Revised: Tables 1 and 2 summarize the binary classification results with target class 0 for standard

Original: MNIST and MNIST with up to 7 pixels translation.
Revised: MNIST and MNIST with up to 7 pixels translation.

Original: The qualitative behavior with SGD training follows the same trend as in Fig.s 2 and 3 for the multiclass results (for this reason an illustration is omitted).
Revised: The qualitative behavior with SGD training follows the same trend as in Fig.s 2 and 3 for the multiclass results (for this reason an illustration is omitted).

Original: As before, by including translations of up to 7 pixels of the digits, we observe a drop in accuracies which is particularly pronounced for the linearized setting.
Revised: As before, by including translations of up to 7 pixels of the digits, we observe a drop in accuracies which is particularly pronounced for the linearized setting.

Original: [Table]
Revised: [Table]

Original: In comparison to the harder multiclass task, the gap in training accuracy between LeNet and LinLeNet is greatly reduced but persists for the less wide networks, especially for
Revised: In comparison to the harder multiclass task [Table] the gap in training accuracy between LeNet and LinLeNet is greatly reduced but persists for the less wide networks, especially for

Original: LinLeNet ×1 in the translated MNIST task.
Revised: LinLeNet ×1 in the translated MNIST task.

Original: While training the standard network consistently leads to perfect training accuracy in the standard MNIST setting, it is not possible to achieve 100% training accuracy when solving the linear system in Eq.
Revised: While training the standard network consistently leads to perfect training accuracy in the standard MNIST setting, it is not possible to achieve 100% training accuracy when solving the linear system in Eq.

Original: (4) with SGD, in most cases.
Revised: (4) with SGD, in most cases.

Original: However, from a width factor of 5 on, we observe for LinLeNet in the standard
Revised: However, from a width factor of 5 on, we observe for LinLeNet in the standard

Original: MNIST task that the linearized networks start agreeing (up to the second decimal place) with the results of the corresponding LeNet.
Revised: MNIST task that the linearized networks start agreeing (up to the second decimal place) with the results of the corresponding LeNet.

Original: In particular, LinLeNet ×60 matches the train and test results of LeNet at all considered widths, which is in agreement with NTK theory.up to 7 pixels.
Revised: In particular, LinLeNet ×60 matches the train and test results of LeNet at all considered widths, which is in agreement with NTK theory.

Original: ( ∗ : These results were not completed by the deadline, but will be included in future versions.

Original: However, the trend is the same as for ×2 .)

Original: [Table]
Revised: [Table]

Original: Solving the linear system with a standard solver: Since
Revised: Solving the linear system with a standard solver: Since

Original: SGD is not able to attain high train accuracy for linearized models for all widths, it raises the question whether a different algorithm can, and if so, what its generalization properties are for the tasks at hand.
Revised: SGD is not able to attain high train accuracy for linearized models for all widths, it raises the question whether a different algorithm can, and if so, what its generalization properties are for the tasks at hand.

Original: An advantage of the binary classiﬁcation setting is that we can directly solve the linear system in Eq.
Revised: An advantage of the binary classification setting is that we can directly solve the linear system in Eq.

Original: (4) for 𝑢 for width multipliers 1, 2, and 5, as the amount of memory required to store the entire matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑥 ) in memory is reduced and becomes manageable.
Revised: (4) for 𝑢 for width multipliers 1, 2, and 5, as the amount of memory required to store the entire matrix ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑥 ) in memory is reduced and becomes manageable.

Original: Larger widths were not feasible for us as more than 1 TB of memory is required even for binary classiﬁcation, without taking additional memory requirements for the computation into account.
Revised: Larger widths were not feasible for us as more than 1 TB of memory is required even for binary classification, without taking additional memory requirements for the computation into account.

Original: We make use of the SciPy least-squares solver which utilizes the highly optimized LAPACK library [Anderson et al., 1999].
Revised: We make use of the SciPy least-squares solver which utilizes the highly optimized LAPACK library [Anderson et al., 1999].

Original: The results are included in Tables 1 and 2.
Revised: The results are included in Tables 1 and 2.

Original: Interestingly, the solver attains perfect training accuracy in all considered cases, but at the cost of a diminished test accuracy for LinLeNet ×1 in standard MNIST (see Table 1) and, particularly, in translated MNIST (see Table 2), indicating overﬁtting of the solver solution.
Revised: Interestingly, the solver attains perfect training accuracy in all considered cases, but at the cost of a diminished test accuracy for LinLeNet ×1 in standard MNIST (see Table 1) and, particularly, in translated MNIST (see Table 2), indicating overfitting of the solver solution.

Original: Apparently, the implicit regularization of the SGD solution signiﬁcantly improves generalization for these widths, while precluding a perfect train accuracy.
Revised: Apparently, the implicit regularization of the SGD solution significantly improves generalization for these widths, while precluding a perfect train accuracy.

Original: For LinLeNets of larger widths, an improved generalization is attained which we view to match the SGD results to a reasonable degree (considering ﬂuctuations in the second decimals place as in the multiclass results, see
Revised: For LinLeNets of larger widths, an improved generalization is attained which we view to match the SGD results to a reasonable degree (considering fluctuations in the second decimals place as in the multiclass results, see

Original: Supp.
Revised: Supp.

Original: Table A.3).
Revised: Table A.2).

Original: In the standard MNIST task, the attained solver solutions for LinLeNet ×2 and LinLeNet ×5 match the test accuracies of their corresponding standard LeNets at otherwise 100% train accuracy.
Revised: In the standard MNIST task, the attained solver solutions for LinLeNet ×2 and LinLeNet ×5 match the test accuracies of their corresponding standard LeNets at otherwise 100% train accuracy.

Original: It should be noted that the solver results were obtained without regularization.
Revised: It should be noted that the solver results were obtained without regularization.

Original: Additional regularization should lead to similar results as for
Revised: Additional regularization should lead to similar results as for

Original: LinLeNet ×1 trained with SGD, that is higher generalization and lower training accuracy.
Revised: LinLeNet ×1 trained with SGD, that is higher generalization and lower training accuracy.

Original: Therefore, it appears that the observed generalization gaps and poor performance of non-wide linearized models in Sec.
Revised: Therefore, it appears that the observed generalization gaps and poor performance of non-wide linearized models in Sec.

Original: 4.1 are not due to poor training optimization.
Revised: 4.1 are not due to poor training optimization.

Original: We suspect that moderately wide linearized networks in the mutliclass experiments operate in a similar regime as LinLeNet ×1 in the binary classiﬁcation setting.
Revised: We suspect that moderately wide linearized networks in the multiclass experiments operate in a similar regime as LinLeNet ×1 in the binary classification setting.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: Motivated by conﬂicting results in NTK literature, we studied two classical convolutional neural networks, LeNet and
Revised: Motivated by conflicting results in NTK literature, we studied two classical convolutional neural networks, LeNet and

Original: AlexNet, and their corresponding linearizations at diﬀerent widths and increasing diﬃculty of classiﬁcation tasks.
Revised: AlexNet, and their corresponding linearizations at different widths and increasing difficulty of classification tasks.

Original: We investigated four regimes of diﬀerent behavior in neural networks (see Fig. 1) which complement previous results on lazy training [Chizat et al., 2019] and random feature models
Revised: We investigated four regimes of different behavior in neural networks (see Fig. 1) which complement previous results on lazy training [Chizat et al., 2019] and random feature models

Original: [Lee et al., 2019, 2020].
Revised: [Lee et al., 2019, 2020] summarized in the following.

Original: Firstly, in agreement with previous results like by Arora et al.
Revised: Firstly, in agreement with previous results like by Arora et al.

Original: [2019], we observed signiﬁcant train and test performance gaps between standard width LeNet and AlexNet and their corresponding linearization.
Revised: [2019], we observed significant train and test performance gaps between standard width LeNet and AlexNet and their corresponding linearization.

Original: By considering diﬀer- ent classiﬁcation tasks of increasing diﬃculty, we showed that the performance gaps increase accordingly suggesting that richer features need to be learned, which the eﬀectively random feature models LinLeNet and LinAlexNet cannot provide.
Revised: By considering different classification tasks of increasing difficulty, we showed that the performance gaps increase accordingly suggesting that richer features need to be learned, which the effectively random feature models LinLeNet and LinAlexNet cannot provide.

Original: Secondly, in agreement with work such as Lee et al.
Revised: Secondly, in agreement with work such as Lee et al.

Original: Jacot et al.
Revised: Jacot et al.

Original: [2018], we showed, however, that width improves the performance of linearized networks signiﬁcantly.
Revised: [2018], we showed, however, that width improves the performance of linearized networks significantly.

Original: The comparatively minor improvements in performance of standard networks might hide a transition from feature learning to utilizing random features at moderate widths.
Revised: We hypothesize that the comparatively minor improvements in performance of standard networks might hide a transition from feature learning to utilizing random features at moderate widths.

Revised: This might be related to previous results suggesting that the intermediate representations of networks of increasing width become increasingly alike to each other and to the representation in the large width limit [Kornblith et al., 2019].

Original: Thirdly, we showed that numerically aspects like the eﬀective rank (see Fig. 6) and distribution of singular values (see
Revised: Thirdly, we showed that numerical aspects like the effective rank (see Fig. 6) and distribution of singular values (see

Original: Fig.
Revised: Fig.

Original: 7 and A.10) of the feature mapping ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) have a role in explaining low training accuracy of SGD trained non-wide linearized models.
Revised: 7 and A.3) of the feature mapping ∇ 𝑤 𝑓 ( 𝑤 0 , 𝑋 ) have a role in explaining low training accuracy of SGD trained non-wide linearized models.

Original: Increasing width appears to remedy these numerical issues of the associated kernel.
Revised: Increasing width appears to remedy these numerical issues of the associated kernel.

Original: In summary, our investigation is based on the ﬁnite-width
Revised: In summary, our investigation is based on the finite-width

Original: NTK at initialization and explores deviations, as described above, but also convergence to NTK theory.
Revised: NTK at initialization and explores deviations, as described above, but also convergence to NTK theory.

Original: In particular, we show agreement in performance of standard networks and their linearization as well as an instance where a wide
Revised: In particular, we show agreement in performance of standard networks and their linearization as well as an instance where a wide

Original: LinLeNet( ×60 ) outperforms its standard width LeNet( ×1 ) on CIFAR-10.
Revised: LinLeNet( ×60 ) outperforms its standard width LeNet( ×1 ) on CIFAR-10.

Original: Our study highlights the need to study theoretical descriptions of neural network generalization beyond the ﬁnitewidth NTK at initialization, for instance by considering time-dependent NTK [Huang and Yau, 2019, Jacot et al., 2018] for ﬁnite-width networks  or by further developing the various proposed mean-ﬁeld theories [Chizat and Bach, 2018, Hu et al., 2019, Javanmard et al., 2019, Mei et al., 2019, Nguyen, 2019, Rotskoﬀ et al., 2019].
Revised: Our study highlights the need to study theoretical descriptions of neural network generalization beyond the finitewidth NTK at initialization, for instance by considering time-dependent NTK [Huang and Yau, 2019, Jacot et al., 2018] for finite-width networks (see e.g. Fort et al. [2020]) or by further developing the various proposed mean-field theories [Chizat and Bach, 2018, Hu et al., 2019, Javanmard et al., 2019, Mei et al., 2019, Nguyen, 2019, Rotskoff et al., 2019].

Original: Additionally, it highlights the need to study the nature of the apparent transition to eﬀectively random features at moderate widths in standard neural network training
Revised: Additionally, it highlights the need to study the nature of the potential transition to effectively random features at moderate widths in standard neural network training.

Original: A SUPPLEMENTARY MATERIAL A.1 SNAKES DATASET
Revised: Author Contributions

Original: For a challenging classiﬁcation task, we chose a subset of ImageNet 2012 [Russakovsky et al., 2015] comprised of ten snake categories illustrated in Fig. A.8.

Original: The extracted dataset contains 1300 training and 50 test images per class, resulting in 13000 train and 500 test images in total. As a benchmark performance results, we evaluate a standard pre-trained AlexNet on this dataset, achieving 47 . 6% test accuracy. Training our implementation of AlexNet with cross entropy loss on the snakes dataset provides 98 .
Revised: D. Belius and M. Samarin jointly conceived the idea and wrote the paper.

Original: 5% train and 51 . 4% test accuracy (single run). In our experiments, we used a mean squared error loss, which in comparison led to 99 .
Revised: M. Samarin wrote the code, performed the experiments, created the figures and revised the paper.

Original: 1% train and 53 . 8% test accuracy (single run). These results indicate that both loss function lead to comparable performance and outperform a standard pre-trained AlexNet (trained on full ImageNet) with respect to generalization. A.2 EARLY TRAINING TRAJECTORY Following Lee et al.
Revised: V. Roth participated in the discussions and the refinement of the results.

Original: [2019], we study training trajectories for data samples 𝑥 of the MNIST test set during training. For illustration, we plot the standard LeNet output 𝑓 𝑙 ( 𝑤 𝑡 , 𝑥 ) and the linearization 𝑓 𝑙 lin ( 𝑢 𝑡 , 𝑥 ) against the iteration 𝑡 = 0 , 1 , … for diﬀerent widths. Note that 𝑤 𝑡 and 𝑢 𝑡 are the weights after 𝑡 gradient updates for LeNet and LinLeNet trained on MNIST.
Revised: Acknowledgements

Original: As we use one-hot encoding, output 𝑙 denotes the predicted output for the correct class of the data point 𝑥 . The same hyperparameters as for the other MNIST experiments are used (see Sec. 4.1). A ﬁxed random seed ensures that both LeNet and LinLeNet at a particular width factor are initialized exactly the same and receive the same mini-batches during training. Exemplary results are shown in Fig.
Revised: We are grateful to Levent Sagun, Peter Zaspel, and Ivan

Original: A.9 for a digit 8 of thetest set, with similar results being obtained for other samples, too. At small widths, training trajectories immediately diverge. With increasing width, the curves behave more similar; they are however not close in a path-wise sense, but the statistics of training trajectories become more alike. A.3 EFFECTIVE RANK
Revised: Dokmanić for enlightening discussions regarding the results presented in this article.

Original: The eﬀective rank was introduced by Roy and Vetterli [2007] and can be viewed as the exponential entropy of normalized singular values. We restate the main deﬁnition in the following. Deﬁnition. Let 𝐴 be a complex-valued non-all-zero matrix of size 𝑀 × 𝑁 with (real positive) singular values 𝜎 1 ≥ 𝜎 2 ≥ ... ≥ 𝜎 𝑄 ≥ 0 , where 𝑄 = min{ 𝑀, 𝑁 } .
Revised: M.S. would like to thank the Swiss

Original: Let 𝜎 = ( 𝜎 1 , 𝜎 2 , ..., 𝜎 𝑄 ) ⊤ and the singular value distribution be [Equation] with 𝑘 = 1 , 2 , ..., 𝑄.
Revised: National Science Foundation for supporting the research with grant 167333 as part of the Swiss National Research Programme NRP 75 "Big Data".

Original: The eﬀective rank of matrix 𝐴 is then deﬁned as [Equation] 6) where 𝐻 ( 𝑝 1 , 𝑝 2 , ..., 𝑝 𝑄 ) is the Shannon entropy 𝐻 ( 𝑝 1 , 𝑝 2 , ..., 𝑝 [Equation]
Revised: Calculations were performed at sciCORE ( http://scicore .

Original: In comparison to the usual notion of rank, an important property of the eﬀective rank is that 𝑒𝑟𝑎𝑛𝑘 ( 𝐴 ) ≤ 𝑟𝑎𝑛𝑘 ( 𝐴 ) [Roy and Vetterli, 2007]. A.4 TRAIN AND TEST ACCURACY VALUES
Revised: unibas .

Original: Tables A.3 and A.4 provide the values for train and test accuracy in the LeNet and AlexNet experiments, respectively. The sample mean accuracy as well as sample standard deviation for 5 independent reruns of models are shown. [Table] MNIST with translation (see Fig. 3)
Revised: ch/ ) scientific computing core facility at University of Basel and Amazon

Original: [Table] CIFAR-10 (see Fig. 4) [Table] [Table]
Revised: Web Services (AWS).