Original: A SYMPTOTICS OF W IDE N ETWORKS FROM F EYNMAN D IAGRAMS
Revised: A SYMPTOTICS OF W IDE N ETWORKS FROM F EYNMAN D IAGRAMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding the asymptotic behavior of wide networks is of considerable interest.
Revised: Understanding the asymptotic behavior of wide networks is of considerable interest.

Original: In this work, we present a general method for analyzing this large width behavior.
Revised: In this work, we present a general method for analyzing this large width behavior.

Original: The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals
Revised: The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals.

Original: We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent.
Revised: We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent.

Original: Going beyond the strict large width limit , we present closed-form expressions for higher-order terms governing wide network training and test these predictions empirically
Revised: Going beyond the strict large width limit, we present closed-form expressions for higher-order terms governing wide network training, and test these predictions empirically.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks achieve remarkable performance on a wide array of machine learning tasks, yet a complete analytic understanding of deep networks remains elusive.
Revised: Neural networks achieve remarkable performance on a wide array of machine learning tasks, yet a complete analytic understanding of deep networks remains elusive.

Original: One promising approach is to consider the large width limit, in which the number of neurons in one or several layers is taken to be large.
Revised: One promising approach is to consider the large width limit, in which the number of neurons in one or several layers is taken to be large.

Original: In this limit one can use a mean-ﬁeld approach to better understand the network’s properties at initialization (Neal (1996); Lee et al.
Revised: In this limit one can use a mean-ﬁeld approach to better understand the network’s properties at initialization (Neal (1996); Lee et al.

Original: (2018)), as well as its training dynamics (Daniely (2017);
Revised: (2018)), as well as its training dynamics (Daniely (2017);

Original: Jacot et al.
Revised: Jacot et al.

Original: Additional related works are cited below.
Revised: Additional related works are cited below.

Original: Suppose that f ( x ) is the network function evaluated at an input x .
Revised: Suppose that f ( x ) is the network function evaluated at an input x .

Original: Let us denote the vector of model parameters by θ , whose elements are initially chosen to be i.i.d.
Revised: Let us denote the vector of model parameters by θ , whose elements are initially chosen to be i.i.d.

Original: Gaussian.
Revised: Gaussian.

Original: In this work we consider a class of functions we call correlation functions , obtained by taking the ensemble averages of f , its products, and its derivatives with respect to the parameters θ , evaluated on arbitrary inputs.
Revised: In this work we consider a class of functions we call correlation functions , obtained by taking the ensemble averages of f , its products, and its derivatives with respect to the parameters θ , evaluated on arbitrary inputs.

Original: Here are a few examples of correlation functions.
Revised: Here are a few examples of correlation functions.

Original: [Equation]
Revised: [Equation]

Original: x 3 ) ∂θ µ ∂θ ν f (

Original: [Equation]

Original: ) (cid:21) .

Original: Correlation functions often show up in the study of wide networks.
Revised: Correlation functions often show up in the study of wide networks.

Original: For example, the ﬁrst correlation function in (1) plays a central role in the Gaussian Process picture of wide networks (Lee et al.
Revised: For example, the ﬁrst correlation function in (1) plays a central role in the Gaussian Process picture of wide networks (Lee et al.

Original: and has been used to diagnose signal propagation in wide networks (Pennington et al.
Revised: and has been used to diagnose signal propagation in wide networks (Pennington et al.

Original: The second example in (1) is the ensemble average of the Neural Tangent Kernel (NTK), which controls the evolution of wide networks under gradient ﬂow (Jacot et al.
Revised: The second example in (1) is the ensemble average of the Neural Tangent Kernel (NTK), which controls the evolution of wide networks under gradient ﬂow (Jacot et al.

Original: and the third example shows up when computing the time derivative of the NTK with MSE loss.
Revised: and the third example shows up when computing the time derivative of the NTK with MSE loss.

Original: While correlation functions can be computed analytically in some special cases (Cho & Saul (2009)), they are not analytically tractable in general.
Revised: While correlation functions can be computed analytically in some special cases (Cho & Saul (2009)), they are not analytically tractable in general.

Original: In this work, we present a method for bounding the asymptotic behavior of such functions at large width.
Revised: In this work, we present a method for bounding the asymptotic behavior of such functions at large width.

Revised: In many cases of interest, the bound we obtain is tight.

Original: Derivation of the method relies on Feynman diagrams (Feynman (1949)), a technique for calculating multivariate Gaussian integrals, and speciﬁcally on the ’t Hooft expansion (’t Hooft (1974)).
Revised: Derivation of the method relies on Feynman diagrams (Feynman (1949)), a technique for calculating multivariate Gaussian integrals, and speciﬁcally on the ’t Hooft expansion (’t Hooft (1974)).

Original: However, applying the method is straightforward and does not require any knowledge of Feynman diagrams. 
Revised: However, applying the method is straightforward and does not require any knowledge of Feynman diagrams. Motivation.

Revised: The method presented here is a general-purpose tool that can be added to a researcher’s toolbox.

Revised: It allows a researcher to quickly identify ways in which network behavior and training dynamics simplify at large width, for example by deriving which quantities vanish in this limit.

Revised: As we demonstrate, using this method one is able to dramatically cut down on the amount of effort required to derive several key results related to wide networks, as well as to go beyond the inﬁnite width limit and investigate ordinary neural networks.

Original: Our contribution.
Revised: Our contribution.

Original: We present a general method for bounding the asymptotic behavior of correlation functions .
Revised: We present a general method for bounding the asymptotic behavior of correlation functions .

Original: The method is an adaptation of Feynman diagrams to the case of wide neural networks.
Revised: The method is an adaptation of Feynman diagrams to the case of wide neural networks.

Original: The adaptation involves a novel treatment of derivatives of the network function, an element that is not present in the original theoretical physics formulation.
Revised: The adaptation involves a novel treatment of derivatives of the network function, an element that is not present in the original theoretical physics formulation.

Original: We apply the method to the study of wide network evolution under gradient descent.
Revised: We apply the method to the study of wide network evolution under gradient descent.

Original: We improve on existing results for gradient ﬂow (Jacot et al.
Revised: We improve on existing results for gradient ﬂow (Jacot et al.

Original: by deriving tighter bounds, and extending the analysis to the case of stochastic gradient descent (SGD).
Revised: by deriving tighter bounds, and extending the analysis to the case of stochastic gradient descent (SGD).

Original: Going beyond the inﬁnite-width limit, we present a formalisn for deriving ﬁnite-width corrections to network evolution, and present explicit formulas for the ﬁrst order correction.
Revised: Going beyond the inﬁnite-width limit, we present a formalisn for deriving ﬁnite-width corrections to network evolution, and present explicit formulas for the ﬁrst order correction.

Original: To our knowledge, this is the ﬁrst time this correction has been calculated.
Revised: To our knowledge, this is the ﬁrst time this correction has been calculated.

Original: Limitations of our approach.
Revised: Limitations of our approach.

Original: The main result of this paper is a conjecture.
Revised: The main result of this paper is a conjecture.

Original: We test our predictions extensively using numerical experiments, and prove the conjecture in some cases, but we do not have a proof that applies to all the cases we tested, including for deep networks with general non-linearities.
Revised: We test our predictions extensively using numerical experiments, and prove the conjecture in some cases, but we do not have a proof that applies to all the cases we tested, including for deep networks with general non-linearities.

Original: Furthermore, our method can only be used to derive asymptotic bounds at large width; it does not produce the width-independent coefﬁcient, which is often of interest.
Revised: Furthermore, our method can only be used to derive asymptotic bounds at large width; it does not produce the width-independent coefﬁcient, which is often of interest.

Original: Related work.
Revised: Related work.

Original: For additional works on wide networks, including relating them to Gaussian processes, see de G. Matthews et al.
Revised: For additional works on wide networks, including relating them to Gaussian processes, see de G. Matthews et al.

Original: ; Novak et al.
Revised: ; Novak et al.

Original: Garriga-Alonso et al.
Revised: Garriga-Alonso et al.

Original: ; Yang (2019); Pennington & Bahri (2017); Pennington & Worah (2018); Schoenholz et al.
Revised: ; Yang (2019); Pennington & Bahri (2017); Pennington & Worah (2018); Schoenholz et al.

Original: ; Xiao et al.
Revised: ; Xiao et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Daniely et al. (2016); Lee et al.
Revised: Daniely et al. (2016); Lee et al.

Original: Du et al.
Revised: Du et al.

Original: (2018b;a); Allen-Zhu et al.
Revised: (2018b;a); Allen-Zhu et al.

Original: For additional works discussing the training dynamics of wide networks see Geiger et al.
Revised: For additional works discussing the training dynamics of wide networks see Geiger et al.

Original: (2019a)
Revised: (2019a)

Original: ; Arora et al. (2019).
Revised: ; Arora et al. (2019).

Original: For a previous use of diagrams in this context, see Pennington &
Revised: For a previous use of diagrams in this context, see Pennington &

Original: Worah (2017).
Revised: Worah (2017).

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2 we present our main conjecture and supporting evidence.
Revised: In Section 2 we present our main conjecture and supporting evidence.

Original: In Section 3 we apply the method to gradient descent evolution of wide networks, and in Section 4 we present details on Feynman diagrams, which is the basic technique used in our proofs.
Revised: In Section 3 we apply the method to gradient descent evolution of wide networks, and in Section 4 we present details on Feynman diagrams, which is the basic technique used in our proofs.

Original: We conclude with a Discussion.
Revised: We conclude with a Discussion.

Original: Proofs, additional applications, and details can be found in the Appendices.
Revised: Proofs, additional applications, and details can be found in the Appendices.

Original:  C ORRELATION FUNCTION ASYMPTOTICS
Revised:  C ORRELATION FUNCTION ASYMPTOTICS

Original: In this section we present our main result: a method for computing asymptotic bounds on correlation functions of wide networks.
Revised: In this section we present our main result: a method for computing asymptotic bounds on correlation functions of wide networks.

Original: We present the result as a conjecture, supported by analytic and empirical evidence.
Revised: We present the result as a conjecture, supported by analytic and empirical evidence.

Original: 2. N OTATION
Revised: 2. N OTATION

Original: Let f ( x ) ∈ R be the network output of a deeplinear network with d hidden layers and input x ∈ R D in , deﬁned by
Revised: Let f ( x ) ∈ R be the network output of a deep network with d hidden layers and input x ∈ R D in , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Here U ∈ R n × D in , V ∈ R n , W 1 , . . .
Revised: Here U ∈ R n × D in , V ∈ R n , W 1 , . . .

Original: , W d − 1 are weight matrices of dimension n , and σ : R → R is the non-linearity.
Revised: , W d − 1 are weight matrices of dimension n , and σ : R → R is the non-linearity.

Original: 1 We denote the vector of all model parameters by θ .
Revised: 1 We denote the vector of all model parameters by θ .

Original: At initialization, the elements of θ are independent Gaussian variables, with each element θ µ ∼ N (0 , 1) .
Revised: At initialization, the elements of θ are independent Gaussian variables, with each element θ µ ∼ N (0 , 1) .

Original: The corresponding distribution of θ is denoted by P 0 .
Revised: The corresponding distribution of θ is denoted by P 0 .

Original: Let us now deﬁne correlation functions , the class of functions that is the focus of this work.
Revised: Let us now deﬁne correlation functions , the class of functions that is the focus of this work.

Original: These functions involve derivative tensors of the network function.
Revised: These functions involve derivative tensors of the network function.

Original: We denote the rank- k derivative tensor by T µ 1 ...µ k ( x ; f ) :
Revised: We denote the rank- k derivative tensor by T µ 1 ...µ k ( x ; f ) :

Original: =
Revised: =

Original: ∂ k f ( x ) /∂θ µ 1 · · · ∂θ µ k .
Revised: ∂ k f ( x ) /∂θ µ 1 · · · ∂θ µ k .

Original: For k = 0 we deﬁne T ( x ; f ) :
Revised: For k = 0 we deﬁne T ( x ; f ) :

Original: = f ( x ) , and still refer to this as a derivative tensor for consistency.
Revised: = f ( x ) , and still refer to this as a derivative tensor for consistency.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A correlation function is the expectation value of a product of derivative tensors, evaluated at arbitrary inputs, where the tensor indices are summed in pairs over all the model parameters.
Revised: A correlation function is the expectation value of a product of derivative tensors, evaluated at arbitrary inputs, where the tensor indices are summed in pairs over all the model parameters.

Original: A general correlation function C takes the form
Revised: A general correlation function C takes the form

Original: [Equation]
Revised: [Equation]

Original: Here, 0 ≤ k 1 ≤ · · · ≤ k m − 1 ≤ k m are integers, 3 m and k m are even, π ∈ S k m is a permutation, and ∆ ( π ) µ1 ...µ km =
Revised: Here, 0 ≤ k 1 ≤ · · · ≤ k m − 1 ≤ k m are integers, 3 m and k m are even, π ∈ S k m is a permutation, and ∆ ( π ) µ

Original: δ µ π (1) µ π (2) · · · δ µ π ( km − 1) µ π ( km ) .
Revised: [Equation] [Equation]

Original: We use δ to denote the Kronecker delta.
Revised: We use δ to denote the Kronecker delta.

Original: If two derivative tensors in a correlation function have matching indices that are summed over, we say that they are contracted .
Revised: If two derivative tensors in a correlation function have matching indices that are summed over, we say that they are contracted .

Original: For example, the correlation function (cid:80) µ E θ [ ∂f ( x 1 ) /∂θ µ · ∂f ( x 2 ) /∂θ µ ] has one pair of contracted tensors.
Revised: For example, the correlation function (cid:80) [Equation] ] has one pair of contracted tensors.

Original: See (1) for additional examples of correlation functions.
Revised: See (1) for additional examples of correlation functions.

Original: 2. A SYMPTOTIC BOUNDS ON WIDE NETWORKS
Revised: A SYMPTOTIC BOUNDS ON WIDE NETWORKS

Original: We now present our main conjecture, which allows us to place asymptotic bounds on general correlation functions of wide networks.
Revised: We now present our main conjecture, which allows us to place asymptotic bounds on general correlation functions of wide networks.

Original: Conjecture 1.
Revised: Conjecture 1.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function.
Revised: , x m ) be a correlation function.

Original: The cluster graph G C
Revised: The cluster graph G C

Original: ( V, E ) of C is a graph with vertices V
Revised: ( V, E ) of C is a graph with vertices V

Original: = { v 1 , . .
Revised: = { v 1 , . .

Original: . , v m }  and edges E
Revised: , v m } , where v i = T ( x i ) for all i , and edges E

Original: = { ( v i , v j ) | ( T ( x i ) , T ( x j )) contracted in C } .
Revised: = { ( v i , v j ) | ( T ( x i ) , T ( x j )) contracted in C } .

Original: Suppose that the cluster graph G C has n e connected components with an even size (even number of vertices), and n o components of odd size.
Revised: Suppose that the cluster graph G C has n e connected components with an even size (even number of vertices), and n o components of odd size.

Original: Then C ( x 1 , . . .
Revised: Then C ( x 1 , . . .

Original: , x m )
Revised: , x m )

Original: =
Revised: =

Original: O ( n s C ) , where
Revised: O ( n s C ) , where

Original: [Equation]
Revised: [Equation]

Original: We will refer to the connected components of a cluster graph G C as the clusters of C .
Revised: We will refer to the connected components of a cluster graph G C as the clusters of C .

Original: Table 1 lists examples of bounds derived using the Conjecture for several correlation functions.
Revised: Table 1 lists examples of bounds derived using the Conjecture for several correlation functions.

Original: The intuition behind Conjecture 1 comes from the following result for deep linear networks.
Revised: The intuition behind Conjecture 1 comes from the following result for deep linear networks.

Original: Theorem 1.
Revised: Theorem 1.

Original: Conjecture 1 holds for correlation functions of networks with linear activations.
Revised: Conjecture 1 holds for correlation functions of networks with linear activations, including for deep linear networks with biases (afﬁne transformations).

Original: Let us discuss the intuition behind this theorem.
Revised: Let us discuss the intuition behind this theorem.

Original: Computing correlation functions of deep linear networks amounts to evaluating Gaussian integrals with polynomial integrands in θ .
Revised: Computing correlation functions of deep linear networks amounts to evaluating Gaussian integrals with polynomial integrands in θ .

Original: One can evaluate such integrals using Isserlis’ theorem, which tells us how to express moments of multivariate Gaussian variables in terms of their second moments.
Revised: One can evaluate such integrals using Isserlis’ theorem, which tells us how to express moments of multivariate Gaussian variables in terms of their second moments.

Original: For example, given centered Gaussian variables z 1 , ..., z 4 ,
Revised: For example, given centered Gaussian variables z 1 , ..., z 4 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, correlation functions of deep linear networks can be expressed in terms of the covariances
Revised: Therefore, correlation functions of deep linear networks can be expressed in terms of the covariances

Original: E 
Revised: E θ

Original: [Equation]
Revised: [ U iα U jβ ]

Original:  and E [Equation]
Revised: = δ ij δ αβ , E θ [ V i V j ] = δ ij , and E θ (cid:104) W ( l ) ij W

Original:  l
Revised: ( l

Original: [Equation]
Revised: [Equation]

Revised: kl (cid:105) =

Original: = δ ik δ jl .
Revised: δ ik δ jl .

Original: For example, for a deep linear network with 2 hidden layers, we have
Revised: For example, for a deep linear network with 2 hidden layers, we have

Original: [Equation]
Revised: [Equation]

Original: Every correlation function of a deep linear network can be similarly reduced to sums over products of Kronecker delta functions and width-independent functions of the inputs.
Revised: Every correlation function of a deep linear network can be similarly reduced to sums over products of Kronecker delta functions and width-independent functions of the inputs.

Original: The asymptotic large width behavior is determined by these sums over delta functions, which are tedious to compute by hand.
Revised: The asymptotic large width behavior is determined by these sums over delta functions, which are tedious to compute by hand.

Original: Feynman diagrams are a graphical tool for computing these sums, allowing us to obtain the asymptotic behavior with minimal effort.
Revised: Feynman diagrams are a graphical tool for computing these sums, allowing us to obtain the asymptotic behavior with minimal effort.

Original: This tool, which is described in detail in Section 4, is used to prove Theorem 1.
Revised: This tool, which is described in detail in Section 4, is used to prove Theorem 1.

Original: For networks with non-linear activations we further show the following
Revised: For networks with non-linear activations we further show the following

Original: Theorem 2.
Revised: Theorem 2.

Original: Conjecture 1 holds for (1) networks with ReLU activations, where all inputs are set to be equal, and for (2) networks with one hidden layer and smooth activation.
Revised: Conjecture 1 holds for (1) networks with ReLU activations, where all inputs are set to be equal, and for (2) networks with one hidden layer and smooth activation.

Original: For case (1), the idea behind the proof is to put an asymptotic bound on the ReLU network in terms of a corresponding deep linear network.
Revised: For case (1), the idea behind the proof is to put an asymptotic bound on the ReLU network in terms of a corresponding deep linear network.

Original: For case (2), the basic idea is that each network function contains a single sum over the width, and by keeping track of these sums using Feynman diagrams we are able to bound the asymptotic behavior.
Revised: For case (2), the basic idea is that each network function contains a single sum over the width, and by keeping track of these sums using Feynman diagrams we are able to bound the asymptotic behavior.

Original: We refer the reader to Appendix C for details.
Revised: We refer the reader to Appendix C for details.

Original: 2. N UMERICAL EXPERIMENTS
Revised: 2. N UMERICAL EXPERIMENTS

Original: Table 1 lists asymptotic bounds on several correlation functions, derived using Conjecture 1.
Revised: Table 1 lists asymptotic bounds on several correlation functions, derived using Conjecture 1.

Original: These are compared against the asymptotic behavior computed using numerical experiments.
Revised: These are compared against the asymptotic behavior computed using numerical experiments.

Original: In addition to the results presented here, we performed experiments using the same correlation functions and experimental setup [Table] but with weights sampled uniformly from {± 1 } instead of from a Gaussian distribution.
Revised: In addition to the results presented here, we performed experiments using the same correlation functions and experimental setup, but with weights sampled uniformly from {± 1 } instead of from a Gaussian distribution.

Original: The results are shown in Appendix A.1.
Revised: The results are shown in Appendix A.1.

Original: In all cases tested, we found that Conjectureholds.
Revised: In all cases that were tested empirically, we found that Conjecture 1 holds.

Original: In most cases, we  ﬁnd that the  bound is  tight. 
Revised: For networks with smooth activations, we found that Conjecture 1 always gives a tight bound. For networks with linear or ReLU activations, we always ﬁnd that the Conjecture holds as an upper bound, but that sometimes the bound is not tight. One such case is highlighted in Table 1.

Original: For cases where the bound is not tight, a tight bound can be obtained  using the complete Feynman diagram analysis presented below.
Revised: In such cases, a tight bound can be obtained for deep linear networks using the complete Feynman diagram analysis presented below.

Original: [Table]
Revised: [Table]

Original: -connected networks with 3 hidden layers and with linear, ReLU, and tanh activations.

Original: The numerical results are obtained by computing the correlation functions for networks with widths 2 7 , 2 8 , . . .

Original: , 2 13 , each averaged over 1,000 initializations, and ﬁtting the exponent.

Original: Inputs are chosen to be random vectors of dimension 4.

Original:  A PPLICATIONS TO TRAINING DYNAMICS
Revised:  A PPLICATIONS TO TRAINING DYNAMICS

Original: In this section we apply Conjecture 1 to study the evolution of wide networks under gradient ﬂow and gradient descent.
Revised: In this section we apply Conjecture 1 to study the evolution of wide networks under gradient ﬂow and gradient descent.

Original: We begin by brieﬂy reviewing existing results.
Revised: We begin by brieﬂy reviewing existing results.

Original: Let D tr be a training set of size M , and let L = (cid:80)
Revised: Let D tr be a training set of size M , and let L = (cid:80)

Original: ( x,y ) ∈ D tr (cid:96) ( x, y ) be the MSE loss, with single sample loss (cid:96) ( x, y )
Revised: ( x,y ) ∈ D tr (cid:96) ( x, y ) be the MSE loss, with single sample loss (cid:96) ( x, y )

Original: =
Revised: =

Original: 12 ( f ( x ) − y ) 2 .
Revised: 12 ( f ( x ) − y ) 2 .

Original: The gradient ﬂow equation is dθdt =
Revised: The gradient ﬂow equation is dθdt =

Original: −∇ θ L .
Revised: −∇ θ L .

Original: The evolution of the network function under gradient ﬂow is given by
Revised: The evolution of the network function under gradient ﬂow is given by

Original: [Equation]
Revised: [Equation]

Revised: ∂(cid:96) ( x (cid:48) , y (cid:48) )

Original:  Here, Θ is the Neural Tangent Kernel (NTK), deﬁned by Θ( x 1 , x 2 ) :
Revised: ∂f . Here, Θ is the Neural Tangent Kernel (NTK), deﬁned by Θ( x 1 , x 2 ) :

Original: = ∇ θ f T ( x 1 ) ∇ θ f ( x 2 ) .
Revised: = ∇ θ f T ( x 1 ) ∇ θ f ( x 2 ) .

Original: The authors of Jacot et al.
Revised: The authors of Jacot et al.

Original: (2018) showed that the kernel is constant during training up to O ( n − 1 / 2 ) corrections.
Revised: (2018) showed that the kernel is constant during training up to O ( n − 1 / 2 ) corrections.

Original: This leads to a dramatic simpliﬁcation in training dynamics (Jacot et al.
Revised: This leads to a dramatic simpliﬁcation in training dynamics (Jacot et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: In particular, for MSE loss the network map evaluated on the training data evolves as f ( t ) =
Revised: In particular, for MSE loss the network map evaluated on the training data evolves as f ( t ) =

Original: y + e − t Θ (0) ( f (0) − y ) .
Revised: y + e − t Θ (0) ( f (0) − y ) .

Original: 4 We will use our technology to derive a tighter bound on ﬁnite-width corrections to the kernel during training and present explicit formulas for the leading correction.
Revised: 5 We will use our technology to derive a tighter bound on ﬁnite-width corrections to the kernel during training and present explicit formulas for the leading correction.

Original: The following result is useful in analyzing the behavior of correlation functions under gradient ﬂow.
Revised: The following result is useful in analyzing the behavior of correlation functions under gradient ﬂow.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let C ( (cid:126)x ) =
Revised: Let C ( (cid:126)x ) =

Original: E θ [ F ( (cid:126)x )] be a correlation function, where (cid:126)x = ( x 1 , . . .
Revised: E θ [ F ( (cid:126)x )] be a correlation function, where (cid:126)x = ( x 1 , . . .

Original: , x m ) , and suppose that C
Revised: , x m ) , and suppose that C

Original: = O ( n s C ) for s C as deﬁned in Conjecture 1.
Revised: = O ( n s C ) for s C as deﬁned in Conjecture 1.

Original: Then E θ (cid:104)
Revised: Then E θ (cid:104)

Original: d k F ( (cid:126)x ) dt k (cid:105) =
Revised: d k F ( (cid:126)x ) dt k (cid:105) =

Original: O ( n s C ) for all k .
Revised: O ( n s C ) for all k .

Original: Here we prove the statement for k = 1 .
Revised: Here we prove the statement for k = 1 .

Original: Appendix D.2 contains a proof for the general case.
Revised: Appendix D.2 contains a proof for the general case.

Original: proof ( k = 1 ).
Revised: proof ( k = 1 ).

Original: Let C have n e even clusters and n o odd clusters.
Revised: Let C have n e even clusters and n o odd clusters.

Original: Consider E 
Revised: Consider E θ (cid:104) dF ( (cid:126)x ) dt (cid:105) = − [Equation] ( (cid:126)x ) [Equation]

Original: [Equation]
Revised: [Equation]

Revised: (cid:48) )

Revised: ∂θ µ f ( x (cid:48) )

Revised: (cid:105) .

Original: Denote by n (cid:48) e ( n (cid:48)
Revised: Denote by n (cid:48) e ( n (cid:48)

Original: o )
Revised: o )

Original: the number of even (odd) clusters in this correlation function, which has m (cid:48)
Revised: the number of even (odd) clusters in this correlation function, which has m (cid:48)

Original: = m
Revised: = m

Original: + 2 derivative tensors.
Revised: + 2 derivative tensors.

Original: One can check that either ( n (cid:48) e , n (cid:48) 
Revised: One can check that either ( n (cid:48) e , n (cid:48) o )

Original: o )
Revised: =

Original: = ( n e + 1 , n o ) or ( n (cid:48) e , n (cid:48)
Revised: ( n e + 1 , n o ) or ( n (cid:48) e , n (cid:48)

Original: o )
Revised: o )

Original: =
Revised: =

Original: ( n e − 1 , n o + 2) , depending on whether the ∂ µ F derivative is acting on an odd or even cluster in F . 

Revised: ( n e − 1 , n o + 2) , depending on whether the ∂ µ F derivative is acting on an odd or even cluster in F .

Original: Therefore, n (cid:48) 
Revised: Therefore, n (cid:48) e +

Original: e + n (cid:48) o 2 − m (cid:48) 2 ≤ s C .
Revised: [Equation]

Original: With this result, it is easy to understand the constancy of the NTK at large width.
Revised: With this result, it is easy to understand the constancy of the NTK at large width.

Original: The ﬁrst derivative of the NTK is given by
Revised: The ﬁrst derivative of the NTK is given by

Original: [Equation]
Revised: [Equation]

Revised: Here, ( x 1 ↔ x 2 ) means “add the same term, but exchange x 1 and x 2 ”.

Original: This correlation function has n e = 0 , n o = 2 , and m = 4 , and is therefore O ( n − 1 ) by Conjecture 1.
Revised: This correlation function has n e = 0 , n o = 2 , and m = 4 , and is therefore O ( n − 1 ) by Conjecture 1.

Original: By Lemma 1, all higher-order time derivatives of the NTK are O ( n − 1 ) as well.
Revised: By Lemma 1, all higher-order time derivatives of the NTK are O ( n − 1 ) as well.

Original: If we now assume that the time-evolved kernel Θ( t ) is analytic in training time t , and that we are free to exchange the Taylor expansion in time with the large width limit, then we ﬁnd that E θ [Θ( t ) − Θ(0)] = (cid:80) ∞ k = [Equation]
Revised: If we now assume that the time-evolved kernel Θ( t ) is analytic in training time t , and that we are free to exchange the Taylor expansion in time with the large width limit, then we ﬁnd that E θ [

Original: k k !

Original: E

Original: [Equation]
Revised: [Equation]

Original: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original: 0) dt k (cid:105) = O ( n − 1 ) for any ﬁxed t .
Revised: ) for any ﬁxed t .

Original: This bound, which is tighter than that found in Jacot et al.
Revised: This bound, which is tighter than that found in Jacot et al.

Original: (2018), was noticed empirically in Lee et al.
Revised: (2018), was noticed empirically in Lee et al.

Original: (2019) as well as in our own experiments, see Figure 1a.
Revised: (2019) as well as in our own experiments, see Figure 1a.

Original: This analysis can be extended to show that E θ [Θ( t ) − Θ(0)]
Revised: This analysis can be extended to show that E θ [Θ( t ) − Θ(0)]

Original: = O ( n − 1 ) for SGD as well.
Revised: = O ( n − 1 ) for SGD as well.

Original: The technique is similar, and again relies on Conjecture 1.
Revised: The technique is similar, and again relies on Conjecture 1.

Original: We refer the reader to Appendix E.1 for details.
Revised: We refer the reader to Appendix E.1 for details.

Original: These results improve on existing ones in several ways.
Revised: These results improve on existing ones in several ways.

Original: Our method applies to the case of SGD as well as to networks where all layer widths are increased simultaneously — a setup that has proven to be difﬁcult to analyze.
Revised: Our method applies to the case of SGD as well as to networks where all layer widths are increased simultaneously — a setup that has proven to be difﬁcult to analyze.

Original: In addition, the O ( n − 1 ) bound we derive on kernel corrections during SGD is empirically tight, and improves on the existing bound of O ( n − 1 / 2 ) which was derived for gradient ﬂow (Jacot et al.
Revised: In addition, the O ( n − 1 ) bound we derive on kernel corrections during SGD is empirically tight, and improves on the existing bound of O ( n − 1 / 2 ) which was derived for gradient ﬂow (Jacot et al.

Original: 3. F INITE WIDTH CORRECTIONS
Revised: 3. F INITE WIDTH CORRECTIONS

Original: Next, we will compute the explicit time dependence of Θ and f at order O ( n − 1 ) under gradient ﬂow.
Revised: Next, we will compute the explicit time dependence of Θ and f at order O ( n − 1 ) under gradient ﬂow.

Original: This is the leading correction to the inﬁnite width result.
Revised: This is the leading correction to the inﬁnite width result.

Original: We deﬁne the functions O 1 ( x ) :
Revised: We deﬁne the functions O 1 ( x ) :

Original: = f ( x ) and
Revised: = f ( x ) and

Original: [Equation]
Revised: [Equation]

Original: Notice that O 2 =
Revised: Notice that O 2 =

Original: Θ is the kernel.
Revised: Θ is the kernel.

Original: It is easy to check that
Revised: It is easy to check that

Original: [Equation]
Revised: [Equation]

Original: We can use equations (9) and (10) to solve for the evolution of the kernel and network map.
Revised: We can use equations (9) and (10) to solve for the evolution of the kernel and network map.

Original: Notice that each function O s has s derivative tensors and a single cluster.
Revised: Notice that each function O s has s derivative tensors and a single cluster.

Original: As a result, correlation functions involving operators with larger s are increasingly suppressed in width.
Revised: As a result, correlation functions involving operators with larger s are increasingly suppressed in width.

Original: In particular, E θ [ dO 4 /dt ]
Revised: In particular, E θ [ dO 4 /dt ]

Original: = − 
Revised: = − (cid:80) x ∈ D

Original: [Equation]
Revised: tr E θ [ O 5 ( x ) f ( x )]

Original:  2 ) at all times, using Conjecture 1 and Lemma 1.
Revised: = O ( n − 2 ) at all times, using Conjecture 1 and Lemma 1.

Original: Thus to solve for f and Θ at O ( n − 1 ) we can set O s = 0 for all s ≥ 5 .
Revised: Thus to solve for f and Θ at O ( n − 1 ) we can set O s = 0 for all s ≥ 5 .

Original: Let us denote the time-evolved kernel by Θ( t ) =
Revised: Let us denote the time-evolved kernel by Θ( t ) =

Original: Θ (0) +Θ 1 ( t )+ O ( n − 2 ) , where Θ (0) is the kernel at initialization, and Θ 1 ( t ) is the O ( n − 1 ) correction we are seeking.
Revised: Θ (0) +Θ 1 ( t )+ O ( n − 2 ) , where Θ (0) is the kernel at initialization, and Θ 1 ( t ) is the O ( n − 1 ) correction we are seeking.

Original: Integrating equations (10) starting with s = 4 , we ﬁnd
Revised: Integrating equations (10) starting with s = 4 , we ﬁnd

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we have introduced the notation ∆ f ( x ; t ) =
Revised: Here we have introduced the notation ∆ f ( x ; t ) =

Original: e − t Θ 0 ( f (0) − y ) .
Revised: e − t Θ 0 ( f (0) − y ) .

Original: A detailed derivation can be found in Appendix E.4.
Revised: A detailed derivation can be found in Appendix E.4.

Original: There we also evaluate the integrals in (11) in terms of the NTK spectrum.
Revised: There we also evaluate the integrals in (11) in terms of the NTK spectrum.

Original: To obtain the O ( n − 1 ) correction to the network map (evaluated for simplicity on the training data) [Figure] we further integrate (10) for s = 

Revised: To obtain the O ( n − 1 ) correction to the network map (evaluated for simplicity on the training data), we further integrate (10) for s =

Original: and ﬁnd
Revised: and ﬁnd

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we have denote the inﬁnite width evolution by f 0 ( t ) =
Revised: Here we have denote the inﬁnite width evolution by f 0 ( t ) =

Original: y + e − t Θ (0) ( f (0) − y ) .
Revised: y + e − t Θ (0) ( f (0) − y ) .

Original: Figures 1b and 1c compare these predictions against empirical results.
Revised: Figures 1b and 1c compare these predictions against empirical results.

Original: [Figure]
Revised: [Figure]

Revised: ( n − 1 ) predicted evolution (dashed) for the network function and the kernel.

Revised: All experiments were performed on two-class MNIST, computing a single randomly-chosen component of Θ or f .

Revised: See

Revised: Appendix A for additional experimental details.

Original:  F EYNMAN DIAGRAMS FOR DEEP LINEAR NETWORKS
Revised:  F EYNMAN DIAGRAMS FOR DEEP LINEAR NETWORKS

Original: In this section we present the Feynman diagram technique, and show how it allows us to compute the asymptotic behavior of correlation functions.
Revised: In this section we present the Feynman diagram technique, and show how it allows us to compute the asymptotic behavior of correlation functions.

Original: We end this section with a proof of Theorem 1 for the case of networks with a single hidden layer and linear activations.
Revised: We end this section with a proof of Theorem 1 for the case of networks with a single hidden layer and linear activations.

Original: Given a correlation function C , we map it to a family of graphs called Feynman diagrams.
Revised: Given a correlation function C , we map it to a family of graphs called Feynman diagrams.

Original: The graphs are independent of the inputs, and are deﬁned as follows.
Revised: The graphs are independent of the inputs, and are deﬁned as follows.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function for a network with d hidden layers.
Revised: , x m ) be a correlation function for a network with d hidden layers.

Original: The family Γ( C ) is the set of all graphs that have the following properties.
Revised: The family Γ( C ) is the set of all graphs that have the following properties.

Original: There are m vertices v 1 , . .
Revised: There are m vertices v 1 , . .

Original: . , v m , each of degree d + 1 . 2.
Revised: . , v m , each of degree d + 1 . 2.

Original: Each edge has a type t ∈ { U, W 1 , . . .
Revised: Each edge has a type t ∈ { U, W 1 , . . .

Original: , W d − 1 , V } .
Revised: , W d − 1 , V } .

Original: Every vertex has one edge of each type.
Revised: Every vertex has one edge of each type.

Revised: [Figure]

Original: If two derivative tensors T µ 1 ,...,µ k ( x i ) , T ν 1 ,...,ν k (cid:48) ( x j ) are contracted k times in C , the graph must have at least k edges (of any type) connecting thevertices v i , v j .
Revised: If two derivative tensors T µ [Figure] [Figure] x j ) are contracted k times in C , the graph must have at least k edges (of any type) connecting the

Revised: vertices v i , v j .

Original: The graphs in Γ( C ) are called the Feynman diagrams of C .
Revised: The graphs in Γ( C ) are called the Feynman diagrams of C .

Original: Single hidden layer.
Revised: Single hidden layer.

Original: For the rest of this section we focus on networks with a single hidden layer.
Revised: For the rest of this section we focus on networks with a single hidden layer.

Original: We refer the reader to Appendix B for a full treatment of deep linear networks.
Revised: We refer the reader to Appendix B for a full treatment of deep linear networks.

Original: For networks with one hidden layer and linear activation, the network output is f ( x ) = n − 1 / 2 V T Ux .
Revised: For networks with one hidden layer and linear activation, the network output is f ( x ) = n − 1 / 2 V T Ux .

Original: Consider the correlation function C ( x 1 , x 2 )
Revised: Consider the correlation function C ( x 1 , x 2 )

Original: =
Revised: =

Original: E θ [ f ( x 1 ) f
Revised: E θ [ f ( x 1 ) f

Original: ( x 2 )] .
Revised: ( x 2 )] .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: As the factors of x 1 and x 2 are independent of n , we see that C ( x 1 , x 2 ) =
Revised: As the factors of x 1 and x 2 are independent of n , we see that C ( x 1 , x 2 ) =

Original: O ( n 0 ) .
Revised: O ( n 0 ) .

Original: Notice that there are two relevant contributions to this answer: each factor of the network function in the integrand contributes n − 1 / 2 , and the summed-over product of Kronecker deltas contributes n .
Revised: Notice that there are two relevant contributions to this answer: each factor of the network function in the integrand contributes n − 1 / 2 , and the summed-over product of Kronecker deltas contributes n .

Original: Other details,[Figure] such as the input dependence, are irrelevant.
Revised: Other details, such as the input dependence, are irrelevant.

Original:  diagrams allow us to encode only those details that affect the n scaling, ignoring the rest.
Revised: Feynman diagrams allow us to encode only those details that affect the n scaling, ignoring the rest.

Original: The set Γ( C ) for the correlation function (13) consists of a single Feynman diagram, shown in
Revised: The set Γ( C ) for the correlation function (13) consists of a single Feynman diagram, shown in

Original: Figure 2a.
Revised: Figure 2a.

Original: The asymptotic bound on a correlation function is obtained by the following result, which is due to ’t Hooft (1974).
Revised: The asymptotic bound on a correlation function is obtained by the following result, which is due to ’t Hooft (1974).

Original: Theorem 3.
Revised: Theorem 3.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function with one hidden layer and linear activation.
Revised: , x m ) be a correlation function with one hidden layer and linear activation.

Original: Then C
Revised: Then C

Original: =
Revised: =

Original: O ( n s ) where s = max γ ∈ Γ( C ) l γ − m 2 , and l γ is the number of loops in γ .
Revised: O ( n s ) where s = max γ ∈ Γ( C ) l γ − m 2 , and l γ is the number of loops in γ .

Original: Let us give some intuition for Theorem 3.
Revised: Let us give some intuition for Theorem 3.

Original: Each Feynman diagram γ encodes a subset of the terms contributing to the correlation function.
Revised: Each Feynman diagram γ encodes a subset of the terms contributing to the correlation function.

Original: To get the asymptotic bound on the correlation function, we sum over the contributions of individual diagrams.
Revised: To get the asymptotic bound on the correlation function, we sum over the contributions of individual diagrams.

Original: We can compute the asymptotic behavior of a single diagram γ using the following Feynman rules : (1) each vertex contributes a factor of n − 1 / 2 , and (2) each loop contributes a factor of n .
Revised: We can compute the asymptotic behavior of a single diagram γ using the following Feynman rules : (1) each vertex contributes a factor of n − 1 / 2 , and (2) each loop contributes a factor of n .

Original: Therefore, if a diagram has l γ loops, its contribution to the correlation function scales as n l γ − m/ 2 .
Revised: Therefore, if a diagram has l γ loops, its contribution to the correlation function scales as n l γ − m/ 2 .

Original: Rule (1) is due to the explicit n − 1 / 2 factor in the network deﬁnition.
Revised: Rule (1) is due to the explicit n − 1 / 2 factor in the network deﬁnition.

Original: Rule (2) follows from applying Isserlis’ theorem, as follows.
Revised: Rule (2) follows from applying Isserlis’ theorem, as follows.

Original: Each covariance factor (such as the factor E
Revised: Each covariance factor (such as the factor E

Original: [ V i V j ] =
Revised: [ V i V j ] =

Original: δ ij in eq.
Revised: δ ij in eq.

Original: (13)) corresponds to an edge in a Feynman diagram.
Revised: (13)) corresponds to an edge in a Feynman diagram.

Original: A loop in the diagram corresponds to a sum over a product of Kronecker deltas, yielding a factor of n .
Revised: A loop in the diagram corresponds to a sum over a product of Kronecker deltas, yielding a factor of n .

Original: Returning to our example, the graph in Figure 2a has two vertices and one closed loop, so we recover the asymptotic behavior O ( n 0 ) from Theorem 3.
Revised: Returning to our example, the graph in Figure 2a has two vertices and one closed loop, so we recover the asymptotic behavior O ( n 0 ) from Theorem 3.

Original: As another example, consider the correlation function C ( x 1 , x 2 , x 3 , x 4 ) =
Revised: As another example, consider the correlation function C ( x 1 , x 2 , x 3 , x 4 ) =

Original: E θ [ f ( x 1 ) f
Revised: E θ [ f ( x 1 ) f

Original: ( x 2 ) f ( x 3 ) f
Revised: ( x 2 ) f ( x 3 ) f

Original: ( x 4 )] .
Revised: ( x 4 )] .

Original: It has two Feynman diagrams, shown in
Revised: It has two Feynman diagrams, shown in

Original: Figures 2b and 2c.
Revised: Figures 2b and 2c.

Original: Using the Feynman rules, we ﬁnd that the disconnected graph represents terms that scale as O ( n 0 ) , while the connected graph represents terms that scale as O ( n − 1 ) .
Revised: Using the Feynman rules, we ﬁnd that the disconnected graph represents terms that scale as O ( n 0 ) , while the connected graph represents terms that scale as O ( n − 1 ) .

Original: Therefore,
Revised: Therefore,

Original: C ( x 1 , x 2 , x 3 , x 4 ) =
Revised: C ( x 1 , x 2 , x 3 , x 4 ) =

Original: O ( n 0 ) .
Revised: O ( n 0 ) .

Original: This is an example of a more general phenomenon, that connected graphs vanish faster at large n compared with disconnected graphs. 
Revised: This is an example of a more general phenomenon, that connected graphs vanish faster at large n compared with disconnected graphs. [Figure]

Original: Correlation functions with derivatives.
Revised: Correlation functions with derivatives.

Original: We now extend the Feynman diagram technique to correlation functions that include derivatives of f . 6 As a concrete example, consider the correlation function C ( x, x (cid:48) )
Revised: We now extend the Feynman diagram technique to correlation functions that include derivatives of f . 8 As a concrete example, consider the correlation function C ( x, x (cid:48) )

Original: =
Revised: =

Original: E θ [Θ( x, x (cid:48) )] , where Θ is the kernel deﬁned in Section 3.
Revised: E θ [Θ( x, x (cid:48) )] , where Θ is the kernel deﬁned in Section 3.

Original: For a single hidden layer, we have
Revised: For a single hidden layer, we have

Original: [Equation]
Revised: [Equation]

Original: The two derivative tensors in this correlation function are contracted: their indices are set to be equal and summed over.
Revised: The two derivative tensors in this correlation function are contracted: their indices are set to be equal and summed over.

Original: Therefore, according to Deﬁnition 2, Γ( C ) includes all diagrams in which the corresponding vertices share at least one edge.
Revised: Therefore, according to Deﬁnition 2, Γ( C ) includes all diagrams in which the corresponding vertices share at least one edge.

Original: The resulting diagrams are shown in
Revised: The resulting diagrams are shown in

Original: Figure 3.
Revised: Figure 3.

Original: The edges forced by the contraction are explicitly marked by dashed lines for clarity, but mathematically they are ordinary edges.
Revised: The edges forced by the contraction are explicitly marked by dashed lines for clarity, but mathematically they are ordinary edges.

Original: We see that in fact there is only one diagram contributing to this correlation function — the same one shown in Figure 2a.
Revised: We see that in fact there is only one diagram contributing to this correlation function — the same one shown in Figure 2a.

Original: Following the Feynman rules, we ﬁnd that E θ[Θ( x, x (cid:48) )]
Revised: Following the Feynman rules, we ﬁnd that E θ

Original: [Equation]
Revised: [Equation]

Original: The fact that contracted derivatives should be mapped to forced edges in the Feynman diagrams is proved in Appendix B.
Revised: The fact that contracted derivatives should be mapped to forced edges in the Feynman diagrams is proved in Appendix B.

Original: The basic reason behind this rule is the relation 
Revised: The basic reason behind this rule is the relation (cid:80)

Original: [Equation]
Revised: [Equation]

Original: [Figure] Namely, when derivatives act in pairs they yield a Kronecker delta factor ( δ ij ), which is equal to the factor obtained from a covariance ( E [ V i V j ] ).
Revised: Namely, when derivatives act in pairs they yield a Kronecker delta factor ( δ ij ), which is equal to the factor obtained from a covariance ( E [ V i V j ] ).

Original: While Isserlis’ theorem instructs us to sum over all possible covariance conﬁgurations (and therefore over all possible edge conﬁgurations), a pair of summed derivative leads to a particular covariance factor.
Revised: While Isserlis’ theorem instructs us to sum over all possible covariance conﬁgurations (and therefore over all possible edge conﬁgurations), a pair of summed derivative leads to a particular covariance factor.

Original: Therefore, we should only consider graphs that include the edge corresponding to this covariance factor.
Revised: Therefore, we should only consider graphs that include the edge corresponding to this covariance factor.

Original: We are now ready to prove Theorem 1 for the case of single hidden layer with linear activations.
Revised: We are now ready to prove Theorem 1 for the case of single hidden layer with linear activations.

Original: A proof for the general case can be found in Appendix B.
Revised: A proof for the general case can be found in Appendix B.

Original: Proof (Theorem 1, one hidden layer).
Revised: Proof (Theorem 1, one hidden layer).

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function for a network with a single hidden layer and linear activation.
Revised: , x m ) be a correlation function for a network with a single hidden layer and linear activation.

Original: Let G C ( V, E ) be the cluster graph of C , and let γ ( V, E (cid:48) )
Revised: Let G C ( V, E ) be the cluster graph of C , and let γ ( V, E (cid:48) )

Original: ∈ Γ( C ) be a Feynman diagram.
Revised: ∈ Γ( C ) be a Feynman diagram.

Original: Notice that E ⊂ E (cid:48) .
Revised: Notice that E ⊂ E (cid:48) .

Original: Indeed, E (cid:48) contains an edge corresponding to each pair of contracted derivative tensors in C , and E = { ( v i , v j ) | ( T ( x i ) , T ( x j )) contracted in C } .
Revised: Indeed, E (cid:48) contains an edge corresponding to each pair of contracted derivative tensors in C , and E = { ( v i , v j ) | ( T ( x i ) , T ( x j )) contracted in C } .

Original: In addition, notice that γ only contains connected components ( i.e. loops) with an even number of vertices, because every vertex has exactly one edge of each type.
Revised: In addition, notice that γ only contains connected components ( i.e. loops) with an even number of vertices, because every vertex has exactly one edge of each type.

Original: Therefore, n γ ≤ n e + n o 2 , where n γ is the number of loops in γ , and n e ( n o ) is the number of even (odd) components in G C .
Revised: Therefore, n γ ≤ n e + n o 2 , where n γ is the number of loops in γ , and n e ( n o ) is the number of even (odd) components in G C .

Original: The bound is saturated when every even component of G C belongs to a different component of γ , and pairs of odd components in G C belong to different components of γ .
Revised: The bound is saturated when every even component of G C belongs to a different component of γ , and pairs of odd components in G C belong to different components of γ .

Original: From Theorem 3, we have that C
Revised: From Theorem 3, we have that C

Original: = O ( n s ) where s = max γ ∈ Γ( C ) n [Equation] − [Equation]
Revised: = O ( n s ) where s = max γ ∈ Γ( C ) n γ − m 2 ≤ n e + n o 2 − m 2 .

Original: 2 ≤ n e + n o 2 − m 2 .

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Ensemble averages of the network function and its derivatives are an important class of functions that often show up in the study of wide neural networks.
Revised: Ensemble averages of the network function and its derivatives are an important class of functions that often show up in the study of wide neural networks.

Original: Examples include the ensemble average of the train and test losses, the covariance of the network function, and the Neural Tangent Kernel (Jacot et al.
Revised: Examples include the ensemble average of the train and test losses, the covariance of the network function, and the Neural Tangent Kernel (Jacot et al.

Original: In this work we presented Conjecture 1, which allows one to derive the asymptotic behavior of such functions at large width.
Revised: In this work we presented Conjecture 1, which allows one to derive the asymptotic behavior of such functions at large width.

Original: For the case of deep linear networks, we presented a complete analytic understanding of the Conjecture based on Feynman diagrams.
Revised: For the case of deep linear networks, we presented a complete analytic understanding of the Conjecture based on Feynman diagrams.

Original: In addition, we presented empirical and anlytic evidence showing that the Conjecture also holds for deep networks with non-linear activations, as well as for networks with non-Gaussian initialization.
Revised: In addition, we presented empirical and analytic evidence showing that the Conjecture also holds for deep networks with non-linear activations, as well as for networks with non-Gaussian initialization.

Original: We found that the Conjecture holds in all cases we tested.
Revised: We found that the Conjecture holds in all cases we tested.

Original: The basic tools presented in this work can be applied to many aspects of wide network research, greatly simplifying theoretical calculations.
Revised: The basic tools presented in this work can be applied to many aspects of wide network research, greatly simplifying theoretical calculations.

Original: We presented several applications of our method to the asymptotic behavior of wide networks during stochastic gradient descent, and additional applications are presented in
Revised: We presented several applications of our method to the asymptotic behavior of wide networks during stochastic gradient descent, and additional applications are presented in

Original: Appendix E. We were able to improve upon known results by tightening existing bounds, and by applying the technique to SGD as well as to gradient ﬂow.
Revised: Appendix E. We were able to improve upon known results by tightening existing bounds, and by applying the technique to SGD as well as to gradient ﬂow.

Original: In addition, we took a step beyond the inﬁnite width limit, deriving closed-form expressions for the ﬁrst ﬁnite-width correction to the network evolution.
Revised: In addition, we took a step beyond the inﬁnite width limit, deriving closed-form expressions for the ﬁrst ﬁnite-width correction to the network evolution.

Original: These novel results open the door to studying ﬁnite-width networks by systematically expanding around the inﬁnite width limit.
Revised: These novel results open the door to studying ﬁnite-width networks by systematically expanding around the inﬁnite width limit.

Original: A central question in the study of wide networks is whether the inﬁnite width limit is a good model for describing the behavior of realistic deep networks (Chizat et al.
Revised: A central question in the study of wide networks is whether the inﬁnite width limit is a good model for describing the behavior of realistic deep networks (Chizat et al.

Original: ; Ghorbani et al.
Revised: ; Ghorbani et al.

Original: Geiger et al. (2019b)).
Revised: Geiger et al. (2019b)).

Original: In this work we take a step toward answering this question, by working out the next order in a perturbative expansion around the inﬁnite width limit, potentially bringing us closer to an analytic description of ﬁnite-width networks.
Revised: In this work we take a step toward answering this question, by working out the next order in a perturbative expansion around the inﬁnite width limit, potentially bringing us closer to an analytic description of ﬁnite-width networks.

Original: We hope that the techniques presented here provide a basis to systematically answering these and other questions about the behavior of wide networks.
Revised: We hope that the techniques presented here provide a basis to systematically answering these and other questions about the behavior of wide networks.

Original: A E XPERIMENTAL DETAILS AND ADDITIONAL RESULTS A. N ON -G AUSSIAN INITIALIZATION
Revised: A E XPERIMENTAL DETAILS AND ADDITIONAL RESULTS A. N ON -G AUSSIAN INITIALIZATION

Original: In Table 1 we listed asymptotic bounds on several correlation functions [Table] where the model parameters were initialized from a Gaussian distribution.
Revised: In Table 1 we listed asymptotic bounds on several correlation functions, where the model parameters were initialized from a Gaussian distribution.

Original: Table 2 shows additional results using the same experimental setup[Table] but with weights sampled uniformly from {± 1 } .
Revised: Table 2 shows additional results using the same experimental setup but with weights sampled uniformly from {± 1 } .

Original: We again ﬁnd good agreement with the predictions of Conjecture 
Revised: We again ﬁnd good agreement with the predictions of Conjecture 1. Correlation function C [Equation]

Original: [Table]
Revised: [Table]

Original: from a Gaussian distribution. We ﬁnd good agreement with the theoretical predictoins, and in many cases the bound is tight.
Revised: [Table]

Original: A. E XPERIMENTAL DETAILS
Revised: A. E XPERIMENTAL DETAILS

Original: The experiments in Figure 1 were performed on two-class MNIST, computing a single randomlychosen component of the kernel Θ .
Revised: The experiments in Figure 1 were performed on two-class MNIST, computing a single randomlychosen component of the kernel Θ .

Original: Sub-ﬁgure (a) uses networks trained for 1024 steps with learning rate 1.0 and 1000 samples per class, averaged over 100 initializations.
Revised: Sub-ﬁgure (a) uses networks trained for 1024 steps with learning rate 1.0 and 1000 samples per class, averaged over 100 initializations.

Original: Each curve in ﬁgure (b) represents a single instance of the network map evaluated on a random image over the corse of training.
Revised: Each curve in ﬁgure (b) represents a single instance of the network map evaluated on a random image over the corse of training.

Original: The models were trained with 10 samples per class and learning rate 0.1.
Revised: The models were trained with 10 samples per class and learning rate 0.1.

Original: The input to the network is normalized by the square root of the input dimension as in (Jacot et al.
Revised: The input to the network is normalized by the square root of the input dimension as in (Jacot et al.

Original: [Equation]
Revised: [Equation]

Original: B F EYNMAN DIAGRAMS FOR DEEP LINEAR NETWORKS
Revised: B F EYNMAN DIAGRAMS FOR DEEP LINEAR NETWORKS

Original: Feynman diagrams can be used to derive asymptotic upper bounds on deep linear networks in the large width limit.
Revised: Feynman diagrams can be used to derive asymptotic upper bounds on deep linear networks in the large width limit.

Original: In this section we describe the method in detail, and use it to prove Theorem 1.
Revised: In this section we describe the method in detail, and use it to prove Theorem [Equation]

Original: B. F EYNMAN DIAGRAMS AND DOUBLE - LINE DIAGRAMS
Revised: B. F EYNMAN DIAGRAMS AND DOUBLE - LINE DIAGRAMS

Original: In this section we build on the results of Section 4 and consider correlation functions of deep linear networks with d hidden layers.
Revised: In this section we build on the results of Section 4 and consider correlation functions of deep linear networks with d hidden layers.

Original: The network function was deﬁned in (2), and here we set the activation σ to be the identity.
Revised: The network function was deﬁned in (2) [Equation] and here we set the activation σ to be the identity.

Original: Deﬁnition 2 describes how to map a correlation function C to Γ( C ) , a family of graphs called Feynman diagrams.
Revised: Deﬁnition 2 describes how to map a correlation function C to Γ( C ) , a family of graphs called Feynman diagrams.

Original: The Feynman diagram method relies on Isserlis’ theorem, which allows us to express arbitrary moments of multivariate Gaussian variables in terms of their covariance.
Revised: The Feynman diagram method relies on Isserlis’ theorem [Equation] which allows us to express arbitrary moments of multivariate Gaussian variables in terms of their covariance [Equation]

Original: Theorem 4 (Isserlis) .
Revised: Theorem 4 (Isserlis) [Equation]

Original: Let z =
Revised: Let z =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z l ) be a centered multivariate Gaussian variable.
Revised: , z l ) be a centered multivariate Gaussian variable.

Original: For any positive integer k ,
Revised: For any positive integer k ,

Original: [Equation]
Revised: [Equation]

Original: In particular, if the covariance matrix of z is the identity then
Revised: In particular [Equation] if the covariance matrix of z is the identity then

Original: [Equation]
Revised: [Equation]

Original: Using this theorem
Revised: Using this theorem

Original: [Equation]
Revised: [Equation]

Original: a correlation function C can be expressed as a sum over permutations as in (16).
Revised: a correlation function C can be expressed as a sum over permutations as in (16).

Original: Each term in this sum maps to a Feynman diagram in Γ( C ) 
Revised: Each term in this sum maps to a Feynman diagram in Γ( C ) .

Original: [Equation] As an example [Equation] consider the correlation function C ( x, x (cid:48) )
Revised: As an example, consider the correlation function C ( x, x (cid:48) )

Original: =
Revised: =

Original: E θ [ f ( x ) f
Revised: E θ [ f ( x ) f

Original: ( x (cid:48) )] for a network withhidden layers.
Revised: ( x (cid:48) )] for a network withhidden layers.

Original: An explicit calculation gives
Revised: An explicit calculation gives

Original: [Equation]
Revised: [Equation]

Original: To get from (19) to (20) [Equation] we applied Isserlis’ theorem for every choice of indices i, j, k, l, α, β .
Revised: To get from (19) to (20), we applied Isserlis’ theorem for every choice of indices i, j, k, l, α, β .

Original: We ﬁnd that there is at most one permutation π of the network parameters such that the covariances do not vanish.
Revised: We ﬁnd that there is at most one permutation π of the network parameters such that the covariances do not vanish.

Original: This is because the covariance of parameters across different layers vanishes identically (for example E θ (cid:104)
Revised: This is because the covariance of parameters across different layers vanishes identically (for example E θ (cid:104)

Original: V
Revised: V

Original: i W 1 jk (cid:105) = 0 ).
Revised: i W 1 jk (cid:105) = 0 ).

Original: Correspondingly, this correlation function has a single Feynman diagram, shown in Figure 4a.
Revised: Correspondingly, this correlation function has a single Feynman diagram, shown in Figure 4a.

Original: For networks with one hidden layer, we saw in Section 4 that the asymptotic behavior is determined by the number of loops in a graph.
Revised: For networks with one hidden layer, we saw in Section 4 that the asymptotic behavior is determined by the number of loops in a graph.

Original: This observation does not immediately generalize to networks with general depth, because it is not obvious how to count loops in diagrams such as the one in
Revised: This observation does not immediately generalize to networks with general depth, because it is not obvious how to count loops in diagrams such as the one in

Original: Figure 4a.
Revised: Figure 4a.

Original: The problem can be traced back to the fact that weight matrices have covariances of the form E
Revised: The problem can be traced back to the fact that weight matrices have covariances of the form E

Original: [ W ij W kl ]
Revised: [ W ij W kl ]

Original: = δ
Revised: = δ

Original: ik δ jl involving two Kronecker deltas, but the procedure described so far assumes that each covariance (and each edge in the graph) corresponds to a single Kronecker delta.
Revised: ik δ jl involving two Kronecker deltas, but the procedure described so far assumes that each covariance (and each edge in the graph) corresponds to a single Kronecker delta.

Original: A similar question appeared in the context of theoretical physics, and the correct generalization is due to ’t Hooft (1974).
Revised: A similar question appeared in the context of theoretical physics, and the correct generalization is due to ’t Hooft (1974).

Original: The idea is to treat each Feynman diagram as the triangulation of a Riemann surface, and to deﬁne the number of loops in a graph to be the number of faces of the triangulation.
Revised: The idea is to treat each Feynman diagram as the triangulation of a Riemann surface, and to deﬁne the number of loops in a graph to be the number of faces of the triangulation.

Original: In practice, this involves mapping each Feynman diagram to a new double-line diagram : A graph in which each edge corresponds to a single Kronecker delta factor, and loops correspond to triangulation faces of the original diagram. 
Revised: In practice, this involves mapping each Feynman diagram to a new double-line diagram : A graph in which each edge corresponds to a single Kronecker delta factor, and loops correspond to triangulation faces of the original diagram. [Figure]

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Let γ ∈ Γ( C ) be a Feynman diagram for a correlation function C involving k derivative tensors for a network of depth d .
Revised: Let γ ∈ Γ( C ) be a Feynman diagram for a correlation function C involving k derivative tensors for a network of depth d .

Original: Its double-line graph , DL( γ ) is a graph with kd vertices of degree 2, deﬁned by the following blow-up procedure.
Revised: Its double-line graph , DL( γ ) is a graph with kd vertices of degree 2, deﬁned by the following blow-up procedure.

Original: • Each vertex v ( i ) in γ is mapped to d vertices v ( i ) 1 , . .
Revised: • Each vertex v ( i ) in γ is mapped to d vertices v ( i ) 1 , . .

Original: , v ( i ) d in DL( γ ) .
Revised: , v ( i ) d in DL( γ ) .

Original: • Each edge ( v (1) , v (2) ) in γ of type W l is mapped to two edges ( v(1) l , v (2) l ) , ( v (1) l +1 , v (2) l +1 ) . • Each edge ( v (1) , v (2) ) in γ of type U is mapped to a single edge ( v (1)1 , v (2)1 ) .
Revised: • Each edge ( v ( i ) , v ( j ) ) in γ of type U is mapped to a single edge ( v ( i ) 1 , v ( j ) 1 ) . • Each edge ( v ( i ) , v ( j ) ) in γ of type W l is mapped to two edges ( v ( i ) l , v ( j ) l ) , ( v ( i ) l +1 , v ( j ) l +1 ) .

Original: • Each edge ( v (1) , v (2) ) in γ of type V is mapped to a single edge ( v (1) d , v (2) d ) .
Revised: • Each edge ( v ( i ) , v ( j ) ) in γ of type V is mapped to a single edge ( v ( i ) d , v ( j ) d ) .

Original: The number of faces in γ is given by the number of loops in the double-line graph DL( γ ) .
Revised: The number of faces in γ is given by the number of loops in the double-line graph DL( γ ) .

Original: Figure 4 shows the Feynman diagram and corresponding double-line diagram for E θ [ f ( x ) f
Revised: Figure 4 shows the Feynman diagram and corresponding double-line diagram for E θ [ f ( x ) f

Original: ( x (cid:48) )] with 2 hidden layers.
Revised: ( x (cid:48) )] with 2 hidden layers.

Original: We can interpret this Feynman diagram as a triangulation of the disc: a 2-dimensional surface with a single boundary.
Revised: We can interpret this Feynman diagram as a triangulation of the disc: a 2-dimensional surface with a single boundary.

Original: The triangulation has 2 vertices, 3 edges corresponding to the edges of the Feynman diagram, and 2 faces correponding to the loops of the double-line diagram.
Revised: The triangulation has 2 vertices, 3 edges corresponding to the edges of the Feynman diagram, and 2 faces correponding to the loops of the double-line diagram.

Original: Figureshows additional examples of double-line diagrams, and Figure 6 shows the double-line diagrams of a correlation function with derivatives.
Revised: Figureshows additional examples of double-line diagrams, and Figure 6 shows the double-line diagrams of a correlation function with derivatives.

Original: As explained in Section 4, contracted derivative tensors in a [Figure]
Revised: As explained in Section 4, contracted derivative tensors in a correlation function C map to forced edges in Γ( C ) , and these are marked with dashed lines on the diagrams.

Original: [Figure]
Revised: [Figure]

Original: correlation function C map to forced edges in Γ( C ) , and these are marked with dashed lines on the diagrams.

Original: Generally, every Feynman diagram of a deep linear network can be interpreted as the triangulation of a 2-dimensional manifold with at least one boundary.
Revised: Generally, every Feynman diagram of a deep linear network can be interpreted as the triangulation of a 2-dimensional manifold with at least one boundary.

Original: Intuitively, the presence of the boundary is due to the fact that the U, V weights at both ends of network have only one dimension that scales linearly with the width.
Revised: Intuitively, the presence of the boundary is due to the fact that the U, V weights at both ends of network have only one dimension that scales linearly with the width.

Original: As a result, in the double-line diagrams the U, V edges become single lines rather than double lines.
Revised: As a result, in the double-line diagrams the U, V edges become single lines rather than double lines.

Original: These ‘missing’ lines translate into fewer faces in the triangulation and the ‘missing’ faces can be geometrically interpreted as boundaries in the corresponding surface.
Revised: These ‘missing’ lines translate into fewer faces in the triangulation, and the ‘missing’ faces can be geometrically interpreted as boundaries in the corresponding surface.

Original: The discussion above is summarized by the following result, due to ’t Hooft (1974) that describes how the asymptotic behavior of a general correlation function can be computed using the Feynman rules for deep linear networks.
Revised: The discussion above is summarized by the following result, due to ’t Hooft (1974), that describes how the asymptotic behavior of a general correlation function can be computed using the Feynman rules for deep linear networks.

Original: Theorem 5.
Revised: Theorem 5.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function of a deep linear network with d hidden layers, and let γ ∈ Γ( C ) be a Feynman diagram.
Revised: , x m ) be a correlation function of a deep linear network with d hidden layers, and let γ ∈ Γ( C ) be a Feynman diagram.

Original: The diagram represents a subset of terms that contribute to C  and its asymptotic behavior is determined by the Feynman rules: the subset is O ( n s γ ) where s γ
Revised: The diagram represents a subset of terms that contribute to C , and its asymptotic behavior is determined by the Feynman rules: the subset is O ( n s γ ) where s γ

Original: =
Revised: =

Original: l γ − dm 2 , and l γ is the number of loops in the double-line diagram DL( γ ) .
Revised: l γ − dm 2 , and l γ is the number of loops in the double-line diagram DL( γ ) .

Original: Furthremore, the correlation function is C
Revised: Furthremore, the correlation function is C

Original: =
Revised: =

Original: O
Revised: O

Original: ( n s ) , where s = max γ ∈ Γ( C ) s γ .
Revised: ( n s ) , where s = max γ ∈ Γ( C ) s γ .

Original: The intuition for the formula s γ =
Revised: The intuition for the formula s γ =

Original: l γ − dm 2 is similar to the single hidden layer case, Theorem 3.
Revised: l γ − dm 2 is similar to the single hidden layer case, Theorem 3.

Original: The term l γ counts the number of factors of the form (cid:80)
Revised: The term l γ counts the number of factors of the form (cid:80)

Original: i 1 ,...,i k δ i 1 i 2 δ i 2 i 3 · · · δ i k i 1 = n that appear in the
Revised: i 1 ,...,i k δ i 1 i 2 δ i 2 i 3 · · · δ i k i 1 = n that appear in the

Original: Correlation function after applying Isserlis’ theorem.
Revised: Correlation function after applying Isserlis’ theorem.

Original: The term (cid:0) − dm 2 (cid:1) is due to the explicit n − d/normalization of the network function.
Revised: The term (cid:0) − dm 2 (cid:1) is due to the explicit n − d/normalization of the network function.

Original: B. A SYMPTOTICS OF DEEP LINEAR NETWORKS
Revised: B. A SYMPTOTICS OF DEEP LINEAR NETWORKS

Original: We now prove Theorem 1.
Revised: We now prove Theorem 1.

Original: The theorem follows from the following lemma, again due to ’t Hooft (1974), that relates the asymptotic behavior to the number of connected components in a Feynman diagram.
Revised: The theorem follows from the following lemma, again due to ’t Hooft (1974), that relates the asymptotic behavior to the number of connected components in a Feynman diagram.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function for a deep linear network.
Revised: , x m ) be a correlation function for a deep linear network.

Original: Let c γ be the number of connected components of a graph γ ∈ Γ( C ) .
Revised: Let c γ be the number of connected components of a graph γ ∈ Γ( C ) .

Original: Then C
Revised: Then C

Original: = O ( n s ) , where
Revised: = O ( n s ) , where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove the result for 1D inputs ( D in = 1 ), and it is easy to generalize to arbitrary input dimension.
Revised: We prove the result for 1D inputs ( D in = 1 ), and it is easy to generalize to arbitrary input dimension.

Original: Let γ ∈ Γ( C ) be a Feynman diagram and let γ (cid:48) be a connected component of γ with v γ (cid:48) vertices.
Revised: Let γ ∈ Γ( C ) be a Feynman diagram and let γ (cid:48) be a connected component of γ with v γ (cid:48) vertices.

Original: Notice that we can apply the Feynman rules of Theorem 5 separately to each component γ (cid:48) and ﬁnd a bound O ( n s γ (cid:48) ) .
Revised: Notice that we can apply the Feynman rules of Theorem 5 separately to each component γ (cid:48) and ﬁnd a bound O ( n s γ (cid:48) ) .

Original: Then, γ = O ( n s γ ) where s γ = (cid:80) γ (cid:48)
Revised: Then, γ = O ( n s γ ) where s γ = (cid:80) γ (cid:48)

Original: s γ (cid:48) , and the sum runs over the connected components of s γ .
Revised: s γ (cid:48) , and the sum runs over the connected components of s γ .

Original: We will show below that s γ (cid:48) ≤ 1 − v γ (cid:48) 2 , and therefore s γ ≤ c γ − m 2 , which is sufﬁcient to prove (22).
Revised: We will show below that s γ (cid:48) ≤ 1 − v γ (cid:48) 2 , and therefore s γ ≤ c γ − m 2 , which is sufﬁcient to prove (22).

Original: Let us prove the remaining statement about s γ (cid:48) .
Revised: Let us prove the remaining statement about s γ (cid:48) .

Original: The Euler character of the graph γ (cid:48) with v = v γ (cid:48) vertices, e edges and f faces is χ = v − e + f .
Revised: The Euler character of the graph γ (cid:48) with v = v γ (cid:48) vertices, e edges and f faces is χ = v − e + f .

Original: The degree of each vertex in the graph is d + 1 , and therefore e = ( d +1) v. Using Theorem 5 the graph is O ( n s γ (cid:48) ) where s γ (cid:48)
Revised: The degree of each vertex in the graph is d + 1 , and therefore e = ( d +1) v. Using Theorem 5 the graph is O ( n s γ (cid:48) ) where s γ (cid:48)

Original: =
Revised: =

Original: f − dv 2 .
Revised: f − dv 2 .

Original: We therefore ﬁnd that χ = v 2 + s γ (cid:48) .
Revised: We therefore ﬁnd that χ = v 2 + s γ (cid:48) .

Original: The graph γ (cid:48) is a triangulation of some connected surface with at least one boundary.
Revised: The graph γ (cid:48) is a triangulation of some connected surface with at least one boundary.

Original: The Euler character for such a surface is bounded by χ ≤ 1 , and therefore s γ (cid:48) ≤ 1 − v 2 .
Revised: The Euler character for such a surface is bounded by χ ≤ 1 , and therefore s γ (cid:48) ≤ 1 − v 2 .

Original: Let us now prove Theorem 1.
Revised: Let us now prove Theorem 1.

Original: Proof.
Revised: Proof.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) be a correlation function for a deep linear network.
Revised: , x m ) be a correlation function for a deep linear network.

Original: Suppose that the cluster graph G C has n e even size components and n o odd size components.
Revised: Suppose that the cluster graph G C has n e even size components and n o odd size components.

Original: Let γ ∈ Γ( C ) be a
Revised: Let γ ∈ Γ( C ) be a

Original: Feynman diagram with c γ connected components.
Revised: Feynman diagram with c γ connected components.

Original: We will show that c γ ≤ n e + n o 2 .
Revised: We will show that c γ ≤ n e + n o 2 .

Original: It then follows immediately from Lemma 2 that C
Revised: It then follows immediately from Lemma 2 that C

Original: =
Revised: =

Original: O ( n s ) where s = n e + n o 2 − m 2 , concluding the proof.
Revised: O ( n s ) where s = n e + n o 2 − m 2 , concluding the proof.

Original: Let us derive the bound on c γ .
Revised: Let us derive the bound on c γ .

Original: First, all vertices that belong to a given cluster (a component of G C ) will also belong to the same connected component in γ .
Revised: First, all vertices that belong to a given cluster (a component of G C ) will also belong to the same connected component in γ .

Original: This is because every edge in G C is also an edge in γ (note that G C and γ have the same set of vertices).
Revised: This is because every edge in G C is also an edge in γ (note that G C and γ have the same set of vertices).

Original: Therefore, c γ ≤ n e + n o .
Revised: Therefore, c γ ≤ n e + n o .

Original: Second, note that every connected component of the graph γ has an even number of vertices.
Revised: Second, note that every connected component of the graph γ has an even number of vertices.

Original: Indeed, each edge has a type t , and each vertex has exactly one edge of each type.
Revised: Indeed, each edge has a type t , and each vertex has exactly one edge of each type.

Original: Therefore, a connected component with v vertices has v 2 edges of each type, and so v must be even.
Revised: Therefore, a connected component with v vertices has v 2 edges of each type, and so v must be even.

Original: It follows that the vertices of even clusters can form their own connected components in a Feynman diagrams, while odd clusters must be connected in sets of 2 or more to form connected components.
Revised: It follows that the vertices of even clusters can form their own connected components in a Feynman diagrams, while odd clusters must be connected in sets of 2 or more to form connected components.

Original: The bound on c γ then follows.
Revised: The bound on c γ then follows.

Revised: For deep linear networks with bias the proof is the same, except we use Lemma 3 below instead of

Revised: Lemma 2.

Original: C N ON -L INEARITIES
Revised: B. D EEP LINEAR NETWORKS WITH BIAS

Revised: In this section we argue that Conjecture 1 holds for deep linear networks with bias. Let us add bias to the deﬁnition (2) of a deep linear network (setting σ to be the identity). We choose x = 1 without loss of generality.

Revised: The activations z l at layer l and the network function f are given by

Revised: [Equation]

Revised: Here b 1 , . . .

Revised: , b d − 1 ∈ R n are biases whose elements are chosen independently from N (0 , 1) at initialization.

Revised: The network function can be written as

Revised: [Equation]

Original: {+where we deﬁned b 
+}
Revised: {+where we deﬁned b 
+}

Revised: = U .

Revised: In the rest of this Section we extend our deﬁnitions of Feynman diagrams and double-line diagrams to deep linear networks with bias, and then extend several of our results to this case.

Revised: Deﬁnition 4.

Revised: Let C ( x 1 , . . .

Revised: , x m ) be a correlation function for a network with d hidden layers and bias.

Revised: The family Γ( C ) is the set of all graphs that have the following properties.

Revised: There are m vertices v 1 , . .

Revised: . , v m , where each v i has degree k i ∈ { 2 , . . .

Revised: , d + 1 } .

Revised: Each edge has a type t .

Revised: Every vertex of degree k has one edge of each type t ∈ { b d +1 − k , W d +2 − k , . .

Revised: , W d − 1 , V } .

Revised: If two derivative tensors T µ 1 ,...,µ (cid:96) ( x i ) , T ν 1 ,...,ν (cid:96) (cid:48) ( x j ) are contracted s times in C , the graph must have at least s edges (of any type) connecting the vertices v i , v j .

Revised: The intuition behind this deﬁnition is that term k in the sum in (26) is equal to the network function of a deep linear network with depth k − 1 .

Revised: Correlation functions of such a function can be represented diagramatically by including vertices of degree k .

Revised: To represent the entire sum we must consider all diagrams with vertices of varying degree.

Revised: We now extend Deﬁnition 3 of double-line graphs to accommodate vertices with varying degree.

Revised: Deﬁnition 5.

Revised: Let γ ∈ Γ( C ) be a Feynman diagram for a correlation function C involving m derivative tensors for a network of depth d with biases.

Revised: Its double-line graph , DL( γ ) is a graph with vertices of degree 2, deﬁned by the following blow-up procedure.

Revised: • Each vertex v ( i ) in γ of degree

Revised: k

Revised: i is mapped to k i − 1 vertices v ( i ) 1 , . .

Revised: , v ( i ) k

Revised: i − 1 in DL( γ ) .

Revised: • Each edge ( v ( i ) , v ( j ) ) in γ of type W l is mapped to two edges, ( v ( i ) l + k i − d − 1 , v ( j ) l + k j − d − 1 ) and ( v ( i ) l + k i − d , v ( j ) l + k j − d ) .

Revised: • Each edge ( v ( i ) , v ( j ) ) in γ of type b d +1 − k

Revised: i is mapped to a single edge ( v ( i ) 1 , v ( j ) 1 ) (notice that k i = k j ).

Revised: [Figure]

Revised: • Each edge ( v ( i ) , v ( j ) ) in γ of type V is mapped to a

Revised: [Figure]

Revised: As in Deﬁnition 3, the number of faces in γ is given by the number of loops in the double-line graph

Revised: DL( γ ) .

Revised: Figure 7 is an example of the new single line and double line diagrams that appear in the correlation function E θ [ f ( x 1 ) f

Revised: ( x 2 ) f ( x 3 ) f

Revised: ( x 4 )] for a two-hidden-layer network with biases.

Revised: Let us now extend Theorem 5 (the Feynman rules) to accomodate bias.

Revised: The only difference is in the explicit normalization.

Revised: Previously, each vertex had degree d + 1 and contributed a factor of n − d/due to the explicit normalization of the function map.

Revised: With biases, as a result of (26), a vertex with degree k contributes a factor of n ( k − 1) / 2 .

Revised: Theorem 6.

Revised: Let C ( x 1 , . . .

Revised: , x m ) be a correlation function of a deep linear network with d hidden layers and with biases.

Revised: Let γ ∈ Γ( C ) be a Feynman diagram.

Revised: The diagram represents a subset of terms that contribute to C , and its asymptotic behavior is determined by the Feynman rules: the subset is O ( n s γ ) where s γ

Revised: =

Revised: l γ − (cid:80) mi =1 k i − 1 2 , with k

Revised: i

Revised: the degree of the i -th vertex, and l γ the number of loops in the double-line diagram DL( γ ) .

Revised: Furthremore, the correlation function is C

Revised: =

Revised: O

Revised: ( n s ) , where s = max γ ∈ Γ( C ) s γ .

Revised: We now introduce the following result, which generalizes Lemma 2 to the case of networks with bias.

Revised: This Lemma is used above in the proof of Theorem 1.

Revised: Lemma 3.

Revised: Let C ( x 1 , . . .

Revised: , x m ) be a correlation function for a deep linear network with biases.

Revised: Let c γ be the number of connected components of a graph γ ∈ Γ( C ) .

Revised: Then C

Revised: = O ( n s ) , where

Revised: [Equation]

Revised: Proof.

Revised: It is enough to show that each connected component γ (cid:48) in γ is bounded as O ( n s γ (cid:48) ) where s γ (cid:48) ≤ 1 − v γ (cid:48) 2 .

Revised: The graph γ (cid:48) is a triangulation of a Riemann surface with f faces e edges, and v γ (cid:48) vertices of degrees k 1 , . . .

Revised: , k v γ (cid:48) .

Revised: The Feynman rules give

Revised: [Equation]

Revised: Using the relation e = (cid:80) v γ (cid:48)

Revised: i =

Revised: 1 k

Revised: i 2 and the deﬁnition of the Euler character, χ = v γ (cid:48) − e + f we have

Revised: [Equation]

Revised: The diagram γ (cid:48) is a triangulation of a Riemann surface with at least one boundary, thus χ ≤ 1 and s γ (cid:48) ≤ 1 − v γ (cid:48) 2 .

Revised: C N ON -L INEARITIES

Original: In previous sections we presented the Feynman diagram method for computing the large width asymptotics of correlation functions.
Revised: In previous sections we presented the Feynman diagram method for computing the large width asymptotics of correlation functions.

Original: In this section we show that the method applies as-is for deep networks with ReLU non-linearities and all-equal inputs, as well as to networks with a single hidden layer, a broader class of non-linearities, and arbitrary inputs.
Revised: In this section we show that the method applies as-is for deep networks with ReLU non-linearities and all-equal inputs, as well as to networks with a single hidden layer, a broader class of non-linearities, and arbitrary inputs.

Original: Theorem 2 follows immediately from the results presented in this section.
Revised: Theorem 2 follows immediately from the results presented in this section.

Original: C. D EEP NETWORKS WITH R E LU NON - LINEARITIES
Revised: C. D EEP NETWORKS WITH R E LU NON - LINEARITIES

Original: The following result guarantees that the presence of ReLU non-linearities does not change the asymptotic upper bound compared with linear activations, when all inputs are the same.
Revised: The following result guarantees that the presence of ReLU non-linearities does not change the asymptotic upper bound compared with linear activations, when all inputs are the same.

Original: Theorem 6.
Revised: Theorem 7.

Original: Let f nl be a network function of the form (2) with ReLU activation.
Revised: Let f nl be a network function of the form (2) with ReLU activation.

Original: Let f be a network with the same architecture but with linear activation.
Revised: Let f be a network with the same architecture but with linear activation.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ; f nl ) be a correlation function of the deep network f nl with width n , and let f be a deep linear network with the same width and depth.
Revised: , x m ; f nl ) be a correlation function of the deep network f nl with width n , and let f be a deep linear network with the same width and depth.

Original: Suppose that all inputs are the same, x 1 = x 2 = · · · = x m .
Revised: Suppose that all inputs are the same, x 1 = x 2 = · · · = x m .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Intuitively, we will rely on the fact that for ReLU networks we can, in some cases, treat the binary neuron activations as being statistically independent of the weights.
Revised: Intuitively, we will rely on the fact that for ReLU networks we can, in some cases, treat the binary neuron activations as being statistically independent of the weights.

Original: This result is due to (Hanin &
Revised: This result is due to (Hanin &

Original: Nica (2018)).
Revised: Nica (2018)).

Original: Given this result, we can bound the contribution of the binary activations, and the remaining Gaussian integral is equivalent to that found in a deep linear network.
Revised: Given this result, we can bound the contribution of the binary activations, and the remaining Gaussian integral is equivalent to that found in a deep linear network.

Original: The proof does not work for correlation functions with non-equal inputs, because in that case the independence result of
Revised: The proof does not work for correlation functions with non-equal inputs, because in that case the independence result of

Original: Hanin & Nica (2018) no longer holds.
Revised: Hanin & Nica (2018) no longer holds.

Original: Proof.
Revised: Proof.

Original: We can write the network function as
Revised: We can write the network function as

Original: [Equation]
Revised: [Equation]

Original: Here, H is the Heaviside step function acting elementwise on its vector argument.
Revised: Here, H is the Heaviside step function acting elementwise on its vector argument.

Original: We now introduce the construction from Hanin & Nica (2018).
Revised: We now introduce the construction from Hanin & Nica (2018).

Original: Let ξ j , η j , j = 1 , . . .
Revised: Let ξ j , η j , j = 1 , . . .

Original: , d be diagonal matrices of dimension n , whose diagonal elements are ± 1 -Bernoulli( p ) variables with p = 12 .
Revised: , d be diagonal matrices of dimension n , whose diagonal elements are ± 1 -Bernoulli( p ) variables with p = 12 .

Original: We deﬁne the new variables 
Revised: We deﬁne the new variables ˆ

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Let ˆ f nl be a network function with the same architecture as f nl but using the re-deﬁned weights.
Revised: Let ˆ f nl be a network function with the same architecture as f nl but using the re-deﬁned weights.

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: The following was shown in Hanin & Nica (2018).
Revised: The following was shown in Hanin & Nica (2018).

Original: f nl and ˆ f nl are equal in distribution.
Revised: f nl and ˆ f nl are equal in distribution.

Original: { ˆ D j ( x ) , ρ j , j = 1 , . .
Revised: { ˆ D j ( x ) , ρ j , j = 1 , . .

Original: , d } are independent of { U, V, W 1 , . .
Revised: , d } are independent of { U, V, W 1 , . .

Original: , W d − 1 } .
Revised: , W d − 1 } .

Original: { ˆ D j ( x ) , j = 1 , . . . , d } are independent of each other 
Revised: { ˆ D j ( x ) , j = 1 , . . . , d } are independent of each other for ﬁxed x .

Original: for

Original: ﬁxed

Original: x

Original: The diagonal entries of each diagonal matrix ˆ D j ( x ) are independent, and take the values { 0 , 1 } with 
Revised: The diagonal entries of each diagonal matrix ˆ D j ( x ) are independent, and take the values { 0 , 1 } with probability 12 .

Original: probability

Original: Now, the correlation function can be written as
Revised: Now, the correlation function can be written as

Original: [Equation]
Revised: [Equation]

Original: The second equality follows from the fact that f nl d = ˆ f nl .
Revised: The second equality follows from the fact that f nl d = ˆ f nl .

Original: Let us assume for now that the correlation function has no contracted derivatives.
Revised: Let us assume for now that the correlation function has no contracted derivatives.

Original: The we have
Revised: The we have

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:126)i = { i 1 , 1 , . .
Revised: Here, (cid:126)i = { i 1 , 1 , . .

Original: , i m,d } and (cid:126)α = { α 1 , . . .
Revised: , i m,d } and (cid:126)α = { α 1 , . . .

Original: , α m } , and
Revised: , α m } , and

Original: [Equation]
Revised: [Equation]

Original: In writing the equation (33) we used the facts that ˆ D, ρ are independent of the parameters, and that parameters in different layers are independent.
Revised: In writing the equation (40) we used the facts that ˆ D, ρ are independent of the parameters, and that parameters in different layers are independent.

Original: We now use Theorem 4 (Isserlis), which says that each of the expectation values over products of V , W j , and U elements is equal to a sum over permutations, where each term is a product over Kronecker delta functions — the covariance matrices of the parameters.
Revised: We now use Theorem 4 (Isserlis), which says that each of the expectation values over products of V , W j , and U elements is equal to a sum over permutations, where each term is a product over Kronecker delta functions — the covariance matrices of the parameters.

Original: [Equation]
Revised: [Equation]

Original: Here, ˜∆ (cid:126)σ(cid:126)i,(cid:126)α is a product of Kronecker delta functions.
Revised: Here, ˜∆ (cid:126)σ(cid:126)i,(cid:126)α is a product of Kronecker delta functions.

Original: The precise form of this object will not be important for our purpose, though it can be easily derived using Theorem 4.
Revised: The precise form of this object will not be important for our purpose, though it can be easily derived using Theorem 4.

Original: We can now bound the correlation function as follows.
Revised: We can now bound the correlation function as follows.

Original: [Equation]
Revised: [Equation]

Original:  (cid:126)α
Revised: (cid:88) (cid:126)α

Original: [Equation]
Revised: [Equation]

Original: (cid:126)σ(cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12)
Revised: (cid:126)σ(cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: (cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: 43) ≤

Revised: [Equation]

Original: (cid:126)i
Revised: (cid:126)i

Original: [Equation]
Revised: [Equation]

Original:  (cid:126)α(cid:12)(cid:12)(cid:12)
Revised: (cid:88) (cid:126)α

Original: [Equation]
Revised: [Equation]

Revised: (cid:126)σ(cid:126)i

Original: [Equation]
Revised: [Equation]

Original: (cid:126)i
Revised: (cid:126)α

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, E max = max (cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12) E (cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12) , and v ∈ R D in is the all-ones vector.
Revised: Here, E max = max (cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12) E (cid:126)i,(cid:126)α (cid:12)(cid:12)(cid:12) , and v ∈ R D in is the all-ones vector.

Original: The diagonal elements of ρ l , ˆ D l ( x ) are identical variables.
Revised: The diagonal elements of ρ l , ˆ D l ( x ) are identical variables.

Original: Therefore, E (cid:126)i,(cid:126)α only takes an O (1) number of values in the large width limit.
Revised: Therefore, E (cid:126)i,(cid:126)α only takes an O (1) number of values in the large width limit.

Original: Note that each of these values are independent of n , and therefore E max =
Revised: Note that each of these values are independent of n , and therefore E max =

Original: O (1) .
Revised: O (1) .

Original: We managed to bound the correlation function for f nl in terms of the correlation function for the corresponding linear network f with ﬁxed inputs.
Revised: We managed to bound the correlation function for f nl in terms of the correlation function for the corresponding linear network f with ﬁxed inputs.

Original: The asymptotics of the linear-network correlation function do not depend on the inputs, and therefore this concludes the proof.
Revised: The asymptotics of the linear-network correlation function do not depend on the inputs, and therefore this concludes the proof.

Original: C. S INGLE HIDDEN LAYER NETWORKS
Revised: C. S INGLE HIDDEN LAYER NETWORKS

Original: For networks with a single hidden layer, deﬁned by
Revised: For networks with a single hidden layer, deﬁned by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we can extend our asymptotic analysis to smooth non-linearities.
Revised: we can extend our asymptotic analysis to smooth non-linearities.

Original: We will show in Theorem 7 that for any correlation function C , we have 
Revised: We will show in Theorem 8 that for any correlation function C , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f is a deep linear network of equal width and sufﬁcient depth.
Revised: where f is a deep linear network of equal width and sufﬁcient depth.

Original: Therefore, computing the asymptotics using Feynman diagrams for deep linear networks yields a bound on networks with a single hidden layer and smooth non-linearities.
Revised: Therefore, computing the asymptotics using Feynman diagrams for deep linear networks yields a bound on networks with a single hidden layer and smooth non-linearities.

Original: Before delving into the proof of this claim, we consider a few simple examples.
Revised: Before delving into the proof of this claim, we consider a few simple examples.

Original: Let us begin with the correlation function E θ [ f nl ( x ) f nl ( x (cid:48) )] .
Revised: Let us begin with the correlation function E θ [ f nl ( x ) f nl ( x (cid:48) )] .

Original: [Equation]
Revised: [Equation]

Original: Here we used two facts.
Revised: Here we used two facts.

Original: First, the weights V i are unaffected by the non-linearity, so we can carry out the V integral.
Revised: First, the weights V i are unaffected by the non-linearity, so we can carry out the V integral.

Original: Second, the summand in the last equation, E (cid:2) σ ( U Ti x ) σ ( U Ti x (cid:48) )
Revised: Second, the summand in the last equation, E (cid:2) σ ( U Ti x ) σ ( U Ti x (cid:48) )

Original: (cid:3) , is independent of both i and n because U i are i.i.d.
Revised: (cid:3) , is independent of both i and n because U i are i.i.d.

Original: variables.
Revised: variables.

Original: Next, consider the following correlation function.
Revised: Next, consider the following correlation function.

Original: For simplicity, here we set the input dimension to be D in = 1 , and we set all inputs to 1; this does not change the asymptotics.
Revised: For simplicity, here we set the input dimension to be D in = 1 , and we set all inputs to 1; this does not change the asymptotics.

Original: [Equation]
Revised: [Equation]

Original: In the last line, we again used the fact that the summands are equal and independent of n .
Revised: In the last line, we again used the fact that the summands are equal and independent of n .

Original: In general, a correlation function C ( x 1 , . . .
Revised: In general, a correlation function C ( x 1 , . . .

Original: , x m ; f nl ) can be reduced to the form
Revised: , x m ; f nl ) can be reduced to the form

Original: [Equation]
Revised: [Equation]

Original: This form is obtained by carrying out the V i integrals, as well as all the sums that can be trivially carried out due to the presence of Kronecker deltas.
Revised: This form is obtained by carrying out the V i integrals, as well as all the sums that can be trivially carried out due to the presence of Kronecker deltas.

Original: Here, the α sum represents a sum over all K terms that appear from performing the V i integrals using Isserlis’ theorem; σ ( (cid:96) ) denotes the (cid:96) -th derivative of the non-linearity; k αs is the number of σ ( · ) ( U Ti s x ) factors sharing the i s index; ¯ σ α ∈ S m is a permutation; and F ( α ) is a function whose exact form is not important for us.
Revised: Here, the α sum represents a sum over all K terms that appear from performing the V i integrals using Isserlis’ theorem; σ ( (cid:96) ) denotes the (cid:96) -th derivative of the non-linearity; k αs is the number of σ ( · ) ( U Ti s x ) factors sharing the i s index; ¯ σ α ∈ S m is a permutation; and F ( α ) is a function whose exact form is not important for us.

Original: We will denote by r max the maximum number of index sums appearing in (44), namely r max := max α r α .
Revised: We will denote by r max the maximum number of index sums appearing in (51), namely r max := max α r α .

Original: Our approach to establishing the asymptotic scaling will be to ﬁrst bound the maximum number of index sums appearing in any term in our correlation function, written in the form (44), and then to argue that the summands are bounded by an n -independent constant.
Revised: Our approach to establishing the asymptotic scaling will be to ﬁrst bound the maximum number of index sums appearing in any term in our correlation function, written in the form (51), and then to argue that the summands are bounded by an n -independent constant.

Original: Let us introduce a family of diagrams, Γ (cid:48)
Revised: Let us introduce a family of diagrams, Γ (cid:48)

Original: ( C ) , which are different in general than the Feynman diagrams.
Revised: ( C ) , which are different in general than the Feynman diagrams.

Original: A given diagram g ∈ Γ (cid:48) ( C ) is constructed as follows.
Revised: A given diagram g ∈ Γ (cid:48) ( C ) is constructed as follows.

Original: Deﬁnition 4.
Revised: Deﬁnition 6.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ; f nl ) be a correlation function, where f nl is the output of a network with one hidden layer, deﬁned in (39) .
Revised: , x m ; f nl ) be a correlation function, where f nl is the output of a network with one hidden layer, deﬁned in (46) .

Original: The family Γ (cid:48) ( C ) is the set of all graphs that have the following properties.
Revised: The family Γ (cid:48) ( C ) is the set of all graphs that have the following properties.

Original: • Each derivative tensor in C is mapped to a vertex in the graph.
Revised: • Each derivative tensor in C is mapped to a vertex in the graph.

Original: • Each edge has a type that corresponds to one of the weight vectors U or V .
Revised: • Each edge has a type that corresponds to one of the weight vectors U or V .

Original: • Each vertex has exactly one edge of V type.
Revised: • Each vertex has exactly one edge of V type.

Original: • If two derivative tensors are contracted in C , the graph must have at least one edge (of any type) connecting the corresponding vertices for each contraction.
Revised: • If two derivative tensors are contracted in C , the graph must have at least one edge (of any type) connecting the corresponding vertices for each contraction.

Original: The reason for introducing this graphical structure is that it allows us to make two important statements.
Revised: The reason for introducing this graphical structure is that it allows us to make two important statements.

Original: If we deﬁne ˜ c g the number of connected components in a graph g ∈ Γ (cid:48) ( C ) and ˜ c max :=
Revised: If we deﬁne ˜ c g the number of connected components in a graph g ∈ Γ (cid:48) ( C ) and ˜ c max :=

Original: max g ∈ Γ (cid:48)
Revised: max g ∈ Γ (cid:48)

Original: ( C ) ˜
Revised: ( C ) ˜

Original: c g then the following holds.
Revised: c g then the following holds.

Original: The maximal number of sums appearing in a correlation function of the form (44) is ˜ c max .
Revised: The maximal number of sums appearing in a correlation function of the form (51) is ˜ c max .

Original: ˜ c max = c max , where cmax is the maximal number of connected components in the family of
Revised: ˜ c max = c max , where c

Original:  Feynman diagrams corresponding to a correlation function C ( x 1 , . . .
Revised: max is the maximal number of connected components in the family of Feynman diagrams corresponding to a correlation function C ( x 1 , . . .

Original: , x m  f d ) , where f d is a deep linear network with d hidden layers and d is large enough 
Revised: , x m ; f d ) , where f d is a deep linear network with d hidden layers, and d is large enough that none of the derivative tensors vanish.

Original: that none of the derivative tensors

Original: vanish.

Original: These two results, combined with a bound on the summands occurring in C ( x 1 , . . .
Revised: These two results, combined with a bound on the summands occurring in C ( x 1 , . . .

Original: , x m ; f nl ) will establish the bound (40).
Revised: , x m ; f nl ) will establish the bound (47).

Original: Lemma 3.
Revised: Lemma 4.

Original: A correlation function C has r max = ˜ c max .
Revised: A correlation function C has r max = ˜ c max .

Original: Proof.
Revised: Proof.

Original: Each vertex corresponds to a derivative tensor, T µ 1 ...µ k , which contains a single sum over paired U i and V i indices.
Revised: Each vertex corresponds to a derivative tensor, T µ 1 ...µ k , which contains a single sum over paired U i and V i indices.

Original: If two vertices are connected by one or more edges, there is a Kronecker delta factor that sets the corresponding indices to be equal (due either to a derivative contraction, or to a V covariance), reducing the number of sums by one.
Revised: If two vertices are connected by one or more edges, there is a Kronecker delta factor that sets the corresponding indices to be equal (due either to a derivative contraction, or to a V covariance), reducing the number of sums by one.

Original: The result is that any connected component of a graph g ∈ Γ (cid:48)
Revised: The result is that any connected component of a graph g ∈ Γ (cid:48)

Original: ( C ) corresponds to a single sum, and the total numer of index sums corresponding to a particular graph is ˜ c g , the number of connected components.
Revised: ( C ) corresponds to a single sum, and the total numer of index sums corresponding to a particular graph is ˜ c g , the number of connected components.

Original: As a result, the maximum number of sums corresponding to the collection of graphs Γ (cid:48) ( C ) is ˜ c max .
Revised: As a result, the maximum number of sums corresponding to the collection of graphs Γ (cid:48) ( C ) is ˜ c max .

Original: Lemma 4.
Revised: Lemma 5.

Original: The maximal number of connected components, ˜ c max , over the collection of graphs Γ (cid:48) ( C ) corresponding to C ( x 1 , . . .
Revised: The maximal number of connected components, ˜ c max , over the collection of graphs Γ (cid:48) ( C ) corresponding to C ( x 1 , . . .

Original: , x m ; f nl ) is bounded by the maximal number of components, c max , over the collection Γ( C ) corresponding to C ( x 1 , . . .
Revised: , x m ; f nl ) is bounded by the maximal number of components, c max , over the collection Γ( C ) corresponding to C ( x 1 , . . .

Original: , x m ; f d ) of Feynman diagrams, where f d is a deep linear network of sufﬁcient depth d .
Revised: , x m ; f d ) of Feynman diagrams, where f d is a deep linear network of sufﬁcient depth d .

Original: Proof.
Revised: Proof.

Original: Let n e ( n o ) be the number of even(odd) clusters in the cluster graph G C of C ( x 1 , . . .
Revised: Let n e ( n o ) be the number of even(odd) clusters in the cluster graph G C of C ( x 1 , . . .

Original: , x m ; f d ) .
Revised: , x m ; f d ) .

Original: The cluster graph, G C is a subgraph of any graph g ∈ Γ (cid:48)
Revised: The cluster graph, G C is a subgraph of any graph g ∈ Γ (cid:48)

Original: ( C ) .
Revised: ( C ) .

Original: We can thus think about the embedding of even and odd clusters into g .
Revised: We can thus think about the embedding of even and odd clusters into g .

Original: In any graph g ∈ Γ (cid:48) ( C ) , an even cluster may belong to its own connected component, while for odd clusters there must be an even number of them in any connected component.
Revised: In any graph g ∈ Γ (cid:48) ( C ) , an even cluster may belong to its own connected component, while for odd clusters there must be an even number of them in any connected component.

Original: This is because an even (odd) cluster contains an even (odd) number of factors of V , which must be paired up in any connected component.
Revised: This is because an even (odd) cluster contains an even (odd) number of factors of V , which must be paired up in any connected component.

Original: We ﬁnd that
Revised: We ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: The last inequality was used below Lemma 2 in the proof of Theorem 1.
Revised: The last inequality was used below Lemma 2 in the proof of Theorem 1.

Original: Theorem 7.
Revised: Theorem 8.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ; f nl ) be a correlation function for a one hidden layer network.
Revised: , x m ; f nl ) be a correlation function for a one hidden layer network.

Original: Let c max be the maximal number of connected components over the collection of graphs Γ( C ) corresponding to C ( x 1 , . . .
Revised: Let c max be the maximal number of connected components over the collection of graphs Γ( C ) corresponding to C ( x 1 , . . .

Original: , x m ; f d ) , where f d ( x ) is a deep linear network map, with with depth d greater than or equal to the maximum number of derivatives appearing in any single derivative tensor in C .
Revised: , x m ; f d ) , where f d ( x ) is a deep linear network map, with with depth d greater than or equal to the maximum number of derivatives appearing in any single derivative tensor in C .

Original: Then C
Revised: Then C

Original: = O ( n s ) where s = c max − m 2 .
Revised: = O ( n s ) where s = c max − m 2 .

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The correlation function in (44) can be bound as
Revised: The correlation function in (51) can be bound as

Original: [Equation]
Revised: [Equation]

Original: For ﬁxed inputs, S ( α ) i 1 ,...,
Revised: For ﬁxed inputs, S ( α ) i 1 ,...,

Original: i rα can take at most O (1) different values as a function of its indices, and the values are independent of n .
Revised: i rα can take at most O (1) different values as a function of its indices, and the values are independent of n .

Original: This is because the variables U i are identical.
Revised: This is because the variables U i are identical.

Original: We deﬁne s max as the maximum value of |S ( α ) i 1 ,...,i rα | as a function of α and the indices.
Revised: We deﬁne s max as the maximum value of |S ( α ) i 1 ,...,i rα | as a function of α and the indices.

Original: Combining this with the above lemmas we can then write.
Revised: Combining this with the above lemmas we can then write.

Original: [Equation]
Revised: [Equation]

Original: The result of the theorem follows from Lemma 2.
Revised: The result of the theorem follows from Lemma 2.

Original: D C ORRELATION FUNCTION ASYMPTOTICS
Revised: D C ORRELATION FUNCTION ASYMPTOTICS

Original: In this section we prove several general results about correlation function asymptotics in the large width limit.
Revised: In this section we prove several general results about correlation function asymptotics in the large width limit.

Original: Throughout this section, we assume that Conjecture 1 holds.
Revised: Throughout this section, we assume that Conjecture 1 holds.

Original: D. V ARIANCE ASYMPTOTICS
Revised: D. V ARIANCE ASYMPTOTICS

Original: Conjecture 1 can be used to bound the variance of the integrands that appear inside correlation functions.
Revised: Conjecture 1 can be used to bound the variance of the integrands that appear inside correlation functions.

Original: Lemma 5.
Revised: Lemma 6.

Original: Let ˜ C
Revised: Let ˜ C

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x m ) =
Revised: , x m ) =

Original: E θ [ F θ ( x 1 , . . .
Revised: E θ [ F θ ( x 1 , . . .

Original: , x m )] be a correlation function, and let G C be the cluster graph of C with n e even components and n o odd components.
Revised: , x m )] be a correlation function, and let G C be the cluster graph of C with n e even components and n o odd components.

Original: Assume Conjecture 1 holds such that C
Revised: Assume Conjecture 1 holds such that C

Original: = O ( n s C ) , where s C = n e + n o 2 − m 2 .
Revised: = O ( n s C ) , where s C = n e + n o 2 − m 2 .

Original: Then Var θ
Revised: Then Var θ

Original: [ F θ ( x 1 , . . .
Revised: [ F θ ( x 1 , . . .

Original: , x m )]
Revised: , x m )]

Original: = O ( n 2 s C ) .
Revised: = O ( n 2 s C ) .

Original: Proof.
Revised: Proof.

Original: To bound the variance, it is enough to bound the correlation function
Revised: To bound the variance, it is enough to bound the correlation function

Original: [Equation]
Revised: [Equation]

Original: because Var θ [ F θ ( x 1 , . . .
Revised: because Var θ [ F θ ( x 1 , . . .

Original: , x m )] ≤ ˜ C ( x 1 , . . .
Revised: , x m )] ≤ ˜ C ( x 1 , . . .

Original: , x m , x 1 , . . .
Revised: , x m , x 1 , . . .

Original: , x m ) .
Revised: , x m ) .

Original: The correlation function ˜ C has 2 n e even clusters and 2 n o odd clusters.
Revised: The correlation function ˜ C has 2 n e even clusters and 2 n o odd clusters.

Original: It follows from Conjecture 1 that ˜ C
Revised: It follows from Conjecture 1 that ˜ C

Original: = O ( n 2 s C ) .
Revised: = O ( n 2 s C ) .

Original: As a corrolary, notice that if C =
Revised: As a corrolary, notice that if C =

Original: O ( n s )
Revised: O ( n s )

Original: according to Conjecture 1, then typical realizations of the integrand will also be O ( n s ) .
Revised: according to Conjecture 1, then typical realizations of the integrand will also be O ( n s ) .

Original: In other words, the asymptotic bound of Conjecture 1 holds for typical initializations, not just in expectation.
Revised: In other words, the asymptotic bound of Conjecture 1 holds for typical initializations, not just in expectation.

Original: Table 3 shows empirical results for the variance of several functions.
Revised: Table 3 shows empirical results for the variance of several functions.

Original: In all cases we ﬁnd that the bound of Lemma 5 holds.
Revised: In all cases we ﬁnd that the bound of Lemma 6 holds.

Original: For (cid:80) µ Var θ [ ∂ µ f ( x 1 ) ∂ µ f ( x 2 )] we prove a tight bound below in
Revised: For (cid:80) µ Var θ [ ∂ µ f ( x 1 ) ∂ µ f ( x 2 )] we prove a tight bound below in

Original: Appendix E.1 for deep linear networks.
Revised: Appendix E.1 for deep linear networks.

Original: [Table]
Revised: [Table]

Original: n e , n [Table] 2 s C lin.
Revised: n e , n o 2 s C lin.

Original: ReLU tanh
Revised: ReLU tanh

Original: Var θ
Revised: Var θ

Original: [ f ( x 1 ) f
Revised: [ f ( x 1 ) f

Original: ( x 2 )] 0,  -0.
Revised: ( x 2 )] 0,  -0.

Original: [Table]
Revised: [Table]

Original: D. G RADIENT F LOW
Revised: D. G RADIENT F LOW

Original: The following results are used in the gradient ﬂow calculations of Section 3.
Revised: The following results are used in the gradient ﬂow calculations of Section 3.

Original: Lemma 6.
Revised: Lemma 7.

Original: Let G (cid:48) be a graph with m (cid:48) vertices, n (cid:48)
Revised: Let G (cid:48) be a graph with m (cid:48) vertices, n (cid:48)

Original: e even components, and n (cid:48)
Revised: e even components, and n (cid:48)

Original: o odd components.
Revised: o odd components.

Original: Let G be a subgraph of G (cid:48) with m vertices, n e even components, and n o odd components.
Revised: Let G be a subgraph of G (cid:48) with m vertices, n e even components, and n o odd components.

Original: Then s ( n e , n o , m ) ≥ s ( n (cid:48) e , n (cid:48)
Revised: Then s ( n e , n o , m ) ≥ s ( n (cid:48) e , n (cid:48)

Original: o , m (cid:48) ) where s ( a, b, c ) := a + b − c 2 .
Revised: o , m (cid:48) ) where s ( a, b, c ) := a + b − c 2 .

Original: Proof.
Revised: Proof.

Original: It is enough to show that s ( n e , n o , m ) does not increase if we (1) add a vertex to G , or (2) add an edge to G , because G (cid:48) can be obtained from G by performing such operations ﬁnitely many times.
Revised: It is enough to show that s ( n e , n o , m ) does not increase if we (1) add a vertex to G , or (2) add an edge to G , because G (cid:48) can be obtained from G by performing such operations ﬁnitely many times.

Original: Adding a vertex to G changes n e (cid:55)→ n e , n o (cid:55)→ n o + 1 , and m (cid:55)→ m + 1 , leaving s ( n e , n o , m ) unchanged.
Revised: Adding a vertex to G changes n e (cid:55)→ n e , n o (cid:55)→ n o + 1 , and m (cid:55)→ m + 1 , leaving s ( n e , n o , m ) unchanged.

Original: Next, if we add an edge to G then m does not change, and there are 4 possibilities for how n e and n o change.
Revised: Next, if we add an edge to G then m does not change, and there are 4 possibilities for how n e and n o change.

Original: The edge connects two even components.
Revised: The edge connects two even components.

Original: Then n e (cid:55)→ n e − 1 , n o (cid:55)→ n o , and s ( n e , n o , m ) decreases by 1. 2.
Revised: Then n e (cid:55)→ n e − 1 , n o (cid:55)→ n o , and s ( n e , n o , m ) decreases by 1. 2.

Original: The edge connects two odd components.
Revised: The edge connects two odd components.

Original: Then n e (cid:55)→ n e +1 , n o (cid:55)→ n o − 2 , and s ( n e , n o , m ) does not change.
Revised: Then n e (cid:55)→ n e +1 , n o (cid:55)→ n o − 2 , and s ( n e , n o , m ) does not change.

Original: The edge connects an even component and an odd component.
Revised: The edge connects an even component and an odd component.

Original: Then n e (cid:55)→ n e − 1 , n o (cid:55)→ n o , and s ( n e , n o , m ) decreases by 1. 4.
Revised: Then n e (cid:55)→ n e − 1 , n o (cid:55)→ n o , and s ( n e , n o , m ) decreases by 1. 4.

Original: The edge connects two vertices that belong to the same component.
Revised: The edge connects two vertices that belong to the same component.

Original: In this case n e , n o , and s ( n e , n o , m ) do not change.
Revised: In this case n e , n o , and s ( n e , n o , m ) do not change.

Original: We now prove Lemma 1 giving the scaling of time derivatives of correlation functions at initialization.
Revised: We now prove Lemma 1 giving the scaling of time derivatives of correlation functions at initialization.

Original: We prove the result for polynomial loss functions.
Revised: We prove the result for polynomial loss functions.

Original: Extension to more general loss functions requires interchanging the large width limit and the Taylor expansion of the loss, which we do not discuss.
Revised: Extension to more general loss functions requires interchanging the large width limit and the Taylor expansion of the loss, which we do not discuss.

Original: proof (Lemma 1).
Revised: proof (Lemma 1).

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: On the right-hand side we have a linear combination (cid:80)
Revised: On the right-hand side we have a linear combination (cid:80)

Original: A α A C A of correlation functions C A , where the coefﬁcients depend on the training set labels.
Revised: A α A C A of correlation functions C A , where the coefﬁcients depend on the training set labels.

Original: Here we used the polynomial loss assumption.
Revised: Here we used the polynomial loss assumption.

Original: For each correlation function C A , its integrand can be obtained from F by ﬁnitely many operations of the form (1) multiply the integrand by f ( x ) for some input x , and (2) act with a pair of contracted derivatives on two of the derivative tensors.
Revised: For each correlation function C A , its integrand can be obtained from F by ﬁnitely many operations of the form (1) multiply the integrand by f ( x ) for some input x , and (2) act with a pair of contracted derivatives on two of the derivative tensors.

Original: In the cluster graph representation, these two operations correspond to (1) adding a vertex, and (2) adding an edge.
Revised: In the cluster graph representation, these two operations correspond to (1) adding a vertex, and (2) adding an edge.

Original: Therefore, the cluster graph G C of
Revised: Therefore, the cluster graph G C of

Original: C =
Revised: C =

Original: E θ [ F ] is a subgraph of the cluster graph G C A of each one of the correlation functions C A .
Revised: E θ [ F ] is a subgraph of the cluster graph G C A of each one of the correlation functions C A .

Original: By
Revised: By

Original: Lemma 6 we have that C A
Revised: Lemma 7 we have that C A

Original: = O ( n s C ) for all A , and therefore the bound applies to E θ (cid:2)
Revised: = O ( n s C ) for all A , and therefore the bound applies to E θ (cid:2)

Original: d k F/dt k (cid:3) as well.
Revised: d k F/dt k (cid:3) as well.

Original: D. S TOCHASTIC G RADIENT DESCENT
Revised: D. S TOCHASTIC G RADIENT DESCENT

Original: In this section we show that the asymptotics of a correlation function do not change under stochastic gradient descent (SGD) updates.
Revised: In this section we show that the asymptotics of a correlation function do not change under stochastic gradient descent (SGD) updates.

Original: Let C ( x 1 , . . .
Revised: Let C ( x 1 , . . .

Original: , x m ) =
Revised: , x m ) =

Original: E θ ∼P 

Revised: E θ ∼P

Original: [ F θ ( x 1 , . . .
Revised: [ F θ ( x 1 , . . .

Original: , x m )] be a correlation function, where F θ is the integrand which explicitly depends on the parameters θ .
Revised: , x m )] be a correlation function, where F θ is the integrand which explicitly depends on the parameters θ .

Original: Under a single
Revised: Under a single

Original: SGD step, the parameters are updated as θ t +

Revised: SGD step, the parameters are updated as θ t +

Original: =
Revised: =

Original: θ t − η ∇ L t ( θ t ) , where L is the mini-batch loss at time t .
Revised: θ t − η ∇ L t ( θ t ) , where L is the mini-batch loss at time t .

Original: Let P t denote the distribution of parameters at step t , where P 0 is the initial distribution.
Revised: Let P t denote the distribution of parameters at step t , where P 0 is the initial distribution.

Original: We deﬁne the evolved correlation function at step t to be
Revised: We deﬁne the evolved correlation function at step t to be

Original: [Equation]
Revised: [Equation]

Original: We have the following
Revised: We have the following

Original: Theorem 8.
Revised: Theorem 9.

Original: Let C be a correlation function for a network with linear activations, and assume that
Revised: Let C be a correlation function for a network with linear activations, and assume that

Original: Conjecture 1 holds, namely C =
Revised: Conjecture 1 holds, namely C =

Original: O ( n s C ) where s C is deﬁned in the Conjecture.
Revised: O ( n s C ) where s C is deﬁned in the Conjecture.

Original: If C t is the evolved correlation function after t SGD steps, then C t =
Revised: If C t is the evolved correlation function after t SGD steps, then C t =

Original: O ( n s C ) .
Revised: O ( n s C ) .

Original: Proof.
Revised: Proof.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: The integrand F θ can be written as a product of derivative tensors of the form T µ 1 ...µ k ( x ; θ ) , with contracted derivatives.
Revised: The integrand F θ can be written as a product of derivative tensors of the form T µ 1 ...µ k ( x ; θ ) , with contracted derivatives.

Original: Suppose that the network has d hidden layers.
Revised: Suppose that the network has d hidden layers.

Original: Then under an SGD step, we have
Revised: Then under an SGD step, we have

Original: [Equation]
Revised: [Equation]

Original: ∂L
Revised: ∂L

Original: [Equation]
Revised: [Equation]

Original: Here D B is the mini-batch, and x (cid:48) a are mini-batch samples.
Revised: Here D B is the mini-batch, and x (cid:48) a are mini-batch samples.

Original: The k sum is truncated because higherorder derivatives of f vanish.
Revised: The k sum is truncated because higherorder derivatives of f vanish.

Original: We can now see how taking a gradient descent step affects the cluster graph.
Revised: We can now see how taking a gradient descent step affects the cluster graph.

Original: After taking a step, each derivative tensor in C t is replaced by a sum (over k, x (cid:48) 1 , . . .
Revised: After taking a step, each derivative tensor in C t is replaced by a sum (over k, x (cid:48) 1 , . . .

Original: , x (cid:48) k ).
Revised: , x (cid:48) k ).

Original: Each term in the combination of these sums is a correlation function, whose cluster graph is a subgraph of C .
Revised: Each term in the combination of these sums is a correlation function, whose cluster graph is a subgraph of C .

Original: Therefore, by Lemma 6,
Revised: Therefore, by Lemma 7,

Original: C t =
Revised: C t =

Original: O ( n s C ) .
Revised: O ( n s C ) .

Original: We note that for general activation functions the sum in (55) may be inﬁnite.
Revised: We note that for general activation functions the sum in (62) may be inﬁnite.

Original: In this case, to complete the proof we would need to show that the inﬁnite sum obeys the same bound as each individual term in the sum.
Revised: In this case, to complete the proof we would need to show that the inﬁnite sum obeys the same bound as each individual term in the sum.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: E A PPLICATIONS
Revised: E A PPLICATIONS

Original: Here we present several applications of our Feynman diagram method for computing large width asymptotics.
Revised: Here we present several applications of our Feynman diagram method for computing large width asymptotics.

Original: We assume throughout this section that Conjecture 1 holds.
Revised: We assume throughout this section that Conjecture 1 holds.

Original: E. N EURAL T ANGENT K ERNEL
Revised: E. N EURAL T ANGENT K ERNEL

Original: In this section we prove two results regarding the NTK at large width.
Revised: In this section we prove two results regarding the NTK at large width.

Original: We show that the kernel converges in probability, and that during gradient descent it is constant up to O ( n − 1 ) corrections.
Revised: We show that the kernel converges in probability, and that during gradient descent it is constant up to O ( n − 1 ) corrections.

Original: Theorem 9.
Revised: Theorem 10.

Original: The Neural Tangent Kernel Θ of a deep linear network converges in probability in the large width limit, and its variance is O ( n − 1 ) .
Revised: The Neural Tangent Kernel Θ of a deep linear network converges in probability in the large width limit, and its variance is O ( n − 1 ) .

Original: Conjecture 1 is not sufﬁcient for proving this theorem, as we need to use a more detailed Feynman diagram argument.
Revised: Conjecture 1 is not sufﬁcient for proving this theorem, as we need to use a more detailed Feynman diagram argument.

Original: Therefore, we only prove the theorem for the case of deep linear networks.
Revised: Therefore, we only prove the theorem for the case of deep linear networks.

Original: Proof.
Revised: Proof.

Original: First, we will show that Var[Θ]
Revised: First, we will show that Var[Θ]

Original: = O ( n − 1 ) .
Revised: = O ( n − 1 ) .

Original: Given a model function f , the variance is given by
Revised: Given a model function f , the variance is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here f (1) = · · · = f (4) =
Revised: Here f (1) = · · · = f (4) =

Original: f ; we introduced the superscripts so we can easily refer to individual factors.
Revised: f ; we introduced the superscripts so we can easily refer to individual factors.

Original: The crux of the argument is that the set of Feynman diagrams representing the expression (56) includes only connected graphs.
Revised: The crux of the argument is that the set of Feynman diagrams representing the expression (63) includes only connected graphs.

Original: Assuming this is the case, according to the bound (22) these diagrams will all scale as O ( n (2 − 4) / 2 ) =
Revised: Assuming this is the case, according to the bound (22) these diagrams will all scale as O ( n (2 − 4) / 2 ) =

Original: O ( n − 1 ) and so will the variance, completing the proof.
Revised: O ( n − 1 ) and so will the variance, completing the proof.

Original: To see why only connected graphs contribute, notice that
Revised: To see why only connected graphs contribute, notice that

Original: • A ( x, x (cid:48) ) includes all diagrams in which the vertices corresponding to f (1) , f (2) share an edge, and also the vertices f (3) , f (4) share an edge (due to the explicit derivatives); • B ( x, x (cid:48) ) includes all diagrams in which all edges are either between f (1) , f (2) or between f (3) , f (4) .
Revised: • A ( x, x (cid:48) ) includes all diagrams in which the vertices corresponding to f (1) , f (2) share an edge, and also the vertices f (3) , f (4) share an edge (due to the explicit derivatives); • B ( x, x (cid:48) ) includes all diagrams in which all edges are either between f (1) , f (2) or between f (3) , f (4) .

Original: Therefore, in the full expression (56), the only remaining diagrams ( i.e. the diagrams that do not cancel between the two terms) are those that include
Revised: Therefore, in the full expression (63), the only remaining diagrams ( i.e. the diagrams that do not cancel between the two terms) are those that include

Original: • an edge connecting f (1) , f (2) , • an edge connecting f (3) , f (4) , and • an edge connecting one of f (1) , f (2) to one of f (3) , f (4) .
Revised: • an edge connecting f (1) , f (2) , • an edge connecting f (3) , f (4) , and • an edge connecting one of f (1) , f (2) to one of f (3) , f (4) .

Original: These diagrams are all connected graphs, and this completes the proof that Var[Θ]
Revised: These diagrams are all connected graphs, and this completes the proof that Var[Θ]

Original: = O ( n − 1 ) .
Revised: = O ( n − 1 ) .

Original: Here and above we have established that Var[Θ]
Revised: Here and above we have established that Var[Θ]

Original: = O ( n − 1 ) and E
Revised: = O ( n − 1 ) and E

Original: [Θ] = O ( n 0 ) .
Revised: [Θ] = O ( n 0 ) .

Original: More generally, let us consider a random variable O equal to the product of f and its derivatives, where the derivatives indices are fully summed in pairs.
Revised: More generally, let us consider a random variable O equal to the product of f and its derivatives, where the derivatives indices are fully summed in pairs.

Original: As we will now show, if E [ O ] =
Revised: As we will now show, if E [ O ] =

Original: O ( n 0 ) and Var
Revised: O ( n 0 ) and Var

Original: [ O ] =
Revised: [ O ] =

Original: O ( n − 1 ) then (1) the limit lim n →∞ E
Revised: O ( n − 1 ) then (1) the limit lim n →∞ E

Original: [ O ] exists, and (2) O converges in probability to this limit.
Revised: [ O ] exists, and (2) O converges in probability to this limit.

Original: In particular, the NTK is an example of such a random variable, and so this will conclude the proof that the NTK converges in probability.
Revised: In particular, the NTK is an example of such a random variable, and so this will conclude the proof that the NTK converges in probability.

Original: First, consider the mean.
Revised: First, consider the mean.

Original: There is a ﬁnite number of diagrams contributing to E [ O ] , and each has a well-deﬁned n scaling.
Revised: There is a ﬁnite number of diagrams contributing to E [ O ] , and each has a well-deﬁned n scaling.

Original: Therefore, we can write
Revised: Therefore, we can write

Original: [Equation]
Revised: [Equation]

Original: for some values of k max ≥ 0 (integer) and c 0 , c 1 , . . .
Revised: for some values of k max ≥ 0 (integer) and c 0 , c 1 , . . .

Original: ∈ R .
Revised: ∈ R .

Original: We see that the mean has a well-deﬁned large n limit,
Revised: We see that the mean has a well-deﬁned large n limit,

Original: [Equation]
Revised: [Equation]

Original: Next, let us show that O − E
Revised: Next, let us show that O − E

Original: [ O ] converges in probability using Chebyshev’s inequality.
Revised: [ O ] converges in probability using Chebyshev’s inequality.

Original: Indeed, by the variance assumption there exists ˜ c > 0 such that
Revised: Indeed, by the variance assumption there exists ˜ c > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Combining the facts that O − E
Revised: Combining the facts that O − E

Original: [ O ] → p 0 and E
Revised: [ O ] → p 0 and E

Original: [ O ] → c 0 , we ﬁnd that O converges in probability to c 0 .
Revised: [ O ] → c 0 , we ﬁnd that O converges in probability to c 0 .

Original: Next, we show that the large width NTK is constant during training, and compute the asymptotics of the higher-order terms.
Revised: Next, we show that the large width NTK is constant during training, and compute the asymptotics of the higher-order terms.

Original: The following argument is phrased for deep linear networks.
Revised: The following argument is phrased for deep linear networks.

Original: More generally, the same argument holds for deep networks with smooth non-linear activations under the additional assumption that the large width limit and Taylor series can be exchanged (note that for such networks, the network function is analytic in the weights).
Revised: More generally, the same argument holds for deep networks with smooth non-linear activations under the additional assumption that the large width limit and Taylor series can be exchanged (note that for such networks, the network function is analytic in the weights).

Original: Theorem 10.
Revised: Theorem 11.

Original: Let f ( x ) be the network output of a deep linear network with MSE loss L .
Revised: Let f ( x ) be the network output of a deep linear network with MSE loss L .

Original: Let Θ t ( x, x (cid:48) ) be the Neural Tangent Kernel at SGD step t , for some inputs x, x (cid:48) .
Revised: Let Θ t ( x, x (cid:48) ) be the Neural Tangent Kernel at SGD step t , for some inputs x, x (cid:48) .

Original: Then in the large width limit, the kernel is constant in t in expectation, and
Revised: Then in the large width limit, the kernel is constant in t in expectation, and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It is enough to show that E θ ∼P 

Revised: It is enough to show that E θ ∼P

Original: [Θ 1 ( x, x
Revised: [Θ 1 ( x, x

Original: [Equation]
Revised: [Equation]

Original: ) − Θ 0 ( x, x (cid:48) )]
Revised: ) − Θ 0 ( x, x (cid:48) )]

Original: = O ( n − 1 ) .
Revised: = O ( n − 1 ) .

Original: It then follows from Theorem 8 that E
Revised: It then follows from Theorem 9 that E

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: concluding the proof.
Revised: concluding the proof.

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The fact that the k, l sums are truncated at d + 1 follows from using linear activations, as higher-order derivatives of the network function vanish in this case.
Revised: The fact that the k, l sums are truncated at d + 1 follows from using linear activations, as higher-order derivatives of the network function vanish in this case.

Original: All terms in the sum over k, l include a tensor product of the form (cid:80) µ,(cid:126)µ,(cid:126)ν T µµ 1 ...µ k ( x ) T µν 1 ...
Revised: All terms in the sum over k, l include a tensor product of the form (cid:80) µ,(cid:126)µ,(cid:126)ν T µµ 1 ...µ k ( x ) T µν 1 ...

Original: ν l ( x (cid:48) )( · · · ) with either k ≥ 1 or l ≥ 1 , where ( · · · ) stands for additional derivative tensor factors.
Revised: ν l ( x (cid:48) )( · · · ) with either k ≥ 1 or l ≥ 1 , where ( · · · ) stands for additional derivative tensor factors.

Original: Therefore, all terms in the k, l sum are correlation functions that have a cluster of size at least 3, including T ( x ) , T ( x (cid:48) ) , and at least one other tensor contracted through the µ 1 or ν 1 index.
Revised: Therefore, all terms in the k, l sum are correlation functions that have a cluster of size at least 3, including T ( x ) , T ( x (cid:48) ) , and at least one other tensor contracted through the µ 1 or ν 1 index.

Original: It follows from the Conjecture that each term in the sum is
Revised: It follows from the Conjecture that each term in the sum is

Original: O ( n − 1 ) .
Revised: O ( n − 1 ) .

Original: Lemma 5 then implies that the variance of these updates is O ( n − 2 ) .
Revised: Lemma 6 then implies that the variance of these updates is O ( n − 2 ) .

Original: E. W IDE NETWORK EVOLUTION IS LINEAR IN THE LEARNING RATE
Revised: E. W IDE NETWORK EVOLUTION IS LINEAR IN THE LEARNING RATE

Original: In this section we prove that, at large width, the NTK determines the evolution of the network function not just for continuous-time gradient descent but also for discrete-time gradient descent.
Revised: In this section we prove that, at large width, the NTK determines the evolution of the network function not just for continuous-time gradient descent but also for discrete-time gradient descent.

Original: A similar result holds for stochastic gradient descent, using a stochastic kernel.
Revised: A similar result holds for stochastic gradient descent, using a stochastic kernel.

Original: Again we prove the deep linear case explicitly, but the result holds for deep networks with smooth non-linear activations under the additional assumption that the large width limit and Taylor series can be exchanged.
Revised: Again we prove the deep linear case explicitly, but the result holds for deep networks with smooth non-linear activations under the additional assumption that the large width limit and Taylor series can be exchanged.

Original: Theorem 11.
Revised: Theorem 12.

Original: Let f ( x ) be the network output of a deep linear network, and let f t ( x ) be the evolved function after t gradient descent steps, deﬁned by θ t +

Revised: Let f ( x ) be the network output of a deep linear network, and let f t ( x ) be the evolved function after t gradient descent steps, deﬁned by θ t +

Original: =
Revised: =

Original: θ t − η ∇ L ( θ t ) .
Revised: θ t − η ∇ L ( θ t ) .

Original: In the large width limit, each gradient descent step update of f t is linear in the learning rate η .
Revised: In the large width limit, each gradient descent step update of f t is linear in the learning rate η .

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: Here, Θ 0 is the NTK at initialization and (cid:96) t is the single-sample loss at time t .
Revised: Here, Θ 0 is the NTK at initialization and (cid:96) t is the single-sample loss at time t .

Original: Proof.
Revised: Proof.

Original: For a deep linear network, under a single gradient descent step we have
Revised: For a deep linear network, under a single gradient descent step we have

Original: [Equation]
Revised: [Equation]

Original: D tr
Revised: D tr

Original: E θ (cid:20) Θ t (  x [Equation]
Revised: E θ (cid:20) Θ t ( x, x (cid:48) )

Original: x (cid:48) ) ∂(cid:96) t ( x (cid:48) , y (cid:48) )
Revised: ∂(cid:96) t ( x (cid:48) , y (cid:48) )

Original: ∂f (cid:21) + d +1 (cid:88) k
Revised: ∂f (cid:21) + d +1 (cid:88) k

Original: = ( − η ) k k ! (cid:88) µ 1 ,... [Equation] µ k
Revised: = ( − η ) k k ! (cid:88) µ 1 ,...,µ k

Original: E θ (cid:20) ∂
Revised: E θ (cid:20) ∂

Original: k f t ( x ) ∂θ µ 1 · · · ∂θ µ k ∂L t ∂θ µ 1 · · · ∂L t ∂θ µ k (cid:21) .
Revised: k f t ( x ) ∂θ µ 1 · · · ∂θ µ k ∂L t ∂θ µ 1 · · · ∂L t ∂θ µ k (cid:21) .

Original: First, consider the sum over k in (66).
Revised: First, consider the sum over k in (73).

Original: Each term in the sum is a correlation function for which the cluster graph contains a connected subgraph of size at least 3, and is therefore O ( n − 1 ) by Lemmaand Theorem 1.
Revised: Each term in the sum is a correlation function for which the cluster graph contains a connected subgraph of size at least 3, and is therefore O ( n − 1 ) by Lemmaand Theorem 1.

Original: In the remaining O ( η ) term, by the same argument as Theorem 10 we can replace Θ t =
Revised: In the remaining O ( η ) term, by the same argument as Theorem 11 we can replace Θ t =

Original: Θ 0 + O ( n − 1 ) in the correlation function.
Revised: Θ 0 + O ( n − 1 ) in the correlation function.

Original: The result is equation (66).
Revised: The result is equation (73).

Original: As mentioned above, we note that the proof goes through when using stochastic gradient descent updates, with the difference that in (66) we should sum over mini-batch samples instead of over the entire training set.
Revised: As mentioned above, we note that the proof goes through when using stochastic gradient descent updates, with the difference that in (73) we should sum over mini-batch samples instead of over the entire training set.

Original: E. S PECTRAL PROPERTIES OF THE NTK AND THE H ESSIAN
Revised: E. S PECTRAL PROPERTIES OF THE NTK AND THE H ESSIAN

Original: With an eye towards understanding the structure of the loss landscape at large width and as another example use case of our approach, we investigate the relation between the spectra of the Hessian, and the NTK.
Revised: With an eye towards understanding the structure of the loss landscape at large width and as another example use case of our approach, we investigate the relation between the spectra of the Hessian, and the NTK.

Original: Among other observations, we present an argument that for a network with d hidden layers and MSE loss, the top D in ordered eigenvalues of the Hessian, λ ( H ) i , are related to those of the kernel λ (Θ) i by
Revised: Among other observations, we present an argument that for a network with d hidden layers and MSE loss, the top D in ordered eigenvalues of the Hessian, λ ( H ) i , are related to those of the kernel λ (Θ) i by

Original: [Equation]
Revised: [Equation]

Original: The remaining Hessian eigenvalues vanish at large width as O ( n − 1 / 2 ) for one hidden layer networks and O ( n − 1 ) for deeper networks.
Revised: The remaining Hessian eigenvalues vanish at large width as O ( n − 1 / 2 ) for one hidden layer networks and O ( n − 1 ) for deeper networks.

Original:  This is experimentally corroborated in Figure 8.
Revised: 12 This is experimentally corroborated in Figure 9.

Original: [Figure]
Revised: [Figure]

Original: The Hessian of a general loss takes the form,
Revised: The Hessian of a general loss takes the form,

Original: [Equation]
Revised: [Equation]

Original: The moments of H are a useful way to understand its spectrum.
Revised: The moments of H are a useful way to understand its spectrum.

Original: As in other examples that we have seen, traces of powers of B lead to connected diagrams with higher and higher numbers of vertices, and so scale as increasingly negative powers of n .
Revised: As in other examples that we have seen, traces of powers of B lead to connected diagrams with higher and higher numbers of vertices, and so scale as increasingly negative powers of n .

Original: More explicitly, traces involving powers of B contain multiple contracted derivative tensors.
Revised: More explicitly, traces involving powers of B contain multiple contracted derivative tensors.

Original: As a result, taking the average of these traces over the weight distribution leads to correlation functions where the associated cluster graph, G C , contain subgraphs with at least three connected vertices.
Revised: As a result, taking the average of these traces over the weight distribution leads to correlation functions where the associated cluster graph, G C , contain subgraphs with at least three connected vertices.

Original: By Lemma 6 and Conjecture 1 these correlation functions vanish at inﬁnite width.
Revised: By Lemma 7 and Conjecture 1 these correlation functions vanish at inﬁnite width.

Original: For example E θ [Tr ( AB )]
Revised: For example E θ [Tr ( AB )]

Original: = O (1 /n ) .
Revised: = O (1 /n ) .

Original: There is a single exception to this, as can be easily seen in ﬁgure 7, E θ (cid:2)
Revised: There is a single exception to this, as can be easily seen in ﬁgure 8, E θ (cid:2)

Original: Tr( B 2 ) (cid:3) is O (1) .
Revised: Tr( B 2 ) (cid:3) is O (1) .

Original: Thus most moments of the Hessian are equal to moments of A in expectation.
Revised: Thus most moments of the Hessian are equal to moments of A in expectation.

Original: [Equation]
Revised: [Equation]

Original: What’s more, we can relate moments of A to those of the kernel, as both are built out of two logit derivatives.
Revised: What’s more, we can relate moments of A to those of the kernel, as both are built out of two logit derivatives.

Original: Explicitly,
Revised: Explicitly,

Original: [Equation]
Revised: [Equation]

Original: and so the moments of the Hessian are also related to those of the kernel.
Revised: and so the moments of the Hessian are also related to those of the kernel.

Original: [Equation]
Revised: [Equation]

Original: Here, we have deﬁned,
Revised: Here, we have deﬁned,

Original: [Equation]
Revised: [Equation]

Original: For the case of MSE loss, we can go even further.
Revised: For the case of MSE loss, we can go even further.

Original: In that case, M ( x a , x b ) = δ ab and B decays to zero during training.
Revised: In that case, M ( x a , x b ) = δ ab and B decays to zero during training.

Original: We thus have,
Revised: We thus have,

Original: Initially:
Revised: Initially:

Original: [Equation]
Revised: [Equation]

Original: At late times:
Revised: At late times:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: These results indicate that the only difference between the spectra of the Hessian and the NTK come from B and that B must have eigenvalues which scale as 1 / (cid:112) dim ( B ) .
Revised: These results indicate that the only difference between the spectra of the Hessian and the NTK come from B and that B must have eigenvalues which scale as 1 / (cid:112) dim ( B ) .

Original: As dim ( B ) =
Revised: As dim ( B ) =

Original: O ( n ) for one hidden layer networks and O ( n 2 ) for deep networks we are left with the relation (69) between the eigenvalues of Θ and H .
Revised: O ( n ) for one hidden layer networks and O ( n 2 ) for deep networks we are left with the relation (76) between the eigenvalues of Θ and H .

Original: As the network trains, the difference between these eigenvalues gets even smaller.
Revised: As the network trains, the difference between these eigenvalues gets even smaller.

Original: These results are conﬁrmed experimentally in Figure 8 and Figure 9.
Revised: These results are conﬁrmed experimentally in Figure 9 and Figure 10.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: eigenvalue of the B matrix shows good agreement with the predicted 1 / √ n fall off for single hidden layer ReLU networks.
Revised: eigenvalue of the B matrix shows good agreement with the predicted 1 / √ n fall off for single hidden layer ReLU networks.

Original: Experiments were performed on ReLU networks trained on two-class MNIST with 100 images per class.
Revised: Experiments were performed on ReLU networks trained on two-class MNIST with 100 images per class.

Original: Moments of the NTK were computed exactly, while moments of the Hessian and B were computed by randomly sampling 1000 vectors and using Hutchinson’s trick.
Revised: Moments of the NTK were computed exactly, while moments of the Hessian and B were computed by randomly sampling 1000 vectors and using Hutchinson’s trick.

Original: Multi-class.
Revised: Multi-class.

Original: This story can be generalized to multi-class neural network maps.
Revised: This story can be generalized to multi-class neural network maps.

Original: In this case the logit is a map
Revised: In this case the logit is a map

Original: f A : R D in → R N class , and the NTK has two class indices.
Revised: f A : R D in → R N class , and the NTK has two class indices.

Original: [Equation]
Revised: [Equation]

Original: Expression (73) relating the moments of the Hessian to the NTK still holds in this context provided we take Tr (( M Θ) m )
Revised: Expression (80) relating the moments of the Hessian to the NTK still holds in this context provided we take Tr (( M Θ) m )

Original: →
Revised: →

Original: Tr (cid:0) ( M AB Θ AB ) m (cid:1) , with the more general matrix,
Revised: Tr (cid:0) ( M AB Θ AB ) m (cid:1) , with the more general matrix,

Original: [Equation]
Revised: [Equation]

Original: At large width, the NTK approaches the identity matrix in class space, Θ AB → N − 1 class
Revised: At large width, the NTK approaches the identity matrix in class space, Θ AB → N − 1 class

Original: δ AB Tr (Θ)
Revised: δ AB Tr (Θ)

Original: (Jacot et al.
Revised: (Jacot et al.

Original: This implies that the NTK spectrum consists of N class repeated copies.
Revised: This implies that the NTK spectrum consists of N class repeated copies.

Original: This has consequences for the spectrum of the Hessian at large width.
Revised: This has consequences for the spectrum of the Hessian at large width.

Original: For instance, for the case of MSE error it also implies N class repeated copies of the Hessian spectrum (See Figure 8b).
Revised: For instance, for the case of MSE error it also implies N class repeated copies of the Hessian spectrum (See Figure 9b).

Original: It is intriguing to think this could serve as a path towards understanding the emergence of the N class (or N class − 1 ) large eigenvalues and corresponding subspace observed in the Hessian spectrum (Sagun et al.
Revised: It is intriguing to think this could serve as a path towards understanding the emergence of the N class (or N class − 1 ) large eigenvalues and corresponding subspace observed in the Hessian spectrum (Sagun et al.

Original: Gur-Ari et al.
Revised: Gur-Ari et al.

Original: ; Ghorbani et al.
Revised: ; Ghorbani et al.

Original: ; Papyan (2019)).
Revised: ; Papyan (2019)).

Original: E. H IGHER - ORDER NETWORK EVOLUTION
Revised: E. H IGHER - ORDER NETWORK EVOLUTION

Original: In this section we will explain how to compute higher-order corrections to training dynamics.
Revised: In this section we will explain how to compute higher-order corrections to training dynamics.

Original: In principle, this prescription allows one to compute model dynamics as an expansion in 1 /n to arbitrary order.
Revised: In principle, this prescription allows one to compute model dynamics as an expansion in 1 /n to arbitrary order.

Original: We apply this explicitly to compute the O (1 /n ) training dynamics.
Revised: We apply this explicitly to compute the O (1 /n ) training dynamics.

Original: In Figure 10, we present experimental conﬁrmation of our predictions for the evolution of the NTK.
Revised: In Figure 11, we present experimental conﬁrmation of our predictions for the evolution of the NTK.

Original: In the main text, we presented these results for the special case of gradient ﬂow with MSE loss.
Revised: In the main text, we presented these results for the special case of gradient ﬂow with MSE loss.

Original: So far we have mostly been using diagrammatic techniques to understanding the leading order scaling of correlation functions.
Revised: So far we have mostly been using diagrammatic techniques to understanding the leading order scaling of correlation functions.

Original: It is interesting to try and understand ﬁnite width networks by asking how training dynamics are modiﬁed beyond the leading order asymptotics.
Revised: It is interesting to try and understand ﬁnite width networks by asking how training dynamics are modiﬁed beyond the leading order asymptotics.

Original: There are three clear sources of corrections to the leading behavior.
Revised: There are three clear sources of corrections to the leading behavior.

Original: The initial kernel, Θ 0 , receives ﬁnite width corrections.
Revised: The initial kernel, Θ 0 , receives ﬁnite width corrections.

Original: The linear (in learning rate) update for the network function, equation (66), is modiﬁed away from inﬁnite width.
Revised: The linear (in learning rate) update for the network function, equation (73), is modiﬁed away from inﬁnite width.

Original: The kernel is not constant at ﬁnite width.
Revised: The kernel is not constant at ﬁnite width.

Original: The ﬁrst source of corrections is automatically taken into account in typical empirical settings, as the ﬁnite-width Θ 0 can be computed explicitly.
Revised: The ﬁrst source of corrections is automatically taken into account in typical empirical settings, as the ﬁnite-width Θ 0 can be computed explicitly.

Original: The other sources of corrections are non-trivial, and will be the focus of this section.
Revised: The other sources of corrections are non-trivial, and will be the focus of this section.

Original: We begin by explaining how to take into account the non-constancy of the kernel order by order in 1 /n , while maintaining the continuous time approximation.
Revised: We begin by explaining how to take into account the non-constancy of the kernel order by order in 1 /n , while maintaining the continuous time approximation.

Original: Next, we back away from the continuous time limit and write down the full discrete evolution.
Revised: Next, we back away from the continuous time limit and write down the full discrete evolution.

Original: We introduce a method to compute arbitrary order corrections, and explicitly work out the corrections at order 1 /n for MSE loss.
Revised: We introduce a method to compute arbitrary order corrections, and explicitly work out the corrections at order 1 /n for MSE loss.

Original: E.4. C ONTINUOUS TIME .
Revised: E.4. C ONTINUOUS TIME .

Original: Continuous time evolution of the model function in neural networks is governed by the differential equation
Revised: Continuous time evolution of the model function in neural networks is governed by the differential equation

Original: [Equation]
Revised: [Equation]

Original: As we have discussed at length, in the large width limit, this equation simpliﬁes and Θ( t ) is asymptotically constant (Jacot et al.
Revised: As we have discussed at length, in the large width limit, this equation simpliﬁes and Θ( t ) is asymptotically constant (Jacot et al.

Original: Our goal is to move beyond this leading behavior at large width and solve (78) order-by-order in a 1 /n expansion.
Revised: Our goal is to move beyond this leading behavior at large width and solve (85) order-by-order in a 1 /n expansion.

Original: In Section 3 we described how to compute the O ( n − 1 ) corrections for the case of MSE loss.
Revised: In Section 3 we described how to compute the O ( n − 1 ) corrections for the case of MSE loss.

Original: Here we explain how to handle corrections more generally as well as giving a more detailed discussion of the MSE case.
Revised: Here we explain how to handle corrections more generally as well as giving a more detailed discussion of the MSE case.

Original: The network map f and kernel, Θ are members of the following family of operators.
Revised: The network map f and kernel, Θ are members of the following family of operators.

Original: [Equation]
Revised: [Equation]

Original: with O 1 := f .
Revised: with O 1 := f .

Original: Here, as above, O 2 =
Revised: Here, as above, O 2 =

Original: Θ is the kernel.
Revised: Θ is the kernel.

Original: With a general loss function, these operators satisfy.
Revised: With a general loss function, these operators satisfy.

Original: [Equation]
Revised: [Equation]

Original: Equations (79) and (80).
Revised: Equations (86) and (87).

Original: Give an inﬁnite tower of ﬁrst order ODEs, the solution of which gives the time evolution of the network map and the kernel.
Revised: Give an inﬁnite tower of ﬁrst order ODEs, the solution of which gives the time evolution of the network map and the kernel.

Original: [Equation]
Revised: [Equation]

Original: Solving this inﬁnite tower is not feasible.
Revised: Solving this inﬁnite tower is not feasible.

Original: If we wish to work to a given order in an expansion in 1 /n , however, there is a dramatic simpliﬁcation, which makes a solution possible.
Revised: If we wish to work to a given order in an expansion in 1 /n , however, there is a dramatic simpliﬁcation, which makes a solution possible.

Original: Firstly, we can truncate these equations.
Revised: Firstly, we can truncate these equations.

Original: To see this note that the operators
Revised: To see this note that the operators

Original: O s contain s contracted derivative tensors.
Revised: O s contain s contracted derivative tensors.

Original: As a result, by Lemma 6 and Conjecture 1, correlation functions involving the operators, O s satisfy
Revised: As a result, by Lemma 7 and Conjecture 1, correlation functions involving the operators, O s satisfy

Original: [Equation]
Revised: [Equation]

Original: where F ( (cid:126)x ; t ) is arbitrary additional contribution to the integrand.
Revised: where F ( (cid:126)x ; t ) is arbitrary additional contribution to the integrand.

Original: Thus, if we wish to work to solve for the time evolution up to corrections which scale as O ( n − r ) we can truncate the tower at s = 2 r and set O 2 r ( t ) to be equal to its initial value.
Revised: Thus, if we wish to work to solve for the time evolution up to corrections which scale as O ( n − r ) we can truncate the tower at s = 2 r and set O 2 r ( t ) to be equal to its initial value.

Original: Note that the leading order solution, (78), is the result of this procedure with r = 1 and the results presented in the main text for the O ( n − 1 ) evolution correspond to r = 2 .
Revised: Note that the leading order solution, (85), is the result of this procedure with r = 1 and the results presented in the main text for the O ( n − 1 ) evolution correspond to r = 2 .

Original: The truncation provides a dramatic simpliﬁcation, however it is not immediately clear how to solve even the truncated differential equations in (80).
Revised: The truncation provides a dramatic simpliﬁcation, however it is not immediately clear how to solve even the truncated differential equations in (87).

Original: We now describe how to organize the perturbative expansion of the operators O s ( t ) (including f and Θ ) in such a way that the differential equations become tractable.
Revised: We now describe how to organize the perturbative expansion of the operators O s ( t ) (including f and Θ ) in such a way that the differential equations become tractable.

Original: The central idea is to write each operator, O s ( t ) , as an expansion.
Revised: The central idea is to write each operator, O s ( t ) , as an expansion.

Original: [Equation]
Revised: [Equation]

Original: where each order O ( r ) s captures the O ( n − r ) evolution of O s .
Revised: where each order O ( r ) s captures the O ( n − r ) evolution of O s .

Original: For example,
Revised: For example,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The notation O ( r ) s means both that any correlation function containing O ( r ) s is O ( n − r ) and, by
Revised: The notation O ( r ) s means both that any correlation function containing O ( r ) s is O ( n − r ) and, by

Original: Lemma 5, that typical realizations of the operators scale as O ( n − r ) .
Revised: Lemma 6, that typical realizations of the operators scale as O ( n − r ) .

Original: Once the operators O s are organized in this way, solving the differential equations (78) is tractable.
Revised: Once the operators O s are organized in this way, solving the differential equations (85) is tractable.

Original: As the differential equations describing the evolution of O ( r ) s for r > 0 only depend on the time dependent solutions of O ( r − 1) s , we can iteratively solve for the O ( r ) s order by order.
Revised: As the differential equations describing the evolution of O ( r ) s for r > 0 only depend on the time dependent solutions of O ( r − 1) s , we can iteratively solve for the O ( r ) s order by order.

Original: For example, in
Revised: For example, in

Original: Section 3, we used the leading order solution for f and Θ to solve for Θ (1) .
Revised: Section 3, we used the leading order solution for f and Θ to solve for Θ (1) .

Original: In principle this procedure can be extended to arbitrary order in 1 /n .
Revised: In principle this procedure can be extended to arbitrary order in 1 /n .

Original: Before going onto explain the ﬁnite step corrections to this procedure, we reproduce the results of section 3 in more detail.
Revised: Before going onto explain the ﬁnite step corrections to this procedure, we reproduce the results of section 3 in more detail.

Original: MSE continuous time example.
Revised: MSE continuous time example.

Original: The MSE loss is,
Revised: The MSE loss is,

Original: [Equation]
Revised: [Equation]

Original: As such the update equation simpliﬁes to,
Revised: As such the update equation simpliﬁes to,

Original: [Equation]
Revised: [Equation]

Original: The leading order solution to equation (86) is exponential, kernel evolution,
Revised: The leading order solution to equation (93) is exponential, kernel evolution,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we are using a condensed notation where f 0 :
Revised: Here we are using a condensed notation where f 0 :

Original: = f (0) (0) , Θ 0 :=
Revised: = f (0) (0) , Θ 0 :=

Original: Θ (0) (0) .
Revised: Θ (0) (0) .

Original: Equation (87) is a vector equation with f 0 , f (0) ( t ) , and y are vectors over the training set and the leading order kernel Θ 0 is a square matrix over the same space.
Revised: Equation (94) is a vector equation with f 0 , f (0) ( t ) , and y are vectors over the training set and the leading order kernel Θ 0 is a square matrix over the same space.

Original: As discussed above to study the O (1 /n ) evolution, we can truncate the set of equations in (80) at s = 4 , and set O 4 ( t )
Revised: As discussed above to study the O (1 /n ) evolution, we can truncate the set of equations in (87) at s = 4 , and set O 4 ( t )

Original: =
Revised: =

Original: O (1)4 ( t ) =
Revised: O (1)4 ( t ) =

Original: O 4 (0) up to corrections which scale as O ( n − 2 ) .
Revised: O 4 (0) up to corrections which scale as O ( n − 2 ) .

Original: Moving up a rung in (80), we have
Revised: Moving up a rung in (87), we have

Original: [Equation]
Revised: [Equation]

Original: To solve for O (1)3 ( t ) , i.e. neglecting O ( n − 2 ) corrections, we can plug in the leading order approximations, to O 4 ( t ) and f ( t ) .
Revised: To solve for O (1)3 ( t ) , i.e. neglecting O ( n − 2 ) corrections, we can plug in the leading order approximations, to O 4 ( t ) and f ( t ) .

Original: [Equation]
Revised: [Equation]

Original: This gives,
Revised: This gives,

Original: [Equation]
Revised: [Equation]

Original: Using the explicit form of f (0) ( t ) we can write,
Revised: Using the explicit form of f (0) ( t ) we can write,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we are again using a condensed notation.
Revised: Here we are again using a condensed notation.

Original: (cid:126)x = ( x 1 , x 2 , x 3 ) .
Revised: (cid:126)x = ( x 1 , x 2 , x 3 ) .

Original: f 0 and y are vectors over the training set while Θ 0 is a square matrix, and O 4 ( (cid:126)x ; 0) is also a vector over the training set, with the value O 4 ( (cid:126)x, x (cid:48) ; 0) on the point x (cid:48)
Revised: f 0 and y are vectors over the training set while Θ 0 is a square matrix, and O 4 ( (cid:126)x ; 0) is also a vector over the training set, with the value O 4 ( (cid:126)x, x (cid:48) ; 0) on the point x (cid:48)

Original: ∈ D tr .
Revised: ∈ D tr .

Original: [Equation]
Revised: [Equation]

Original: Plugging in O (1)3 , and using the eigen-decomposition of Θ 0 to perform the integrals gives,
Revised: Plugging in O (1)3 , and using the eigen-decomposition of Θ 0 to perform the integrals gives,

Original: [Equation]
Revised: [Equation]

Original: Here, we have introduced the eigenvectors, ˆ e i , which are vectors over the training dataset and eigenvalues λ i of the initial kernel, Θ 0 .
Revised: Here, we have introduced the eigenvectors, ˆ e i , which are vectors over the training dataset and eigenvalues λ i of the initial kernel, Θ 0 .

Original: The vector (cid:126)x = ( x 1 , x 2 ) as Θ depends on two inputs.
Revised: The vector (cid:126)x = ( x 1 , x 2 ) as Θ depends on two inputs.

Original: O 3 ( (cid:126)x ; 0) is a vector over the dataset while O 4 ( (cid:126)x ; 0) is a square matrix and ∆ f 0 :
Revised: O 3 ( (cid:126)x ; 0) is a vector over the dataset while O 4 ( (cid:126)x ; 0) is a square matrix and ∆ f 0 :

Original: =
Revised: =

Original: f 0 − y is a vector over the training data.
Revised: f 0 − y is a vector over the training data.

Original: Finally, we can plug in the expression (93) into (86) and give the sub-leading behavior for f ( t )
Revised: Finally, we can plug in the expression (100) into (93) and give the sub-leading behavior for f ( t )

Original: [Equation]
Revised: [Equation]

Original: This completes the full O ( n − 1 ) time dependance.
Revised: This completes the full O ( n − 1 ) time dependance.

Original: This prediction for the O (1 /n ) time dependence is conﬁrmed experimentally in Figure 10.
Revised: This prediction for the O (1 /n ) time dependence is conﬁrmed experimentally in Figure 11.

Original: Note, one consequence of (94) is that the O (1 /n ) corrections to Θ go to a constant at late times.
Revised: Note, one consequence of (101) is that the O (1 /n ) corrections to Θ go to a constant at late times.

Original: Θ (1)
Revised: Θ (1)

Original: [Equation]
Revised: [Equation]

Original: And the asymptotic evolution of ∆ f ( t ) is again constant kernel evolution, but with a corrected kernel.
Revised: And the asymptotic evolution of ∆ f ( t ) is again constant kernel evolution, but with a corrected kernel.

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that these predictions are quite detailed, giving the full time dependence, rather than just the overall scaling with width.
Revised: It is worth noting that these predictions are quite detailed, giving the full time dependence, rather than just the overall scaling with width.

Original: In deriving these results and in the discrete time analysis below, we implicitly make an assumption that we can control the discrete time corrections or in the continuous time case that f ( t ) , Θ( t ) and all the O s ( t ) are differentiable.
Revised: In deriving these results and in the discrete time analysis below, we implicitly make an assumption that we can control the discrete time corrections or in the continuous time case that f ( t ) , Θ( t ) and all the O s ( t ) are differentiable.

Original: This relies on being able to differentiate through the activation function.
Revised: This relies on being able to differentiate through the activation function.

Original: For non-smooth activations such as ReLU, this assumption is suspect, and indeed the evolution is more subtle in the ReLU case and a full analysis is left to future work.
Revised: For non-smooth activations such as ReLU, this assumption is suspect, and indeed the evolution is more subtle in the ReLU case and a full analysis is left to future work.

Original: For this reason we present experimental results for networks with smooth activation functions.
Revised: For this reason we present experimental results for networks with smooth activation functions.

Original: E.4. D ISCRETE TIME .
Revised: E.4. D ISCRETE TIME .

Original: With the 1 /n corrections to continuous time evolution under our belts, it is possible to also keep track of corrections coming from the discrete update step.
Revised: With the 1 /n corrections to continuous time evolution under our belts, it is possible to also keep track of corrections coming from the discrete update step.

Original: The essential point, that it is possible to solve iteratively, order by order in 1 /n is not changed.
Revised: The essential point, that it is possible to solve iteratively, order by order in 1 /n is not changed.

Original: To see this we can look at how the update equations are modiﬁed.
Revised: To see this we can look at how the update equations are modiﬁed.

Original: Beginning with the network output update equation, we have,
Revised: Beginning with the network output update equation, we have,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: In Section E.2 we argued that the O ( η 2 ) terms vanish at large width.
Revised: In Section E.2 we argued that the O ( η 2 ) terms vanish at large width.

Original: In more detail, each factor of the gradient, g µ , contains a contracted derivative tensor.
Revised: In more detail, each factor of the gradient, g µ , contains a contracted derivative tensor.

Original: Thus, the cluster graph for correlation functions involving the η 2 term will always contain three contracted vertices and thus the correlation functions will scale as O ( n − 1 ) .
Revised: Thus, the cluster graph for correlation functions involving the η 2 term will always contain three contracted vertices and thus the correlation functions will scale as O ( n − 1 ) .

Original: The η 3 term will give four vertices, and thus also scale as O (1 /n ) , the η 4 contribution will be O (1 /n 2 ) and so on.
Revised: The η 3 term will give four vertices, and thus also scale as O (1 /n ) , the η 4 contribution will be O (1 /n 2 ) and so on.

Original: Just as in the continuous time case this equation is still solvable order by order in 1 /n .
Revised: Just as in the continuous time case this equation is still solvable order by order in 1 /n .

Original: What we mean by solvable is the following.
Revised: What we mean by solvable is the following.

Original: All terms on the RHS appearing with f t are already higher order as a result of their derivative structure.
Revised: All terms on the RHS appearing with f t are already higher order as a result of their derivative structure.

Original: This means, just as in the continuous case, we can use the lower order solution of f t in the gradient terms on the RHS to solve for f t on the LHS.
Revised: This means, just as in the continuous case, we can use the lower order solution of f t in the gradient terms on the RHS to solve for f t on the LHS.

Original: Similarly, the update equation for Θ receives discrete time modiﬁcations.
Revised: Similarly, the update equation for Θ receives discrete time modiﬁcations.

Original: [Equation]
Revised: [Equation]

Original: Here we have adopted a notation where, (cid:126)x = ( x 1 , . . .
Revised: Here we have adopted a notation where, (cid:126)x = ( x 1 , . . .

Original: , x s ) .
Revised: , x s ) .

Original: Just as for f , increased powers in η are increasingly suppressed in n and we can iteratively solve this equation using solutions at leading orders in n to solve for sub-leading behavior.
Revised: Just as for f , increased powers in η are increasingly suppressed in n and we can iteratively solve this equation using solutions at leading orders in n to solve for sub-leading behavior.

Original: More generally the tower of differential equations deﬁned recursively in (80) becomes
Revised: More generally the tower of differential equations deﬁned recursively in (87) becomes

Original: [Equation]
Revised: [Equation]

Original: As in the continuous time case, at a given order in n
Revised: As in the continuous time case, at a given order in n

Original: [Equation]
Revised: [Equation]

Original: we can truncate this tower and use lower order solutions to solve for the time evolution up to the desired order.
Revised: we can truncate this tower and use lower order solutions to solve for the time evolution up to the desired order.

Original: To ground this discussion in a concrete use case, we again walk through the 1 /n corrections for MSE loss, now in discrete time
Revised: To ground this discussion in a concrete use case, we again walk through the 1 /n corrections for MSE loss, now in discrete time

Original: [Equation]
Revised: [Equation]

Original: MSE discrete time example.
Revised: MSE discrete time example.

Original: For discrete time evolution, the leading order behavior of the model map is
Revised: For discrete time evolution, the leading order behavior of the model map is

Original: [Equation]
Revised: [Equation]

Original: At next order we can proceed by solving the truncated set of equations in (101).
Revised: At next order we can proceed by solving the truncated set of equations in (108).

Original: The ﬁrst equation, for O 4; t still gives a constant solution,
Revised: The ﬁrst equation, for O 4; t still gives a constant solution,

Original: [Equation]
Revised: [Equation]

Original: Here, both the discrete and continuous time terms that would appear on the RHS are O (1 /n 2 ) .
Revised: Here, both the discrete and continuous time terms that would appear on the RHS are O (1 /n 2 ) .

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we must solve for O 3; t .
Revised: we must solve for O 3; t .

Original: The discrete time update is,
Revised: The discrete time update is,

Original: [Equation]
Revised: [Equation]

Original:  To order 1 /n we can drop the order η 2 and higher terms, as they contain expressions with 5 or more contracted derivative tensors.
Revised: [Equation] To order 1 /n we can drop the order η 2 and higher terms, as they contain expressions with 5 or more contracted derivative tensors.

Original: Thus at this order, we are left with the discrete analogue of (89).
Revised: Thus at this order, we are left with the discrete analogue of (96)

Original: [Equation]
Revised: [Equation]

Original: which we can sum to get the discrete version of (91),
Revised: which we can sum to get the discrete version of (98),

Original: [Equation]
Revised: [Equation]

Original: Explicitly, this takes the form
Revised: Explicitly, this takes the form

Original: [Equation]
Revised: [Equation]

Original: Here we have adopted notation similar to above, where O 4;0 ( (cid:126)x ) is a vector over the training dataset.
Revised: Here we have adopted notation similar to above, where O 4;0 ( (cid:126)x ) is a vector over the training dataset.

Original: So far, this procedure has been a discrete analogue of what we have done in the continuous time case, however as we move on to compute Θ and f we will have to keep track of the novel corrections, which vanish in continuous time.
Revised: So far, this procedure has been a discrete analogue of what we have done in the continuous time case, however as we move on to compute Θ and f we will have to keep track of the novel corrections, which vanish in continuous time.

Original: Explicitly, at order 1 /n the discrete update for Θ t is given by,
Revised: Explicitly, at order 1 /n the discrete update for Θ t is given by,

Original: [Equation]
Revised: [Equation]

Original: This can be summed to give Θ (1) t .
Revised: This can be summed to give Θ (1) t .

Original: To move up to the neural network map itself there is one additional complication.
Revised: To move up to the neural network map itself there is one additional complication.

Original: The O ( η 2 /n ) term,
Revised: The O ( η 2 /n ) term,

Original: [Equation]
Revised: [Equation]

Original: , in (99) has non trivial time dependence.
Revised: , in (106) has non trivial time dependence.

Original: We can deal with this just as we have been doing, by taking an extra time derivative and integrating.
Revised: We can deal with this just as we have been doing, by taking an extra time derivative and integrating.

Original: Deﬁning,
Revised: Deﬁning,

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: with the solution
Revised: with the solution

Original: [Equation]
Revised: [Equation]

Original: This can in turn be plugged into the update equation for f t .
Revised: This can in turn be plugged into the update equation for f t .

Original: [Equation]
Revised: [Equation]

Original: Here ( x a , y a ) are elements of the training set D tr and summed over.
Revised: Here ( x a , y a ) are elements of the training set D tr and summed over.

Original: This equation can be solved to give f (1) t .
Revised: This equation can be solved to give f (1) t .

Original: [Equation]
Revised: [Equation]

Original: Where Disc t contains the discrete derivative updates at O (1 /n )
Revised: Where Disc t contains the discrete derivative updates at O (1 /n )

Original: [Equation]
Revised: [Equation]

Original: These expressions may look fairly intimidating.
Revised: These expressions may look fairly intimidating.

Original: The key point is that all terms in the summand in (109) and thus (114) are known functions of time and initial data, just as in the continuous time setting.
Revised: The key point is that all terms in the summand in (116) and thus (121) are known functions of time and initial data, just as in the continuous time setting.