Original: Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Revised: Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In this paper we show that feedforward neural networks corresponding to arbitrarydirected acyclic graphs undergo transition to linearity as their “width” approachesinﬁnity.
Revised: In this paper we show that feedforward neural networks corresponding to arbitrarydirected acyclic graphs undergo transition to linearity as their “width” approachesinﬁnity.

Original: The width of these general networks is characterized by the minimum indegree of their neurons, except for the input and ﬁrst layers.
Revised: The width of these general networks is characterized by the minimum indegree of their neurons, except for the input and ﬁrst layers.

Original: Our results identify themathematical structure underlying transition to linearity and generalize a numberof recent works aimed at characterizing transition to linearity or constancy of the
Revised: Our results identify themathematical structure underlying transition to linearity and generalize a numberof recent works aimed at characterizing transition to linearity or constancy of the

Original: Neural Tangent Kernel for standard architectures.
Revised: Neural Tangent Kernel for standard architectures.

Original:  Introduction
Revised:  Introduction

Original: A remarkable property of wide neural networks, ﬁrst discovered in [10] in terms of the constancyof the Neural Tangent Kernel along the optimization path, is that they transition to linearity (usingthe terminology from [14]), i.e., are approximately linear in a ball of a ﬁxed radius.
Revised: A remarkable property of wide neural networks, ﬁrst discovered in [9] in terms of the constancy ofthe Neural Tangent Kernel along the optimization path, is that they transition to linearity (using theterminology from [13]), i.e., are approximately linear in a ball of a ﬁxed radius.

Original: There has beenan extensive study of this phenomenon for different types of standard neural networks architecturesincluding fully-connected neural networks (FCNs), convolutional neural networks (CNNs), ResNets[12, 4, 3, 7].
Revised: There has been anextensive study of this phenomenon for different types of standard neural networks architecturesincluding fully-connected neural networks (FCNs), convolutional neural networks (CNNs), ResNets[11, 4, 3, 7].

Original: Yet the scope of the transition to linearity and the underlying mathematical structure hasnot been made completely clear.
Revised: Yet the scope of the transition to linearity and the underlying mathematical structure hasnot been made completely clear.

Original: In this paper, we show that the property of transition to linearity holds for a much broader class ofneural networks, feedforward neural networks.
Revised: In this paper, we show that the property of transition to linearity holds for a much broader class ofneural networks, feedforward neural networks.

Original: The architecture of a feedforward neural network cangenerically be described by a DAG [26, 25, 16]: the vertices and the edges correspond to the neuronsand the trainable weight parameters of a neural network, respectively.
Revised: The architecture of a feedforward neural network cangenerically be described by a DAG [23, 22, 15]: the vertices and the edges correspond to the neuronsand the trainable weight parameters of a neural network, respectively.

Original: This DAG structure includesstandard network architectures e.g., FCNs,CNNs, ResNets, as well as DenseNets [9] and networkswith dropout [20], whose property of transition to linearity has not been studied in literature.
Revised: This DAG structure includesstandard network architectures e.g., FCNs, ResNets, as well as DenseNets [8], whose property oftransition to linearity has not been studied in literature.

Original: Thisgeneralization shows that the transition to linearity, or the constant Neural Tangent Kernel, does notdepend on the speciﬁc designs of the networks, and is a more fundamental and universal property.
Revised: This generalization shows that the transitionto linearity, or the constant Neural Tangent Kernel, does not depend on the speciﬁc designs of thenetworks, and is a more fundamental and universal property.

Original: We deﬁne the width of a feedforward neural network as the minimum in-degree of all neurons exceptfor the input and ﬁrst layers, which is a natural generalization of the the minimum number of neuronsin hidden layers which is how the width is deﬁned for standard architectures.
Revised: We deﬁne the width of a feedforward neural network as the minimum in-degree of all neurons exceptfor the input and ﬁrst layers, which is a natural generalization of the the minimum number of neuronsin hidden layers which is how the width is deﬁned for standard architectures.

Original: For a feedforwardneural network, we show it transitions to linearity if its width goes to inﬁnity as long as the in-degreesof individual neurons are bounded by a polynomial of the network width.
Revised: For a feedforwardneural network, we show it transitions to linearity if its width goes to inﬁnity as long as the in-degreesof individual neurons are bounded by a polynomial of the network width.

Original: Speciﬁcally, we control thedeviation of the network function from its linear approximation by the spectral norm of the Hessianof the network function, which, as we show vanishes in a ball of ﬁxed radius, in the inﬁnite widthlimit.
Revised: Speciﬁcally, we control thedeviation of the network function from its linear approximation by the spectral norm of the Hessianof the network function, which, as we show vanishes in a ball of ﬁxed radius, in the inﬁnite widthlimit.

Original: Interestingly, we observe that not only the output neurons, but any pre-activated neuron inthe hidden layers of a feedforward neural network can be regarded as a function with respect to itsparameters, which will also transition to linearity as the width goes to inﬁnity.
Revised: Interestingly, we observe that not only the output neurons, but any pre-activated neuron inthe hidden layers of a feedforward neural network can be regarded as a function with respect to itsparameters, which will also transition to linearity as the width goes to inﬁnity.

Original: The key technical difﬁculty is that all existing analyses for transition to linearity or constant NTK donot apply to this general DAG setting.
Revised: The key technical difﬁculty is that all existing analyses for transition to linearity or constant NTK donot apply to this general DAG setting.

Original: Speciﬁcally, those analyses assume in-degrees of neurons are either the same or proportional to each other up to a constant ratio [5, 12, 3, 27, 14, 2].
Revised: Speciﬁcally, those analyses assume in-degrees of neurons are either the same or proportional to each other up to a constant ratio [5, 11, 3, 24, 13, 2].

Original: However, thegeneral DAG setting allows different scales of neuron in-degrees, for example, the largest in-degreecan be polynomially large in the smallest in-degree.
Revised: However, thegeneral DAG setting allows different scales of neuron in-degrees, for example, the largest in-degreecan be polynomially large in the smallest in-degree.

Original: In such scenarios, the (2 , 2 , 1) -norm in [14] andthe norm of parameter change in [5, 12] scales with the maximum of in-degrees which causes a trivialbound on the NTK change.
Revised: In such scenarios, the (2 , 2 , 1) -norm in [13] andthe norm of parameter change in [5, 11] scales with the maximum of in-degrees which causes a trivialbound on the NTK change.

Original: Instead, we introduce a different set of tools based on the tail boundfor the norm of matrix Gaussian series [23].
Revised: Instead, we introduce a different set of tools based on the tail boundfor the norm of matrix Gaussian series [20].

Original: Speciﬁcally, we show that the Hessian of the networkfunction takes the form of matrix Gaussian series, whose matrix variance relies on the Hessian ofconnected neurons.
Revised: Speciﬁcally, we show that the Hessian of the networkfunction takes the form of matrix Gaussian series, whose matrix variance relies on the Hessian ofconnected neurons.

Original: Therefore, we reconcile the in-degree difference by building a recursive relationbetween the Hessian of neurons, which exactly cancels out the in-degree with the scaling factor.
Revised: Therefore, we reconcile the in-degree difference by building a recursive relationbetween the Hessian of neurons, which exactly cancels out the in-degree with the scaling factor.

Original: Transition to linearity helps understand the training dynamics of wide neural networks and playsan important role in developing the optimization theory for them, as has been shown for certainparticular wide neural networks [6, 5, 4, 12, 28, 27].
Revised: Transition to linearity helps understand the training dynamics of wide neural networks and playsan important role in developing the optimization theory for them, as has been shown for certainparticular wide neural networks [6, 5, 4, 11, 25, 24].

Original: While transition to linearity is not a necessarycondition for successful optimization, it provides a powerful tool for analyzing optimization for manydifferent architectures.
Revised: While transition to linearity is not a necessarycondition for successful optimization, it provides a powerful tool for analyzing optimization for manydifferent architectures.

Original: Speciﬁcally, transition to linearity in a ball of sufﬁcient radius combinedwith a lower bound on the norm of the gradient at its center is sufﬁcient to demonstrate the PL ⇤condition [13] (a version of the Polyak- Ł ojasiewicz condition [19, 15]) which ensures convergenceof optimization.
Revised: Speciﬁcally, transition to linearity in a ball of sufﬁcient radius combinedwith a lower bound on the norm of the gradient at its center is sufﬁcient to demonstrate the PL ⇤condition [12] (a version of the Polyak- Ł ojasiewicz condition [18, 14]) which ensures convergenceof optimization.

Original: We discuss this connection and provide one such lower bound in Section 4.
Revised: We discuss this connection and provide one such lower bound in Section 4.

Original: Summary of contributions.
Revised: Summary of contributions.

Original: We show the phenomenon of transition to linearity in general feedforward neural networks corresponding to a DAG with large in-degree.
Revised: We show the phenomenon of transition to linearity in general feedforward neural networks corresponding to a DAG with large in-degree.

Original: Speciﬁcally, under theassumption that the maximum in-degree of its neurons is bounded by a polynomial of the width m(the minimum in-degree), we prove that the spectral norm of the Hessian of a feedforward neuralnetwork is bounded by ˜ O
Revised: Speciﬁcally, under theassumption that the maximum in-degree of its neurons is bounded by a polynomial of the width m(the minimum in-degree), we prove that the spectral norm of the Hessian of a feedforward neuralnetwork is bounded by ˜ O

Original: (1  p m ) in an O (1) ball.
Revised: (1 / p m ) in an O (1) ball.

Original: Our results generalize the existing literature onthe linearity of wide feedforward neural networks.
Revised: Our results generalize the existing literature onthe linearity of wide feedforward neural networks.

Original: We discuss connections to optimization.
Revised: We discuss connections to optimization.

Original: Underadditional assumptions we show that the norm of the gradient of a feedforward neural network isbounded away from zero at initialization.
Revised: Underadditional assumptions we show that the norm of the gradient of a feedforward neural network isbounded away from zero at initialization.

Original: Together with the Hessian bound this implies convergenceof gradient descent for the loss function.
Revised: Together with the Hessian bound this implies convergenceof gradient descent for the loss function.

Original: 1.1 Notations
Revised: 1.1 Notations

Original: We use bold lowercase letters, e.g., w , to denote vectors, capital letters, e.g., A , to denote matrices,and bold capital letters, e.g., H , to denote higher order tensors or matrix tuples.
Revised: We use bold lowercase letters, e.g., w , to denote vectors, capital letters, e.g., A , to denote matrices,and bold capital letters, e.g., H , to denote higher order tensors or matrix tuples.

Original: For a matrix A , weuse A [ i, :] to denote its i -th row and A [: ,i ] to denote its j -th column.
Revised: For a matrix A , weuse A [ i, :] to denote its i -th row and A [: ,i ] to denote its j -th column.

Original: We use r w f ( w 0 ) to denote the gradient of f with respect to w at w 0 , and H f ( w ) to denote Hessianmatrix (second derivative) of f with respect to w .
Revised: We use r w f ( w 0 ) to denote the gradient of f with respect to w at w 0 , and H f ( w ) to denote Hessianmatrix (second derivative) of f with respect to w .

Original: For vectors, we use k · k to denote Euclidean norm.
Revised: For vectors, we use k · k to denote Euclidean norm.

Original: For matrices, we use k · k to denote spectral norm and k · k F to denote Frobenius norm.
Revised: For matrices, we use k · k to denote spectral norm and k · k F to denote Frobenius norm.

Original: We usek · k 1 to denote function L 1 norm.
Revised: We usek · k 1 to denote function L 1 norm.

Original: For a set S , we use |S| to denote the cardinality of the set.
Revised: For a set S , we use |S| to denote the cardinality of the set.

Original: For 69 n > 0 , [ n ] denotes the set { 1 , 2 , ..., n } .
Revised: For 69 n > 0 , [ n ] denotes the set { 1 , 2 , ..., n } .

Original: We use big-
Revised: We use big-

Original: O notation to hide constant factors, and use big-
Revised: O notation to hide constant factors, and use big-

Original: ˜ O notation to additionally hide logarithmicfactors.
Revised: ˜ O notation to additionally hide logarithmicfactors.

Original: In this paper, the argument of O / ˜ O ( · ) is always with respect to the network width.
Revised: In this paper, the argument of O / ˜ O ( · ) is always with respect to the network width.

Original: Given a vector w and a constant R > 0 , we deﬁne a Euclidean ball B ( w , R ) as:[Equation]
Revised: Given a vector w and a constant R > 0 , we deﬁne a Euclidean ball B ( w , R ) as:

Original: [Equation]
Revised: [Equation]

Original:  Neural networks with acyclic graph architecture
Revised:  Neural networks with acyclic graph architecture

Original: In this section, we provide a deﬁnition and notation for general feedforward neural networks with anarbitrary DAG structure.
Revised: In this section, we provide a deﬁnition and notation for general feedforward neural networks with anarbitrary DAG structure.

Original: This deﬁnition includes standard feedforward neural network architectures,such as FCNs, DenseNet and CNNs.
Revised: This deﬁnition includes standard feedforward neural network architectures,such as FCNs, DenseNet and CNNs.

Original: 2.1 Deﬁning feedforward neural networks
Revised: 2.1 Deﬁning feedforward neural networks

Original: Graph Structure.
Revised: Graph Structure.

Original: Consider a directed acyclic graph (DAG)
Revised: Consider a directed acyclic graph (DAG)

Original: G =
Revised: G =

Original: ( V , E ) , where V and E denote the 79 sets of vertices and edges, respectively.
Revised: ( V , E ) , where V and E denote the 79 sets of vertices and edges, respectively.

Original: See the left panel of Figure 1, for an illustrative example.
Revised: See the left panel of Figure 1, for an illustrative example.

Original: For a directed edge e 2 E , we may also use the notation e
Revised: For a directed edge e 2 E , we may also use the notation e

Original: = ( v 1 , v 2 ) to explicitly write out the startvertex v 1 and end vertex v 2 .
Revised: = ( v 1 , v 2 ) to explicitly write out the startvertex v 1 and end vertex v 2 .

Original: For a vertex v 2 V , we denote its in-degree, in ( v ) , by the number of incoming edges (edges that endwith it):in ( v ) =
Revised: For a vertex v 2 V , we denote its in-degree, in ( v ) , by the number of incoming edges (edges that endwith it):in ( v ) =

Original: |S in ( v ) | , with S in ( v ) :
Revised: |S in ( v ) | , with S in ( v ) :

Original: = { u 2 V : ( u, v ) 2 E} .
Revised: = { u 2 V : ( u, v ) 2 E} .

Original: Similarly, for a vertex v 2 V , we denote its out-degree out ( v ) by the number of outgoing edges(edges that start from it):out ( v ) =
Revised: Similarly, for a vertex v 2 V , we denote its out-degree out ( v ) by the number of outgoing edges(edges that start from it):out ( v ) =

Original: |S out ( v ) | , with S out ( v ) :
Revised: |S out ( v ) | , with S out ( v ) :

Original: = { u 2 V : ( v, u ) 2 E} .
Revised: = { u 2 V : ( v, u ) 2 E} .

Original: We call the set of vertices with zero in-degrees input : V input = { v 2 V : in ( v ) = 0 } , and the set of 87 vertices with zero out-degrees output
Revised: We call the set of vertices with zero in-degrees input : V input = { v 2 V : in ( v ) = 0 } , and the set of 87 vertices with zero out-degrees output

Original: V output = { v 2 V : out ( v ) = 0 } .
Revised: V output = { v 2 V : out ( v ) = 0 } .

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: For each vertex v 2 V\V input , its distance p ( v ) , to the input V input , is deﬁned to be 89 the maximum length of all paths that start from a vertex within
Revised: For each vertex v 2 V\V input , its distance p ( v ) , to the input V input , is deﬁned to be 89 the maximum length of all paths that start from a vertex within

Original: V input and end with v .
Revised: V input and end with v .

Original: It is easy to check that p ( v )
Revised: It is easy to check that p ( v )

Original: =
Revised: =

Original: if v 2 V input .
Revised: if v 2 V input .

Original: Feedforward neural network.
Revised: Feedforward neural network.

Original: Based on a given DAG architecture, we deﬁne the feedforward neuralnetwork.
Revised: Based on a given DAG architecture, we deﬁne the feedforward neuralnetwork.

Original: Each individual vertex corresponds to a neuron additionally equipped with a scalar function(also called activation function).
Revised: Each individual vertex corresponds to a neuron additionally equipped with a scalar function(also called activation function).

Original: Each edge is associated with a real-valued weight, a trainableparameter.
Revised: Each edge is associated with a real-valued weight, a trainableparameter.

Original: Each neuron is deﬁned as a function of the weight parameters and the adjacent neuronsconnected by its incoming edges.
Revised: Each neuron is deﬁned as a function of the weight parameters and the adjacent neuronsconnected by its incoming edges.

Original: The feedforward neural network is considered as the output neurons,corresponding to the output V output , of all weight parameters and input neurons which correspond tothe input V input .
Revised: The feedforward neural network is considered as the output neurons,corresponding to the output V output , of all weight parameters and input neurons which correspond tothe input V input .

Original: Formally, we deﬁne the feedforward neural network as follows.
Revised: Formally, we deﬁne the feedforward neural network as follows.

Original: Deﬁnition 2.2 (Feedforward neural network) .
Revised: Deﬁnition 2.2 (Feedforward neural network) .

Original: Consider a DAG G =
Revised: Consider a DAG G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: For each vertex 99 v 2 V\V input , we associate it with an activation function  v ( · ) : R !
Revised: For each vertex 99 v 2 V\V input , we associate it with an activation function  v ( · ) : R !

Original: R and each of its incoming 100 edges e =
Revised: R and each of its incoming 100 edges e =

Original: ( u, v ) 2 E with a weight variable w e = w ( u,v ) .
Revised: ( u, v ) 2 E with a weight variable w e = w ( u,v ) .

Original: Then we deﬁne the following functions:
Revised: Then we deﬁne the following functions:

Original: [Equation]
Revised: [Equation]

Original: When v 2 V input , f v is preﬁxed as the input data, and we denote f input :=
Revised: When v 2 V input , f v is preﬁxed as the input data, and we denote f input :=

Original: { f v : v 2 V input } .
Revised: { f v : v 2 V input } .

Original: Forv / 2 V input , we call f v neurons and ˜ f v pre-activations .
Revised: Forv / 2 V input , we call f v neurons and ˜ f v pre-activations .

Original: With necessary composition of functions,each f v , and ˜ f v , can be regarded as a function of all related weight variables and inputs f input .
Revised: With necessary composition of functions,each f v , and ˜ f v , can be regarded as a function of all related weight variables and inputs f input .

Original: Thefeedforward neural network is deﬁned to be the function corresponding to the output V output :
Revised: Thefeedforward neural network is deﬁned to be the function corresponding to the output V output :

Original: [Equation]
Revised: [Equation]

Original: where W := { w e : e 2 E} denotes the set of all the weight variables.
Revised: where W := { w e : e 2 E} denotes the set of all the weight variables.

Original: Remark 2.3 .
Revised: Remark 2.3 .

Original: The validity of the deﬁnition is guaranteed by the fact that the DAG is acyclic.
Revised: The validity of the deﬁnition is guaranteed by the fact that the DAG is acyclic.

Original: It makessure that the dependence of each function f v on other neurons can pass all the way down to the inputf input , through Eq.
Revised: It makessure that the dependence of each function f v on other neurons can pass all the way down to the inputf input , through Eq.

Original: Remark 2.4 .
Revised: Remark 2.4 .

Original: For v 2 V input
Revised: For v 2 V input

Original: S
Revised: S

Original: V output , we use the identity function I ( · ) as the activation functions.
Revised: V output , we use the identity function I ( · ) as the activation functions.

Original: Weight initialization and inputs.
Revised: Weight initialization and inputs.

Original: Each weight parameter w e 2 W is initialized i.i.d.
Revised: Each weight parameter w e 2 W is initialized i.i.d.

Original: followingthe standard normal distribution i.e., N (0 , 1) .
Revised: followingthe standard normal distribution i.e., N (0 , 1) .

Original: The inputs are considered given, usually determinedby datasets.
Revised: The inputs are considered given, usually determinedby datasets.

Original: Under this initialization, we introduce the scaling factor 1 / p in ( v ) in Eq.
Revised: Under this initialization, we introduce the scaling factor 1 / p in ( v ) in Eq.

Original: (5) to controlthe value of neurons to be of order O (1) .
Revised: (5) to controlthe value of neurons to be of order O (1) .

Original: Note that this initialization is an extension of the NTKinitialization [10], which was deﬁned for FCNs therein.
Revised: Note that this initialization is an extension of the NTKinitialization [9], which was deﬁned for FCNs therein.

Original: Generality of DAG architecture.
Revised: Generality of DAG architecture.

Original: The feedforward neural networks include FCNs, CNNs, DenseNets [9] as special examples. Notably, neural networks with dropout layers [20], whoseproperty of transition to linearity has not been studied in literature, also ﬁt into our deﬁnition.
Revised: Including FCNs and DenseNets [8] as special examples, theclass of feedforward neural networks allows much more choices of architectures, for example, neuralnetworks with randomly dropped edges.

Original: Pleasesee detailed discussions in Appendix A.
Revised: Please see detailed discussions about these speciﬁc examplesin Appendix A.

Original: We note that our deﬁnition of feedforward neural networksdoes not directly include networks with skip connection, e.g., ResNets. However, as shown in
Revised: We note that networks with nont-trainable skip connections, e.g., ResNets, do notdirectly ﬁt the deﬁnition of feedforward neural networks, but the transition to linearity property stillholds when skip connections present.

Original:  Appendix D, our main results still apply with skip connections present.
Revised: See the detailed discussion in Appendix D.

Original: Therefore, architectures thatare the combination of FCNs, CNNs, ResNets, etc., are included within our framework [11, 21, 8,22].

Original: 2.2 Organizing feedforward networks into layers
Revised: 2.2 Organizing feedforward networks into layers

Original: The architecture of the feedforward neural network is determined by the DAG G .
Revised: The architecture of the feedforward neural network is determined by the DAG G .

Original: The complexstructures of DAGs often lead to complicated neural networks, which are hard to analyze.
Revised: The complexstructures of DAGs often lead to complicated neural networks, which are hard to analyze.

Original: For the ease of analysis, we organize the neurons of the feedforward neural network into layers ,which are sets of neurons.
Revised: For the ease of analysis, we organize the neurons of the feedforward neural network into layers ,which are sets of neurons.

Original: Deﬁnition 2.5 (Layers) .
Revised: Deﬁnition 2.5 (Layers) .

Original: Consider a feedforward neural network f and its corresponding graphstructure G .
Revised: Consider a feedforward neural network f and its corresponding graphstructure G .

Original: A layer of the network is deﬁned to be the set of neurons which have the same distance pto the inputs.
Revised: A layer of the network is deﬁned to be the set of neurons which have the same distance pto the inputs.

Original: Speciﬁcally, the ` -th layer, denoted by f ( ` ) , is
Revised: Speciﬁcally, the ` -th layer, denoted by f ( ` ) , is

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that the layers are mutually exclusive, and the layer index ` is labeled from 0 to ` ,where L + 1 is the total number of layers in the network.
Revised: It is easy to see that the layers are mutually exclusive, and the layer index ` is labeled from 0 to ` ,where L + 1 is the total number of layers in the network.

Original: As p ( v ) = 

Revised: As p ( v ) =

Original: if and only if v 2 V input , the0 -th layer f (0) is exactly the input layer f input .
Revised: if and only if v 2 V input , the0 -th layer f (0) is exactly the input layer f input .

Original: The right panel of Figure 1 provides an illustrativeexample of the layer structures.
Revised: The right panel of Figure 1 provides an illustrativeexample of the layer structures.

Original: In general, the output neurons f output (deﬁned in Eq.
Revised: In general, the output neurons f output (deﬁned in Eq.

Original: (3)) do not have to be in the same layer.
Revised: (3)) do not have to be in the same layer.

Original: For theconvenience of presentation and analysis, we assume that all the output neurons are in the last layer,i.e., layer ` , which is the case for most of commonly used neural networks, e.g., FCNs and CNNs.
Revised: For theconvenience of presentation and analysis, we assume that all the output neurons are in the last layer,i.e., layer ` , which is the case for most of commonly used neural networks, e.g., FCNs and CNNs.

Original: Indeed, our analysis applies to every output neuron (see Theorem 3.8), even if they are not in thesame layer.
Revised: Indeed, our analysis applies to every output neuron (see Theorem 3.8), even if they are not in thesame layer.

Original: With the notion of network layers, we rewrite the neuron functions Eq.
Revised: With the notion of network layers, we rewrite the neuron functions Eq.

Original: (2), as well as related notations,to reﬂect the layer information.
Revised: (2), as well as related notations,to reﬂect the layer information.

Original: For ` -layer, ` = 0 , 1 , · · · , L , we denote the total number of neurons as d ` , and rewrite the layerfunction f ( ` ) into a form of vector-valued function
Revised: For ` -layer, ` = 0 , 1 , · · · , L , we denote the total number of neurons as d ` , and rewrite the layerfunction f ( ` ) into a form of vector-valued function

Original: [Equation]
Revised: [Equation]

Original: where we use f ( ` ) i with index i
Revised: where we use f ( ` ) i with index i

Original: = 1 , 2 , · · · , d ` to denote each individual neuron.
Revised: = 1 , 2 , · · · , d ` to denote each individual neuron.

Original: Correspondingly,we denote its vertex as v ( ` ) i , and S ( ` ) i :=
Revised: Correspondingly,we denote its vertex as v ( ` ) i , and S ( ` ) i :=

Original: S in ( v ( ` ) i ) .
Revised: S in ( v ( ` ) i ) .

Original: Hence, the in-degree in ( v ( ` ) i ) , denoted as m ( ` ) ihere, is equivalent to the cardinality of the set S ( ` ) i .
Revised: Hence, the in-degree in ( v ( ` ) i ) , denoted as m ( ` ) ihere, is equivalent to the cardinality of the set S ( ` ) i .

Original: Remark 2.6 .
Revised: Remark 2.6 .

Original: Note that m ( ` ) i , with the superscript ` , denotes an in-degree, i.e., the number of neuronsthat serve as direct inputs to the current neuron in ` -th layer.
Revised: Note that m ( ` ) i , with the superscript ` , denotes an in-degree, i.e., the number of neuronsthat serve as direct inputs to the current neuron in ` -th layer.

Original: In the context of FCNs, m ( ` ) i is equivalentto the size of its previous layer, i.e., ( `  1) -th layer, and is often denoted as m ( `  1) in literature.
Revised: In the context of FCNs, m ( ` ) i is equivalentto the size of its previous layer, i.e., ( `  1) -th layer, and is often denoted as m ( `  1) in literature.

Original: To write the summation in Eq.
Revised: To write the summation in Eq.

Original: (2) as a matrix multiplication, we further introduce the following twovectors: (a), f S ( ` ) i represents the vector that consists of neuron components f v with v 2 S ( ` ) i ; (b),w ( ` ) i represents the vector that consists of weight parameters w ( u,v ( ` ) i ) with u 2 S ( ` ) i .
Revised: (2) as a matrix multiplication, we further introduce the following twovectors: (a), f S ( ` ) i represents the vector that consists of neuron components f v with v 2 S ( ` ) i ; (b),w ( ` ) i represents the vector that consists of weight parameters w ( u,v ( ` ) i ) with u 2 S ( ` ) i .

Original: Note that bothvectors f S ( ` ) i and w ( ` ) i have the same dimension m ( ` ) i .
Revised: Note that bothvectors f S ( ` ) i and w ( ` ) i have the same dimension m ( ` ) i .

Original: With the above notation, the neuron functions Eq.
Revised: With the above notation, the neuron functions Eq.

Original: (2) can be equivalently rewritten as:
Revised: (2) can be equivalently rewritten as:

Original: [Equation]
Revised: [Equation]

Original: For any ` 2 [ L ] , we denote the weight parameters corresponding to all incoming edges towardneurons at layer ` by
Revised: For any ` 2 [ L ] , we denote the weight parameters corresponding to all incoming edges towardneurons at layer ` by

Original: [Equation]
Revised: [Equation]

Original: Through the way we deﬁne the feedforward neural network, the output of the neural network is afunction of all the weight parameters and the input data, hence we denote it by
Revised: Through the way we deﬁne the feedforward neural network, the output of the neural network is afunction of all the weight parameters and the input data, hence we denote it by

Original: [Equation]
Revised: [Equation]

Original: where w is the collection of all the weight parameters, i.e., w := 
Revised: where w is the collection of all the weight parameters, i.e., w := [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: With all the notations, for a feedforward neural network, we formally deﬁne the width of it:
Revised: With all the notations, for a feedforward neural network, we formally deﬁne the width of it:

Original: Deﬁnition 2.7 (Network width) .
Revised: Deﬁnition 2.7 (Network width) .

Original: The width m of a feedforward neural network is the minimumin-degree of all the neurons except those in the input and ﬁrst layers:
Revised: The width m of a feedforward neural network is the minimumin-degree of all the neurons except those in the input and ﬁrst layers:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.8 .
Revised: Remark 2.8 .

Original: Note that, the network width m is determined by the in-degrees of neurons except forthe input and ﬁrst layers, and not necessarily relates the number of neurons in hidden layers.
Revised: Note that, the network width m is determined by the in-degrees of neurons except forthe input and ﬁrst layers, and not necessarily relates the number of neurons in hidden layers.

Original: But forcertain architectures e.g., FCNs, these two coincide that the minimum in-degree after the ﬁrst layer isthe same as the minimum hidden layer size.
Revised: But forcertain architectures e.g., FCNs, these two coincide that the minimum in-degree after the ﬁrst layer isthe same as the minimum hidden layer size.

Original: We say a feedforward neural network is wide if its width m is large enough.
Revised: We say a feedforward neural network is wide if its width m is large enough.

Original: In this paper, we considerwide feedforward neural networks with a ﬁxed number of layers.
Revised: In this paper, we considerwide feedforward neural networks with a ﬁxed number of layers.

Original:  Transition to linearity of feedforward neural networks
Revised:  Transition to linearity of feedforward neural networks

Original: In this section, we show that the feedforward neural networks exhibit the phenomenon of transitionto linearity, which was previously observed in speciﬁc types of neural networks.
Revised: In this section, we show that the feedforward neural networks exhibit the phenomenon of transitionto linearity, which was previously observed in speciﬁc types of neural networks.

Original: Speciﬁcally, we prove that a feedforward neural network f ( w ; x ) , when considered as a function ofits weight parameters w , is arbitrarily close to a linear function in the ball B ( w 0 , R ) given constant
Revised: Speciﬁcally, we prove that a feedforward neural network f ( w ; x ) , when considered as a function ofits weight parameters w , is arbitrarily close to a linear function in the ball B ( w 0 , R ) given constant

Original: R > 0 , where w 0 is randomly initialized, as long as the width of the network is sufﬁciently large.
Revised: R > 0 , where w 0 is randomly initialized, as long as the width of the network is sufﬁciently large.

Original: First, we make the following assumptions on the input x and the activation functions:
Revised: First, we make the following assumptions on the input x and the activation functions:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: The input is uniformly upper bounded, i.e., k x k 1  C x for some constant C x > 0 .
Revised: The input is uniformly upper bounded, i.e., k x k 1  C x for some constant C x > 0 .

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: All the activation functions  ( · ) are twice differentiable, and there exist constants 0 ,  1 ,  2 > 0 such that, for all activation functions, |  (0) |   0 and the following Lipschitzcontinuity and smoothness conditions are satisﬁed
Revised: All the activation functions  ( · ) are twice differentiable, and there exist constants 0 ,  1 ,  2 > 0 such that, for all activation functions, |  (0) |   0 and the following Lipschitzcontinuity and smoothness conditions are satisﬁed

Original: [Equation]
Revised: [Equation]

Original: for any z 1 , z 2 2 R .
Revised: for any z 1 , z 2 2 R .

Original: We note that the above two assumptions are very common in literature.
Revised: We note that the above two assumptions are very common in literature.

Original: Although ReLU does notsatisfy Assumption 3.2 due to non-differentiability at point 0 , we believe our main claims still hold as
Revised: Although ReLU does notsatisfy Assumption 3.2 due to non-differentiability at point 0 , we believe our main claims still hold as

Original: ReLU can be approximated arbitrarily closely by some differentiable function which satisﬁes ourassumption.
Revised: ReLU can be approximated arbitrarily closely by some differentiable function which satisﬁes ourassumption.

Original: Remark 3.3 .
Revised: Remark 3.3 .

Original: By assuming all the activation functions are twice differentiable, it is not hard to seethat the feedforward neural network i.e., Eq.
Revised: By assuming all the activation functions are twice differentiable, it is not hard to seethat the feedforward neural network i.e., Eq.

Original: (7) is also twice differentiable.
Revised: (7) is also twice differentiable.

Original: Taylor expansion.
Revised: Taylor expansion.

Original: To study the linearity of a general feedforward neural network, we consider its
Revised: To study the linearity of a general feedforward neural network, we consider its

Original: Taylor expansion with second order Lagrange remainder term.
Revised: Taylor expansion with second order Lagrange remainder term.

Original: Given a point w 0 , we can write thenetwork function f ( w ) (omitting the input argument for simplicity) as
Revised: Given a point w 0 , we can write thenetwork function f ( w ) (omitting the input argument for simplicity) as

Original: [Equation]
Revised: [Equation]

Original: where ⇠ is a point on the line segment between w 0 and w .
Revised: where ⇠ is a point on the line segment between w 0 and w .

Original: Above, f lin ( w ) is a linear function and
Revised: Above, f lin ( w ) is a linear function and

Original: R ( w ) is the Lagrange remainder term.
Revised: R ( w ) is the Lagrange remainder term.

Original: Here we assume the output dimension of the network functionis one.

Original: The same analysis can be applied to multiple outputs (see Corollary C.1).

Original: In the rest of the section, we will show that in a ball B ( w 0 , R ) of any constant radius R > 0 ,
Revised: In the rest of the section, we will show that in a ball B ( w 0 , R ) of any constant radius R > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where m is the network width (see Deﬁnition 2.7).
Revised: where m is the network width (see Deﬁnition 2.7).

Original: Hence, f ( w ) can be arbitrarily close to its linearapproximation f lin ( w ) with sufﬁciently large m .
Revised: Hence, f ( w ) can be arbitrarily close to its linearapproximation f lin ( w ) with sufﬁciently large m .

Revised: Note that in Eq.

Revised: (9), we consider a single output ofthe network function.

Revised: The same analysis can be applied to multiple outputs (see Corollary C.1).

Original: Remark 3.4 .
Revised: Remark 3.4 .

Original: For a general function, the remainder term R ( w ) is not expected to vanish at a ﬁnitedistance from w 0 .
Revised: For a general function, the remainder term R ( w ) is not expected to vanish at a ﬁnitedistance from w 0 .

Original: Hence, the transition to linearity in the ball B ( w 0 , R ) is a non-trivial property.
Revised: Hence, the transition to linearity in the ball B ( w 0 , R ) is a non-trivial property.

Original: Onthe other hand, the radius R can be set to be large enough to contain the whole optimization path of
Revised: Onthe other hand, the radius R can be set to be large enough to contain the whole optimization path of

Original: GD/SGD for various types of wide neural networks (see [13, 28], also indicated in [6, 5, 27, 12]).
Revised: GD/SGD for various types of wide neural networks (see [12, 25], also indicated in [6, 5, 24, 11]).

Original: In
Revised: In

Original: Section 4, we will see that such a ball is also large enough to cover the whole optimization path of
Revised: Section 4, we will see that such a ball is also large enough to cover the whole optimization path of

Original: GD/SGD for the general feedforward neural networks.
Revised: GD/SGD for the general feedforward neural networks.

Original: Hence, to study the optimization dynamics ofwide feedforward neural networks, this ball is large enough.
Revised: Hence, to study the optimization dynamics ofwide feedforward neural networks, this ball is large enough.

Original: To prove Eq.
Revised: To prove Eq.

Original: (10), we make an assumption on the width m :
Revised: (10), we make an assumption on the width m :

Original: Assumption 3.5.
Revised: Assumption 3.5.

Original: The maximum in-degree of any neuron is at most polynomial in the network widthm :
Revised: The maximum in-degree of any neuron is at most polynomial in the network widthm :

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a constant.
Revised: where c > 0 is a constant.

Original: This assumption puts a constraint on the neurons with large in-degrees such that the in-degreescannot be super-polynomially large compared to m .
Revised: This assumption puts a constraint on the neurons with large in-degrees such that the in-degreescannot be super-polynomially large compared to m .

Original: A natural question is whether this constraint isnecessary, for example, do our main results still hold in cases some in-degrees are exponentially largein m ?
Revised: A natural question is whether this constraint isnecessary, for example, do our main results still hold in cases some in-degrees are exponentially largein m ?

Original: While we believe the answer is positive, we need this assumption to apply the proof techniques.
Revised: While we believe the answer is positive, we need this assumption to apply the proof techniques.

Original: Speciﬁcally, we apply the tail bound for the norm of matrix Gaussian series [23], where there isa dimension factor equivalent to the number of weight parameters.
Revised: Speciﬁcally, we apply the tail bound for the norm of matrix Gaussian series [20], where there isa dimension factor equivalent to the number of weight parameters.

Original: Thus an exponentially largedimension factor would result in useless bounds.
Revised: Thus an exponentially largedimension factor would result in useless bounds.

Original: It is still an open question whether the dimensionfactor in the bound can be removed or moderated (see the discussion after Theorem 4.1.1 in [23]).
Revised: It is still an open question whether the dimensionfactor in the bound can be removed or moderated (see the discussion after Theorem 4.1.1 in [20]).

Original: With these assumptions, we are ready to present our main result:
Revised: With these assumptions, we are ready to present our main result:

Original: Theorem 3.6 (Scaling of the Hessian norm) .
Revised: Theorem 3.6 (Scaling of the Hessian norm) .

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given aﬁxed R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the randominitialization w 0 , each output neuron f k of a feedforward neural network satisﬁes
Revised: Given aﬁxed R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the randominitialization w 0 , each output neuron f k of a feedforward neural network satisﬁes

Original: [Equation]
Revised: [Equation]

Original: This theorem states that the Hessian matrix, as the second derivative with respect to weight parametersw , of any output neuron can be arbitrarily small, if the network width is sufﬁcient large.
Revised: This theorem states that the Hessian matrix, as the second derivative with respect to weight parametersw , of any output neuron can be arbitrarily small, if the network width is sufﬁcient large.

Original: Note that Eq.
Revised: Note that Eq.

Original: (11) holds for all w 2 B ( w 0 , R ) with high probability over the random initializationw 0 .
Revised: (11) holds for all w 2 B ( w 0 , R ) with high probability over the random initializationw 0 .

Original: The basic idea is that, the spectral norm of Hessian can be bounded at the center of the ball, i.e.,w 0 , though probability bounds due to the randomness of w 0 .
Revised: The basic idea is that, the spectral norm of Hessian can be bounded at the center of the ball, i.e.,w 0 , though probability bounds due to the randomness of w 0 .

Original: For all other points w 2 B ( w 0 , R ) , thedistance k w  w 0 k , being no greater than R , controls k H ( w )  H ( w 0 )
Revised: For all other points w 2 B ( w 0 , R ) , thedistance k w  w 0 k , being no greater than R , controls k H ( w )  H ( w 0 )

Original: k such that it is no larger than 226 the order of k H ( w 0 ) k , hence k H ( w ) k keeps the same order.
Revised: k such that it is no larger than 224 the order of k H ( w 0 ) k , hence k H ( w ) k keeps the same order.

Original: See the proof details in Subsection 3.1.
Revised: See the proof details in Subsection 3.1.

Original: Using the Taylor expansion Eq.
Revised: Using the Taylor expansion Eq.

Original: (9), we can bound the Lagrange remainder and have transition tolinearity of the network:
Revised: (9), we can bound the Lagrange remainder and have transition tolinearity of the network:

Original: Corollary 3.7 (Transition to linearity) .
Revised: Corollary 3.7 (Transition to linearity) .

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given a ﬁxed
Revised: Given a ﬁxed

Original: R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the randominitialization w 0 , each f k will be closely approximated by a linear model:
Revised: R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the randominitialization w 0 , each f k will be closely approximated by a linear model:

Original: [Equation]
Revised: [Equation]

Original: For feedforward neural networks with multiple output neurons, the property of transition to linearityholds with high probability, if the number of output neurons is bounded, i.e., d ` = O (1) .
Revised: For feedforward neural networks with multiple output neurons, the property of transition to linearityholds with high probability, if the number of output neurons is bounded, i.e., d ` = O (1) .

Original: See theresult in Appendix C.
Revised: See theresult in Appendix C.

Original: Furthermore, as deﬁned in Deﬁnition 2.2, each pre-activation, as a function of all related weightparameters and inputs, can be viewed as a feedforward neural network.
Revised: Furthermore, as deﬁned in Deﬁnition 2.2, each pre-activation, as a function of all related weightparameters and inputs, can be viewed as a feedforward neural network.

Original: Therefore, we can apply thesame techniques used for Theorem 3.6 to show that each pre-activation can transition to linearity:
Revised: Therefore, we can apply thesame techniques used for Theorem 3.6 to show that each pre-activation can transition to linearity:

Original: Theorem 3.8.
Revised: Theorem 3.8.

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given a ﬁxed radius R > 0 , for allw 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initliazation of w 0 ,any pre-activation in a feedforward neural network i.e., ˜ f
Revised: Given a ﬁxed radius R > 0 , for allw 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initliazation of w 0 ,any pre-activation in a feedforward neural network i.e., ˜ f

Original: ( ` ) k ( w ) satisﬁes
Revised: ( ` ) k ( w ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Remark 3.9 .
Revised: Remark 3.9 .

Original: Note that pre-activations in the input layer i.e., the input data and in the ﬁrst layer areconstant and linear functions respectively, hence the spectral norm of their Hessian is zero.
Revised: Note that pre-activations in the input layer i.e., the input data and in the ﬁrst layer areconstant and linear functions respectively, hence the spectral norm of their Hessian is zero.

Original: Non-linear activation on output neurons breaks transition to linearity.
Revised: Non-linear activation on output neurons breaks transition to linearity.

Original: In the above discussions,the transition to linearity of networks are under the assumption of identity activation function onevery output neuron.
Revised: In the above discussions,the transition to linearity of networks are under the assumption of identity activation function onevery output neuron.

Original: In fact, the activation function on output neurons is critical to the linearityof neural networks.
Revised: In fact, the activation function on output neurons is critical to the linearityof neural networks.

Original: Simply, composing a non-linear function with a linear function will break thelinearity.
Revised: Simply, composing a non-linear function with a linear function will break thelinearity.

Original: Consistently, as shown in [13] for FCNs, with non-linear activation function on the output,transition to linearity does not hold any more.
Revised: Consistently, as shown in [12] for FCNs, with non-linear activation function on the output,transition to linearity does not hold any more.

Original: 3.1 Proof sketch of Theorem 3.6
Revised: 3.1 Proof sketch of Theorem 3.6

Original: By Lemma I.1, the spectral norm of H f k can be bounded by the summation of the spectral norm ofall the Hessian blocks, i.e., k H f k k  
Revised: By Lemma I.1, the spectral norm of H f k can be bounded by the summation of the spectral norm ofall the Hessian blocks, i.e., k H f k k  P

Original: [Equation]
Revised: [Equation]

Original:  k , where H [Equation] [Equation] ` 
Revised: 1 , ` 2 k H ( ` 1 , ` 2 ) f k k , where H ( ` 1 , ` 2 ) f k :=

Original: [Equation]
Revised: [Equation]

Revised: 2 f k @ w (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  w ( ` 2) .
Revised: @ w ( ` 2) .

Original: Therefore,it sufﬁces to bound the spectral norm of each block.
Revised: Therefore,it sufﬁces to bound the spectral norm of each block.

Original: Without lose of generality, we consider the blockwith 1  ` 1  ` 2  L .
Revised: Without lose of generality, we consider the blockwith 1  ` 1  ` 2  L .

Original: By the chain rule of derivatives, we can write the Hessian block into:
Revised: By the chain rule of derivatives, we can write the Hessian block into:

Original: [Equation]
Revised: [Equation]

Original: For each G L, ` 0 k , since f ( ` 0 ) i =  ⇣ ˜ f
Revised: For each G L, ` 0 k , since f ( ` 0 ) i =  ⇣ ˜ f

Original: ( ` 0 ) i ⌘ , again by the chain rule of derivatives, we have
Revised: ( ` 0 ) i ⌘ , again by the chain rule of derivatives, we have

Original: G L, ` 0 k = 
Revised: G L, ` 0 k = d `

Revised: X i = @ 2 ˜ f

Revised: ( ` 0 ) i @ w (

Original: [Equation]
Revised: [Equation]

Revised: w ( ` 2 ) @

Revised: f k @ ˜ f

Revised: ( ` 0 ) i + q m ( L

Revised: [Equation]

Revised: k

Revised: X i :

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: 2 F S ( L ) k ⇣ w ( L ) k ⌘ i ⇣ ˜ f

Revised: ( ` 0 ) i ⌘ @ ˜ f

Revised: ( ` 0 ) i @

Revised: w ( ` 1 ) @

Original: ˜ f
Revised: ˜ f

Original: ( ` 0 ) i @
Revised: ( ` 0 ) i @

Revised: w ( ` 2 ) !

Revised: T = q m ( L ) k

Original:  w ( ` 1 ) @
Revised: L  X r = ` X s : f ( r ) s 2 F S ( L ) k ⇣ w ( L ) k ⌘ s ⇣ ˜ f ( r ) s ⌘

Original:  ˜ f
Revised: G r, ` 0 s + q m ( L ) k X i : f ( ` 0 ) i 2 F S ( L ) k ⇣ w ( L ) k ⌘ i ⇣ ˜ f ( ` 0 ) i ⌘ @ ˜ f

Original: ( ` 0 ) i @
Revised: ( ` 0 ) i @

Original: w ( ` 2 ) !
Revised: w ( ` 1 ) @

Original: T
Revised: ˜ f

Original: [Equation] [Equation] [Equation]
Revised: ( ` 0 ) i @

Original: [Equation]
Revised: w ( ` 2 ) !

Original: T [Equation] where F S ( L ) k :=
Revised: T , where F S ( L ) k :=

Original: { f : f 2 f S ( L ) k } .
Revised: { f : f 2 f S ( L ) k } .

Original: The ﬁrst quantity on the RHS of the above equation,
Revised: The ﬁrst quantity on the RHS of the above equation,

Original: P ⇣ w ( L ) k ⌘ s ⇣ ˜ f
Revised: P ⇣ w ( L ) k ⌘ s ⇣ ˜ f

Original: ( r ) s ⌘
Revised: ( r ) s ⌘

Original: G r, ` 0 s , is a matrix Gaussianseries with respect to random variables w ( L ) k , conditioned on ﬁxed ⇣ ˜ f
Revised: G r, ` 0 s , is a matrix Gaussianseries with respect to random variables w ( L ) k , conditioned on ﬁxed ⇣ ˜ f

Original: ( r ) s ⌘
Revised: ( r ) s ⌘

Original: G r, ` 0 s for all s such thatf ( r ) s 2 F S ( ` ) k .
Revised: G r, ` 0 s for all s such thatf ( r ) s 2 F S ( ` ) k .

Original: We apply the tail bound for matrix Gaussian series, Theorem 4.1.1 from [23], to boundthis quantity.
Revised: We apply the tail bound for matrix Gaussian series, Theorem 4.1.1 from [20], to boundthis quantity.

Original: To that end, we need to bound its matrix variance, which sufﬁces to bound the spectralnorm of
Revised: To that end, we need to bound its matrix variance, which sufﬁces to bound the spectralnorm of

Original: P s G r, ` 0 s since  0 ( · ) is assumed to be uniformly bounded by Assumption 3.2.
Revised: P s G r, ` 0 s since  0 ( · ) is assumed to be uniformly bounded by Assumption 3.2.

Original: Note thatthere is a recursive relation that the norm bound of G L, ` 0 k depends on the norm bound of G r, ` 0 s whichappears in the matrix variance.
Revised: Note thatthere is a recursive relation that the norm bound of G L, ` 0 k depends on the norm bound of G r, ` 0 s whichappears in the matrix variance.

Original: Therefore, we can recursively apply the argument to bound each G .
Revised: Therefore, we can recursively apply the argument to bound each G .

Original: Similarly, the second quantity on the RHS of the above equation is also a matrix Gaussian series withrespect to w ( L ) k , conditioned on ﬁxed ⇣[Equation] ✓ [Equation]
Revised: Similarly, the second quantity on the RHS of the above equation is also a matrix Gaussian series withrespect to w ( L ) k , conditioned on ﬁxed ⇣

Original: @ w (

Original: [Equation]
Revised: [Equation]

Original: ◆
Revised: ◆

Original: T for all i such that f ( ` 0 ) i 2 F S ( L ) k .
Revised: T for all i such that f ( ` 0 ) i 2 F S ( L ) k .

Original: We use Lemma B.1 to bound its matrix variance, hence the matrix Gaussian series can be bounded.
Revised: We use Lemma B.1 to bound its matrix variance, hence the matrix Gaussian series can be bounded.

Original: Note that such tail bound does not scale with the largest in-degree of the networks, since the in-degreeof f k , i.e., m ( L ) k , will be cancelled out with the scaling factor 1 / q m ( L ) k in the bound of matrix variance.
Revised: Note that such tail bound does not scale with the largest in-degree of the networks, since the in-degreeof f k , i.e., m ( L ) k , will be cancelled out with the scaling factor 1 / q m ( L ) k in the bound of matrix variance.

Original: See the complete proof in Appendix B.
Revised: See the complete proof in Appendix B.

Original:  Relation to optimization
Revised:  Relation to optimization

Original: While transition to linearity is a signiﬁcant and surprising property of wide networks in its ownright, it also plays an important role in building the optimization theory of wide feedforward neuralnetworks.
Revised: While transition to linearity is a signiﬁcant and surprising property of wide networks in its ownright, it also plays an important role in building the optimization theory of wide feedforward neuralnetworks.

Original: Speciﬁcally, transition to linearity provides a path toward showing that the correspondingloss function satisﬁes the PL ⇤ condition in a ball of a certain radius, which is sufﬁcient for exponentialconvergence of optimization to a global minimum by gradient descent or SGD [13].
Revised: Speciﬁcally, transition to linearity provides a path toward showing that the correspondingloss function satisﬁes the PL ⇤ condition in a ball of a certain radius, which is sufﬁcient for exponentialconvergence of optimization to a global minimum by gradient descent or SGD [12].

Original: Consider a supervised learning task.
Revised: Consider a supervised learning task.

Original: Given training inputs and labels { ( x i , y i ) } ni =1 , we use GD/SGDto minimize the square loss:
Revised: Given training inputs and labels { ( x i , y i ) } ni =1 , we use GD/SGDto minimize the square loss:

Original: [Equation]
Revised: [Equation]

Original: where f ( w ; · ) is a feedforward neural network.
Revised: where f ( w ; · ) is a feedforward neural network.

Original: The loss L ( w ) is said to satisfy µ -PL ⇤ condition, a variant of the well-known Polyak- Ł ojasiewiczcondition [19, 15], at point w , if
Revised: The loss L ( w ) is said to satisfy µ -PL ⇤ condition, a variant of the well-known Polyak- Ł ojasiewiczcondition [18, 14], at point w , if

Original: [Equation]
Revised: [Equation]

Original: Satisfaction of this µ -PL ⇤ condition in a ball B ( w 0 , R ) with R
Revised: Satisfaction of this µ -PL ⇤ condition in a ball B ( w 0 , R ) with R

Original: =
Revised: =

Original: O
Revised: O

Original: (1 /µ ) around the starting pointw 0 of GD/SGD guarantees a fast converge of the algorithm to a global minimum in this ball [13].
Revised: (1 /µ ) around the starting pointw 0 of GD/SGD guarantees a fast converge of the algorithm to a global minimum in this ball [12].

Original: In the following, we use the transition to linearity of wide feedforward networks to establish the PL ⇤condition for L ( w ) .
Revised: In the following, we use the transition to linearity of wide feedforward networks to establish the PL ⇤condition for L ( w ) .

Original: Taking derivative on Eq.
Revised: Taking derivative on Eq.

Original: (14), we get
Revised: (14), we get

Original: [Equation]
Revised: [Equation]

Original: (15) where matrix K ( w ) , with elements K i,j ( w ) =
Revised: (15) where matrix K ( w ) , with elements K i,j ( w ) =

Original: r w f ( w ; x i ) T r w f ( w ; x j ) for i, j 2 [ n ] , is called
Revised: r w f ( w ; x i ) T r w f ( w ; x j ) for i, j 2 [ n ] , is called

Original: Neural Tangent Kernel (NTK) [10], and  min ( · ) denotes the smallest eigenvalue of a matrix.
Revised: Neural Tangent Kernel (NTK) [9], and  min ( · ) denotes the smallest eigenvalue of a matrix.

Original: Notethat, by deﬁnition, the NTK matrix is always positive semi-deﬁnite, i.e.,  min ( K ( w ))  0 .
Revised: Note that,by deﬁnition, the NTK matrix is always positive semi-deﬁnite, i.e.,  min ( K ( w ))  0 .

Original: Directly by the deﬁnition of PL ⇤ condition, at a given point w , if  min ( K ( w )) is strictly positive,then the loss function L ( w ) satisﬁes PL ⇤ condition.
Revised: Directly by the deﬁnition of PL ⇤ condition, at a given point w , if  min ( K ( w )) is strictly positive,then the loss function L ( w ) satisﬁes PL ⇤ condition.

Original: To establish convergence of GD/SGD, it is sufﬁcient to verify that PL ⇤ condition is satisﬁed in a ball
Revised: To establish convergence of GD/SGD, it is sufﬁcient to verify that PL ⇤ condition is satisﬁed in a ball

Original: B ( w 0 , R ) with R = O (1 /µ ) .
Revised: B ( w 0 , R ) with R = O (1 /µ ) .

Original: Assuming that  min ( K ( w 0 )) is bounded away from zero, transition tolinearity extends the satisfaction of the PL ⇤ condition from one point w 0 to all points in B ( w 0 , R ) .
Revised: Assuming that  min ( K ( w 0 )) is bounded away from zero, transition tolinearity extends the satisfaction of the PL ⇤ condition from one point w 0 to all points in B ( w 0 , R ) .

Original: PL ⇤ condition at w 0 .
Revised: PL ⇤ condition at w 0 .

Original: For certain neural networks, e.g., FCNs, CNNs and ResNets, strict positiveness of  min ( K ( w 0 )) can be shown, see for example, [6, 5].
Revised: For certain neural networks, e.g., FCNs, CNNs and ResNets, strict positiveness of  min ( K ( w 0 )) can be shown, see for example, [6, 5].

Original: We expect same holds more generally,in the case of general feedforward neural networks.
Revised: We expect same holds more generally,in the case of general feedforward neural networks.

Original: Here we show that  min ( K ( w 0 )) can be bounded from 0 for one data point under certain additional assumptions.
Revised: Here we show that  min ( K ( w 0 )) can be bounded from 0 for one data point under certain additional assumptions.

Original: Since there is only one data point, min ( K ( w 0 ))
Revised: Since there is only one data point, min ( K ( w 0 ))

Original: =
Revised: =

Original: K ( w 0 )
Revised: K ( w 0 )

Original: =
Revised: =

Original: kr w f ( w 0 ) k 2 .
Revised: kr w f ( w 0 ) k 2 .

Original: We also assume the following on activation functionsand the input.
Revised: We also assume the following on activation functionsand the input.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: The input x satisﬁes x ⇠ N ( I d 0 , 0) .
Revised: The input x satisﬁes x ⇠ N ( I d 0 , 0) .

Original: Assumption 4.2.
Revised: Assumption 4.2.

Original: The activation function is homogeneous, i.e.  ( ` ) i ( az ) = a r  ( ` ) i ( z ) , r > 0 for anyconstant a .
Revised: The activation function is homogeneous, i.e.  ( ` ) i ( az ) = a r  ( ` ) i ( z ) , r > 0 for anyconstant a .

Original: And inf ` 2 [ L  1] ,i 

Revised: And inf ` 2 [ L  1] ,i

Original: [ d ` ] E z ⇠ N (0 , 1) h  ( ` ) i ( z )i =
Revised: [ d ` ] E z ⇠ N (0 , 1) h  ( ` ) i ( z )i =

Original: C  > 0 .
Revised: C  > 0 .

Original: Remark 4.3 .
Revised: Remark 4.3 .

Original: Here for simplicity we assume the activation functions are homogeneous with the samer .
Revised: Here for simplicity we assume the activation functions are homogeneous with the samer .

Original: It is not hard to extend the result to the case that each activation function has different r .
Revised: It is not hard to extend the result to the case that each activation function has different r .

Original: Proposition 4.4.
Revised: Proposition 4.4.

Original: With Assumption 4.1 and 4.2, we have for any
Revised: With Assumption 4.1 and 4.2, we have for any

Original: k 2 [ d ` ] ,
Revised: k 2 [ d ` ] ,

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix E.
Revised: The proof can be found in Appendix E.

Original: The above proposition establishes a positive lower bound on kr w f ( w 0 ) k , hence also on 307  min ( K ( w 0 )) .
Revised: The above proposition establishes a positive lower bound on kr w f ( w 0 ) k , hence also on 305  min ( K ( w 0 )) .

Original: Using Eq.
Revised: Using Eq.

Original: (15), we get that the loss function L ( w ) satisﬁes PL ⇤ at w 0 .
Revised: (15), we get that the loss function L ( w ) satisﬁes PL ⇤ at w 0 .

Original: Extending PL ⇤ condition to B ( w 0 , R ) .
Revised: Extending PL ⇤ condition to B ( w 0 , R ) .

Original: Now we use transition to linearity to extend the satisfactionof PL ⇤ condition to the ball B ( w 0 , R ) .
Revised: Now we use transition to linearity to extend the satisfactionof PL ⇤ condition to the ball B ( w 0 , R ) .

Original: In Theorem 3.6, we see that, a feedforward neural networkf ( w ) transitions to linearity, i.e., k H f ( w ) k
Revised: In Theorem 3.6, we see that, a feedforward neural networkf ( w ) transitions to linearity, i.e., k H f ( w ) k

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: O
Revised: O

Original: (1 / p m ) in this ball.
Revised: (1 / p m ) in this ball.

Original: An immediate consequence isthat, for any w 2 B ( w 0 , R ) ,
Revised: An immediate consequence isthat, for any w 2 B ( w 0 , R ) ,

Original: [Equation]
Revised: [Equation]

Original: Since  min ( K ( w 0 )) is bound from 0 and k H f ( w ) k can be arbitrarily small as long as m is largeenough, we have  min ( K ( w )) is lower bounded from 0 in the whole ball.
Revised: Since  min ( K ( w 0 )) is bound from 0 and k H f ( w ) k can be arbitrarily small as long as m is largeenough, we have  min ( K ( w )) is lower bounded from 0 in the whole ball.

Original: Speciﬁcally, there is aµ > 0 such that
Revised: Speciﬁcally, there is aµ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Moreover, the radius R can be set to be O (1 /µ ) , while keeping the above inequality hold.
Revised: Moreover, the radius R can be set to be O (1 /µ ) , while keeping the above inequality hold.

Original: Then,applying the theory in [13], existence of global minima of L ( w ) and convergence of GD/SGD can beestablished.
Revised: Then,applying the theory in [12], existence of global minima of L ( w ) and convergence of GD/SGD can beestablished.

Original: For the case of multiple data points, extra techniques are needed to lower bound the minimumeigenvalue of the tangent kernel.
Revised: For the case of multiple data points, extra techniques are needed to lower bound the minimumeigenvalue of the tangent kernel.

Original: Since we focus more on the transition to linearity of feedforwardneural networks in this paper, we leave it as a future work.
Revised: Since we focus more on the transition to linearity of feedforwardneural networks in this paper, we leave it as a future work.

Original: Non-linear activation function on outputs and transition to linearity.
Revised: Non-linear activation function on outputs and transition to linearity.

Original: In this paper, we mainlydiscussed feedforward neural networks with linear activation function on output neurons.
Revised: In this paper, we mainlydiscussed feedforward neural networks with linear activation function on output neurons.

Original: In most ofthe literature also considers this setting [10, 17,18, 6, 5, 28, 27].
Revised: In most ofthe literature also considers this setting [9, 16, 17, 6, 5, 25, 24].

Original: In fact, as pointed out in [13] for
Revised: In fact, as pointed out in [12] for

Original: FCNs, while this linearity of activation function on the outputs is necessary for transition to linearity,it is not required for successful optimization.
Revised: FCNs, while this linearity of activation function on the outputs is necessary for transition to linearity,it is not required for successful optimization.

Original: Speciﬁcally, simply adding a nonlinear activationfunction on the output layer causes the Hessian norm to be O (1) , independently of the networkwidth.
Revised: Speciﬁcally, simply adding a nonlinear activationfunction on the output layer causes the Hessian norm to be O (1) , independently of the networkwidth.

Original: Thus transition to linearity does not occur.
Revised: Thus transition to linearity does not occur.

Original: However, the corresponding square loss can stillsatisfy the PL ⇤ condition and the existence of global minimuma and efﬁcient optimization can still beestablished.
Revised: However, the corresponding square loss can stillsatisfy the PL ⇤ condition and the existence of global minimuma and efﬁcient optimization can still beestablished.

Original:  Discussion and future directions
Revised:  Discussion and future directions

Original: In this work, we showed that transition to linearity arises in general feedforward neural networks witharbitrary DAG architectures, extending previous results for standard architectures [10, 12, 14].
Revised: In this work, we showed that transition to linearity arises in general feedforward neural networks witharbitrary DAG architectures, extending previous results for standard architectures [9, 11, 13].

Original: Fornon-feedforward networks, such as RNN, recent work [1] showed they also have a constant NTK.
Revised: Fornon-feedforward networks, such as RNN, recent work [1] showed they also have a constant NTK.

Original: Forthis reason, we expect transition to linearity also to occur for of non-feedforward networks.
Revised: Forthis reason, we expect transition to linearity also to occur for of non-feedforward networks.

Original: Anotherdirection of future work is better understanding of optimization that requires a more delicate analysisof the NTK at initialization.
Revised: Anotherdirection of future work is better understanding of optimization that requires a more delicate analysisof the NTK at initialization.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors (a)

Revised: Do the main claims made

Revised: in

Revised: the

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: abstract and introduction accurately reﬂect the paper’scontributions and scope

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes (b) Did you describe the limitations of your work?

Original: [Yes] This is a paper with theoreticalresults hence the assumptions constitute the core limitations.
Revised: [Yes] This is a paper with theoretical results hence the assumptions constitute the core limitations.

Original: We have the discussionfor each assumption.(c)
Revised: We have the discussion for each assumption. (c) Did you discuss any

Original: Did you discuss any potential negative societal impacts of your work?
Revised: potential negative societal impacts of your work?

Original: [No] The work
Revised: [No] The work

Original: presents mathematical results related to deep learning and is not expected to have anynegative societal impact within its scope.
Revised: presents mathematical results related to deep learning and is not expected to have any negative societal impact within its scope.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption
Revised: [Yes] See Assumption

Original: 3.1, 3.2 and 3.5.
Revised: 3.1, 3.2 and 3.5.(b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [Yes] We give a proof sketchin Section 3.1 and the complete proofs can be found in appendix.3.
Revised: [Yes] We give a proof sketch in Section 3.1 and the complete proofs can be found in appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [N/A]

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A

Original: [Table]