Original: Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Revised: Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

Original: Abstract
Revised: Abstract

Original: In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their “width” approaches inﬁnity.
Revised: In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their “width” approaches inﬁnity.

Original: The width of these general networks is characterized by the minimum indegree of their neurons, except for the input and ﬁrst layers.
Revised: The width of these general networks is characterized by the minimum indegree of their neurons, except for the input and ﬁrst layers.

Original: Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the
Revised: Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the

Original: Neural Tangent Kernel for standard architectures
Revised: Neural Tangent Kernel for standard architectures

Original:  Introduction
Revised:  Introduction

Original: A remarkable property of wide neural networks, ﬁrst discovered in [10] in terms of the constancy of the Neural Tangent Kernel along the optimization path, is that they transition to linearity (using the terminology from [14]), i.e., are approximately linear in a ball of a ﬁxed radius.
Revised: A remarkable property of wide neural networks, ﬁrst discovered in [10] in terms of the constancy of the Neural Tangent Kernel along the optimization path, is that they transition to linearity (using the terminology from [14]), i.e., are approximately linear in a ball of a ﬁxed radius.

Original: There has been an extensive study of this phenomenon for different types of standard neural networks architectures including fully-connected neural networks (FCNs), convolutional neural networks (CNNs), ResNets [12, 4, 3, 7].
Revised: There has been an extensive study of this phenomenon for different types of standard neural networks architectures including fully-connected neural networks (FCNs), convolutional neural networks (CNNs), ResNets [12, 4, 3, 7].

Original: Yet the scope of the transition to linearity and the underlying mathematical structure has not been made completely clear.
Revised: Yet the scope of the transition to linearity and the underlying mathematical structure has not been made completely clear.

Original: In this paper, we show that the property of transition to linearity holds for a much broader class of neural networks – feedforward neural networks .
Revised: In this paper, we show that the property of transition to linearity holds for a much broader class of neural networks – feedforward neural networks .

Original: The architecture of a feedforward neural network can generically be described by a DAG [25, 24, 16]: the vertices and the edges correspond to the neurons and the trainable weight parameters of a neural network, respectively.
Revised: The architecture of a feedforward neural network can generically be described by a DAG [25, 24, 16]: the vertices and the edges correspond to the neurons and the trainable weight parameters of a neural network, respectively.

Original: This DAG structure includes standard network architectures e.g., FCNs, CNNs, ResNets, as well as DenseNets [9], whose property of transition to linearity has not been studied in literature.
Revised: This DAG structure includes standard network architectures e.g., FCNs, CNNs, ResNets, as well as DenseNets [9], whose property of transition to linearity has not been studied in literature.

Original: This generalization shows that the transition to linearity, or the constant Neural Tangent Kernel, does not depend on the speciﬁc designs of the networks, and is a more fundamental and universal property.
Revised: This generalization shows that the transition to linearity, or the constant Neural Tangent Kernel, does not depend on the speciﬁc designs of the networks, and is a more fundamental and universal property.

Original: We deﬁne the width of a feedforward neural network as the minimum in-degree of all neurons except for the input and ﬁrst layers, which is a natural generalization of the the minimum number of neurons in hidden layers which is how the width is deﬁned for standard architectures.
Revised: We deﬁne the width of a feedforward neural network as the minimum in-degree of all neurons except for the input and ﬁrst layers, which is a natural generalization of the the minimum number of neurons in hidden layers which is how the width is deﬁned for standard architectures.

Original: For a feedforward neural network, we show it transitions to linearity if its width goes to inﬁnity as long as the in-degrees of individual neurons are bounded by a polynomial of the network width.
Revised: For a feedforward neural network, we show it transitions to linearity if its width goes to inﬁnity as long as the in-degrees of individual neurons are bounded by a polynomial of the network width.

Original: Speciﬁcally, we control the deviation of the network function from its linear approximation by the spectral norm of the Hessian of the network function, which, as we show vanishes in a ball of ﬁxed radius, in the inﬁnite width limit.
Revised: Speciﬁcally, we control the deviation of the network function from its linear approximation by the spectral norm of the Hessian of the network function, which, as we show vanishes in a ball of ﬁxed radius, in the inﬁnite width limit.

Original: Interestingly, we observe that not only the output neurons, but any pre-activated neuron in the hidden layers of a feedforward neural network can be regarded as a function with respect to its parameters, which will also transition to linearity as the width goes to inﬁnity.
Revised: Interestingly, we observe that not only the output neurons, but any pre-activated neuron in the hidden layers of a feedforward neural network can be regarded as a function with respect to its parameters, which will also transition to linearity as the width goes to inﬁnity.

Original: The key technical difﬁculty is that all existing analyses for transition to linearity or constant NTK do not apply to this general DAG setting.
Revised: The key technical difﬁculty is that all existing analyses for transition to linearity or constant NTK do not apply to this general DAG setting.

Original: Speciﬁcally, those analyses assume in-degrees of neurons are either the same or proportional to each other up to a constant ratio [5, 12, 3, 26, 14, 2].
Revised: Speciﬁcally, those analyses assume in-degrees of neurons are either the same or proportional to each other up to a constant ratio [5, 12, 3, 26, 14, 2].

Original: However, the general DAG setting allows different scales of neuron in-degrees, for example, the largest in-degree can be polynomially large in the smallest in-degree.
Revised: However, the general DAG setting allows different scales of neuron in-degrees, for example, the largest in-degree can be polynomially large in the smallest in-degree.

Original: In such scenarios, the (2 , 2 , 1) -norm in [14] and the norm of parameter change in [5, 12] scales with the maximum of in-degrees which causes a trivial bound on the NTK change.
Revised: In such scenarios, the (2 , 2 , 1) -norm in [14] and the norm of parameter change in [5, 12] scales with the maximum of in-degrees which causes a trivial bound on the NTK change.

Original: Instead, we introduce a different set of tools based on the tail bound for the norm of matrix Gaussian series [22].
Revised: Instead, we introduce a different set of tools based on the tail bound for the norm of matrix Gaussian series [22].

Original: Speciﬁcally, we show that the Hessian of the network function takes the form of matrix Gaussian series, whose matrix variance relies on the Hessian of connected neurons.
Revised: Speciﬁcally, we show that the Hessian of the network function takes the form of matrix Gaussian series, whose matrix variance relies on the Hessian of connected neurons.

Original: Therefore, we reconcile the in-degree difference by building a recursive relation between the Hessian of neurons, which exactly cancels out the in-degree with the scaling factor.
Revised: Therefore, we reconcile the in-degree difference by building a recursive relation between the Hessian of neurons, which exactly cancels out the in-degree with the scaling factor.

Original: Transition to linearity helps understand the training dynamics of wide neural networks and plays an important role in developing the optimization theory for them, as has been shown for certain particular wide neural networks [6, 5, 4, 12, 27, 26].
Revised: Transition to linearity helps understand the training dynamics of wide neural networks and plays an important role in developing the optimization theory for them, as has been shown for certain particular wide neural networks [6, 5, 4, 12, 27, 26].

Original: While transition to linearity is not a necessary condition for successful optimization, it provides a powerful tool for analyzing optimization for many different architectures.
Revised: While transition to linearity is not a necessary condition for successful optimization, it provides a powerful tool for analyzing optimization for many different architectures.

Original: Speciﬁcally, transition to linearity in a ball of sufﬁcient radius combined with a lower bound on the norm of the gradient at its center is sufﬁcient to demonstrate the PL ⇤ condition [13] (a version of the Polyak- Ł ojasiewicz condition [19, 15]) which ensures convergence of optimization.
Revised: Speciﬁcally, transition to linearity in a ball of sufﬁcient radius combined with a lower bound on the norm of the gradient at its center is sufﬁcient to demonstrate the PL ⇤ condition [13] (a version of the Polyak- Ł ojasiewicz condition [19, 15]) which ensures convergence of optimization.

Original: We discuss this connection and provide one such lower bound in Section 4.
Revised: We discuss this connection and provide one such lower bound in Section 4.

Original: Summary of contributions.
Revised: Summary of contributions.

Original: We show the phenomenon of transition to linearity in general feedforward neural networks corresponding to a DAG with large in-degree.
Revised: We show the phenomenon of transition to linearity in general feedforward neural networks corresponding to a DAG with large in-degree.

Original: Speciﬁcally, under the assumption that the maximum in-degree of its neurons is bounded by a polynomial of the width m (the minimum in-degree), we prove that the spectral norm of the Hessian of a feedforward neural network is bounded by
Revised: Speciﬁcally, under the assumption that the maximum in-degree of its neurons is bounded by a polynomial of the width m (the minimum in-degree), we prove that the spectral norm of the Hessian of a feedforward neural network is bounded by

Original: ˜ O
Revised: ˜ O

Original: (1 / p m ) in an O (1) ball.
Revised: (1 / p m ) in an O (1) ball.

Original: Our results generalize the existing literature on the linearity of wide feedforward neural networks.
Revised: Our results generalize the existing literature on the linearity of wide feedforward neural networks.

Original: We discuss connections to optimization.
Revised: We discuss connections to optimization.

Original: Under additional assumptions we show that the norm of the gradient of a feedforward neural network is bounded away from zero at initialization.
Revised: Under additional assumptions we show that the norm of the gradient of a feedforward neural network is bounded away from zero at initialization.

Original: Together with the Hessian bound this implies convergence of gradient descent for the loss function.
Revised: Together with the Hessian bound this implies convergence of gradient descent for the loss function.

Original: 1.1 Notations
Revised: 1.1 Notations

Original: We use bold lowercase letters, e.g., w , to denote vectors, capital letters, e.g., A , to denote matrices, and bold capital letters, e.g., H , to denote higher order tensors or matrix tuples.
Revised: We use bold lowercase letters, e.g., w , to denote vectors, capital letters, e.g., A , to denote matrices, and bold capital letters, e.g., H , to denote higher order tensors or matrix tuples.

Original: For a matrix A , we use A [ i, :] to denote its i -th row and A [: ,i ] to denote its j -th column.
Revised: For a matrix A , we use A [ i, :] to denote its i -th row and A [: ,i ] to denote its j -th column.

Original: We use r w f ( w 0 ) to denote the gradient of f with respect to w at w 0 , and H f ( w ) to denote Hessian matrix (second derivative) of f with respect to w .
Revised: We use r w f ( w 0 ) to denote the gradient of f with respect to w at w 0 , and H f ( w ) to denote Hessian matrix (second derivative) of f with respect to w .

Original: For vectors, we use k · k to denote Euclidean norm.
Revised: For vectors, we use k · k to denote Euclidean norm.

Original: For matrices, we use k · k to denote spectral norm and k · k F to denote Frobenius norm.
Revised: For matrices, we use k · k to denote spectral norm and k · k F to denote Frobenius norm.

Original: We use k · k 1 to denote function L 1 norm.
Revised: We use k · k 1 to denote function L 1 norm.

Original: For a set S , we use |S| to denote the cardinality of the set.
Revised: For a set S , we use |S| to denote the cardinality of the set.

Original: For n > 0 , [ n ] denotes the set { 1 , 2 , ..., n } .
Revised: For n > 0 , [ n ] denotes the set { 1 , 2 , ..., n } .

Original: We use big-
Revised: We use big-

Original: O notation to hide constant factors, and use big-
Revised: O notation to hide constant factors, and use big-

Original: ˜ O notation to additionally hide logarithmic factors.
Revised: ˜ O notation to additionally hide logarithmic factors.

Original: In this paper, the argument of O / ˜ O ( · ) is always with respect to the network width.
Revised: In this paper, the argument of O / ˜ O ( · ) is always with respect to the network width.

Original: Given a vector w and a constant R > 0 , we deﬁne a Euclidean ball B ( w , R ) as:
Revised: Given a vector w and a constant R > 0 , we deﬁne a Euclidean ball B ( w , R ) as:

Original: [Equation]
Revised: [Equation]

Original:  Neural networks with acyclic graph architecture
Revised:  Neural networks with acyclic graph architecture

Original: In this section, we provide a deﬁnition and notation for general feedforward neural networks with an arbitrary DAG structure.
Revised: In this section, we provide a deﬁnition and notation for general feedforward neural networks with an arbitrary DAG structure.

Original: This deﬁnition includes standard feedforward neural network architectures, such as FCNs, DenseNet and CNNs.
Revised: This deﬁnition includes standard feedforward neural network architectures, such as FCNs, DenseNet and CNNs.

Original: 2.1 Deﬁning feedforward neural networks
Revised: 2.1 Deﬁning feedforward neural networks

Original: Graph Structure.
Revised: Graph Structure.

Original: Consider a directed acyclic graph (DAG)
Revised: Consider a directed acyclic graph (DAG)

Original: G =
Revised: G =

Original: ( V , E ) , where V and E denote the sets of vertices and edges, respectively.
Revised: ( V , E ) , where V and E denote the sets of vertices and edges, respectively.

Original: See the left panel of Figure 1, for an illustrative example.
Revised: See the left panel of Figure 1, for an illustrative example.

Original: [Figure]
Revised: [Figure]

Original: For a directed edge e 2 E , we may also use the notation e
Revised: For a directed edge e 2 E , we may also use the notation e

Original: = ( v 1 , v 2 ) to explicitly write out the start vertex v 1 and end vertex v 2 .
Revised: = ( v 1 , v 2 ) to explicitly write out the start vertex v 1 and end vertex v 2 .

Original: For a vertex v 2 V , we denote its in-degree, in ( v ) , by the number of incoming edges (edges that end with it): in ( v ) =
Revised: For a vertex v 2 V , we denote its in-degree, in ( v ) , by the number of incoming edges (edges that end with it): in ( v ) =

Original: |S in ( v ) | , with S in ( v ) :
Revised: |S in ( v ) | , with S in ( v ) :

Original: = { u 2 V : ( u, v ) 2 E} .
Revised: = { u 2 V : ( u, v ) 2 E} .

Original: Similarly, for a vertex v 2 V , we denote its out-degree out ( v ) by the number of outgoing edges (edges that start from it):
Revised: Similarly, for a vertex v 2 V , we denote its out-degree out ( v ) by the number of outgoing edges (edges that start from it):

Original: [Equation]
Revised: [Equation]

Original: We call the set of vertices with zero in-degrees input : V input = { v 2 V : in ( v ) = 0 } , and the set of vertices with zero out-degrees output
Revised: We call the set of vertices with zero in-degrees input : V input = { v 2 V : in ( v ) = 0 } , and the set of vertices with zero out-degrees output

Original: V output = { v 2 V : out ( v ) = 0 } .
Revised: V output = { v 2 V : out ( v ) = 0 } .

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: For each vertex v 2 V\V input , its distance p ( v ) , to the input V input , is deﬁned to be the maximum length of all paths that start from a vertex within
Revised: For each vertex v 2 V\V input , its distance p ( v ) , to the input V input , is deﬁned to be the maximum length of all paths that start from a vertex within

Original: V input and end with v .
Revised: V input and end with v .

Original: It is easy to check that p ( v )
Revised: It is easy to check that p ( v )

Original: =
Revised: =

Original: if v 2 V input .
Revised: if v 2 V input .

Original: Feedforward neural network.
Revised: Feedforward neural network.

Original: Based on a given DAG architecture, we deﬁne the feedforward neural network.
Revised: Based on a given DAG architecture, we deﬁne the feedforward neural network.

Original: Each individual vertex corresponds to a neuron additionally equipped with a scalar function (also called activation function).
Revised: Each individual vertex corresponds to a neuron additionally equipped with a scalar function (also called activation function).

Original: Each edge is associated with a real-valued weight, a trainable parameter.
Revised: Each edge is associated with a real-valued weight, a trainable parameter.

Original: Each neuron is deﬁned as a function of the weight parameters and the adjacent neurons connected by its incoming edges.
Revised: Each neuron is deﬁned as a function of the weight parameters and the adjacent neurons connected by its incoming edges.

Original: The feedforward neural network is considered as the output neurons, corresponding to the output V output , of all weight parameters and input neurons which correspond to the input V input .
Revised: The feedforward neural network is considered as the output neurons, corresponding to the output V output , of all weight parameters and input neurons which correspond to the input V input .

Original: Formally, we deﬁne the feedforward neural network as follows.
Revised: Formally, we deﬁne the feedforward neural network as follows.

Original: Deﬁnition 2.2 (Feedforward neural network) .
Revised: Deﬁnition 2.2 (Feedforward neural network) .

Original: Consider a DAG G =
Revised: Consider a DAG G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: For each vertex v 2 V\V input , we associate it with an activation function  v ( · ) : R !
Revised: For each vertex v 2 V\V input , we associate it with an activation function  v ( · ) : R !

Original: R and each of its incoming edges e
Revised: R and each of its incoming edges e

Original: = ( u, v ) 2 E with a weight variable w e = w ( u,v ) .
Revised: = ( u, v ) 2 E with a weight variable w e = w ( u,v ) .

Original: Then we deﬁne the following functions:
Revised: Then we deﬁne the following functions:

Original: [Equation]
Revised: [Equation]

Original: When v 2 V input , f v is preﬁxed as the input data, and we denote f input :=
Revised: When v 2 V input , f v is preﬁxed as the input data, and we denote f input :=

Original: { f v : v 2 V input } .
Revised: { f v : v 2 V input } .

Original: For v / 2 V input , we call f v neurons and ˜ f v pre-activations .
Revised: For v / 2 V input , we call f v neurons and ˜ f v pre-activations .

Original: With necessary composition of functions, each f v , and ˜ f v , can be regarded as a function of all related weight variables and inputs f input .
Revised: With necessary composition of functions, each f v , and ˜ f v , can be regarded as a function of all related weight variables and inputs f input .

Original: The feedforward neural network is deﬁned to be the function corresponding to the output V output :
Revised: The feedforward neural network is deﬁned to be the function corresponding to the output V output :

Original: [Equation]
Revised: [Equation]

Original: where W := { w e : e 2 E} denotes the set of all the weight variables.
Revised: where W := { w e : e 2 E} denotes the set of all the weight variables.

Original: Remark 2.3 .
Revised: Remark 2.3 .

Original: The validity of the deﬁnition is guaranteed by the fact that the DAG is acyclic.
Revised: The validity of the deﬁnition is guaranteed by the fact that the DAG is acyclic.

Original: It makes sure that the dependence of each function f v on other neurons can pass all the way down to the input f input , through Eq.
Revised: It makes sure that the dependence of each function f v on other neurons can pass all the way down to the input f input , through Eq.

Original: Remark 2.4 .
Revised: Remark 2.4 .

Original: For v 2 V input
Revised: For v 2 V input

Original: S
Revised: S

Original: V output , we use the identity function I ( · ) as the activation functions.
Revised: V output , we use the identity function I ( · ) as the activation functions.

Original: Weight initialization and inputs.
Revised: Weight initialization and inputs.

Original: Each weight parameter w e 2 W is initialized i.i.d.
Revised: Each weight parameter w e 2 W is initialized i.i.d.

Original: following the standard normal distribution i.e., N (0 , 1) .
Revised: following the standard normal distribution i.e., N (0 , 1) .

Original: The inputs are considered given, usually determined by datasets.
Revised: The inputs are considered given, usually determined by datasets.

Original: Under this initialization, we introduce the scaling factor 1 / p in ( v ) in Eq.
Revised: Under this initialization, we introduce the scaling factor 1 / p in ( v ) in Eq.

Original: (5) to control the value of neurons to be of order O (1) .
Revised: (5) to control the value of neurons to be of order O (1) .

Original: Note that this initialization is an extension of the NTK initialization [10], which was deﬁned for FCNs therein.
Revised: Note that this initialization is an extension of the NTK initialization [10], which was deﬁned for FCNs therein.

Original: Generality of DAG architecture.
Revised: Generality of DAG architecture.

Original: Including FCNs and DenseNets [9] as special examples, the class of feedforward neural networks allows much more choices of architectures, for example, neural networks with randomly dropped edges.
Revised: Including FCNs and DenseNets [9] as special examples, the class of feedforward neural networks allows much more choices of architectures, for example, neural networks with randomly dropped edges.

Original: Please see detailed discussions about these speciﬁc examples in Appendix A.
Revised: Please see detailed discussions about these speciﬁc examples in Appendix A.

Original: We note that our deﬁnition of feedforward neural networks does not directly include networks with nont-trainable skip connections, e.g., ResNets, and networks with shared weights, e.g.,
Revised: We note that our deﬁnition of feedforward neural networks does not directly include networks with nont-trainable skip connections, e.g., ResNets, and networks with shared weights, e.g.,

Original: CNNs.
Revised: CNNs.

Original: However, with a slight modiﬁcation of the analysis, the property of transition to linearity still holds.
Revised: However, with a slight modiﬁcation of the analysis, the property of transition to linearity still holds.

Original: See the detailed discussion in Appendix D and E.
Revised: See the detailed discussion in Appendix D and E.

Original: 2.2 Organizing feedforward networks into layers
Revised: 2.2 Organizing feedforward networks into layers

Original: The architecture of the feedforward neural network is determined by the DAG G .
Revised: The architecture of the feedforward neural network is determined by the DAG G .

Original: The complex structures of DAGs often lead to complicated neural networks, which are hard to analyze.
Revised: The complex structures of DAGs often lead to complicated neural networks, which are hard to analyze.

Original: For the ease of analysis, we organize the neurons of the feedforward neural network into layers , which are sets of neurons.
Revised: For the ease of analysis, we organize the neurons of the feedforward neural network into layers , which are sets of neurons.

Original: Deﬁnition 2.5 (Layers) .
Revised: Deﬁnition 2.5 (Layers) .

Original: Consider a feedforward neural network f and its corresponding graph structure G .
Revised: Consider a feedforward neural network f and its corresponding graph structure G .

Original: A layer of the network is deﬁned to be the set of neurons which have the same distance p to the inputs.
Revised: A layer of the network is deﬁned to be the set of neurons which have the same distance p to the inputs.

Original: Speciﬁcally, the ` -th layer, denoted by f ( ` ) , is
Revised: Speciﬁcally, the ` -th layer, denoted by f ( ` ) , is

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that the layers are mutually exclusive, and the layer index ` is labeled from 0 to ` , where L + 1 is the total number of layers in the network.
Revised: It is easy to see that the layers are mutually exclusive, and the layer index ` is labeled from 0 to ` , where L + 1 is the total number of layers in the network.

Original: As p ( v ) = 

Revised: As p ( v ) =

Original: if and only if v 2 V input , the 0 -th layer f (0) is exactly the input layer f input .
Revised: if and only if v 2 V input , the 0 -th layer f (0) is exactly the input layer f input .

Original: The right panel of Figure 1 provides an illustrative example of the layer structures.
Revised: The right panel of Figure 1 provides an illustrative example of the layer structures.

Original: In general, the output neurons f output (deﬁned in Eq.
Revised: In general, the output neurons f output (deﬁned in Eq.

Original: (3)) do not have to be in the same layer.
Revised: (3)) do not have to be in the same layer.

Original: For the convenience of presentation and analysis, we assume that all the output neurons are in the last layer, i.e., layer ` , which is the case for most of commonly used neural networks, e.g., FCNs and CNNs.
Revised: For the convenience of presentation and analysis, we assume that all the output neurons are in the last layer, i.e., layer ` , which is the case for most of commonly used neural networks, e.g., FCNs and CNNs.

Original: Indeed, our analysis applies to every output neuron (see Theorem 3.8), even if they are not in the same layer.
Revised: Indeed, our analysis applies to every output neuron (see Theorem 3.8), even if they are not in the same layer.

Original: With the notion of network layers, we rewrite the neuron functions Eq.
Revised: With the notion of network layers, we rewrite the neuron functions Eq.

Original: (2), as well as related notations, to reﬂect the layer information.
Revised: (2), as well as related notations, to reﬂect the layer information.

Original: For ` -layer, ` = 0 , 1 , · · · , L , we denote the total number of neurons as d ` , and rewrite the layer function f ( ` ) into a form of vector-valued function
Revised: For ` -layer, ` = 0 , 1 , · · · , L , we denote the total number of neurons as d ` , and rewrite the layer function f ( ` ) into a form of vector-valued function

Original: [Equation]
Revised: [Equation]

Original: where we use f ( ` ) i with index i
Revised: where we use f ( ` ) i with index i

Original: = 1 , 2 , · · · , d ` to denote each individual neuron.
Revised: = 1 , 2 , · · · , d ` to denote each individual neuron.

Original: Correspondingly, we denote its vertex as v ( ` ) i , and S ( ` ) i :=
Revised: Correspondingly, we denote its vertex as v ( ` ) i , and S ( ` ) i :=

Original: S in ( v ( ` ) i ) .
Revised: S in ( v ( ` ) i ) .

Original: Hence, the in-degree in ( v ( ` ) i ) , denoted as m ( ` ) i here, is equivalent to the cardinality of the set S ( ` ) i .
Revised: Hence, the in-degree in ( v ( ` ) i ) , denoted as m ( ` ) i here, is equivalent to the cardinality of the set S ( ` ) i .

Original: Remark 2.6 .
Revised: Remark 2.6 .

Original: Note that m ( ` ) i , with the superscript ` , denotes an in-degree, i.e., the number of neurons that serve as direct inputs to the current neuron in ` -th layer.
Revised: Note that m ( ` ) i , with the superscript ` , denotes an in-degree, i.e., the number of neurons that serve as direct inputs to the current neuron in ` -th layer.

Original: In the context of FCNs, m ( ` ) i is equivalent to the size of its previous layer, i.e., ( `  1) -th layer, and is often denoted as m ( `  1) in literature.
Revised: In the context of FCNs, m ( ` ) i is equivalent to the size of its previous layer, i.e., ( `  1) -th layer, and is often denoted as m ( `  1) in literature.

Original: To write the summation in Eq.
Revised: To write the summation in Eq.

Original: (2) as a matrix multiplication, we further introduce the following two vectors: (a), f S ( ` ) i represents the vector that consists of neuron components f v with v 2 S ( ` ) i ; (b), w ( ` ) i represents the vector that consists of weight parameters w ( u,v ( ` ) i ) with u 2 S ( ` ) i .
Revised: (2) as a matrix multiplication, we further introduce the following two vectors: (a), f S ( ` ) i represents the vector that consists of neuron components f v with v 2 S ( ` ) i ; (b), w ( ` ) i represents the vector that consists of weight parameters w ( u,v ( ` ) i ) with u 2 S ( ` ) i .

Original: Note that both vectors f S ( ` ) i and w ( ` ) i have the same dimension m ( ` ) i .
Revised: Note that both vectors f S ( ` ) i and w ( ` ) i have the same dimension m ( ` ) i .

Original: With the above notation, the neuron functions Eq.
Revised: With the above notation, the neuron functions Eq.

Original: (2) can be equivalently rewritten as:
Revised: (2) can be equivalently rewritten as:

Original: [Equation]
Revised: [Equation]

Original: For any ` 2 [ L ] , we denote the weight parameters corresponding to all incoming edges toward neurons at layer ` by
Revised: For any ` 2 [ L ] , we denote the weight parameters corresponding to all incoming edges toward neurons at layer ` by

Original: [Equation]
Revised: [Equation]

Original: Through the way we deﬁne the feedforward neural network, the output of the neural network is a function of all the weight parameters and the input data, hence we denote it by
Revised: Through the way we deﬁne the feedforward neural network, the output of the neural network is a function of all the weight parameters and the input data, hence we denote it by

Original: [Equation]
Revised: [Equation]

Original: where w is the collection of all the weight parameters, i.e., w :=
Revised: where w is the collection of all the weight parameters, i.e., w :=

Original: [Equation]
Revised: [Equation]

Original: R P
Revised: R P

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: With all the notations, for a feedforward neural network, we formally deﬁne the width of it:
Revised: With all the notations, for a feedforward neural network, we formally deﬁne the width of it:

Original: Deﬁnition 2.7 (Network width) .
Revised: Deﬁnition 2.7 (Network width) .

Original: The width m of a feedforward neural network is the minimum in-degree of all the neurons except those in the input and ﬁrst layers:
Revised: The width m of a feedforward neural network is the minimum in-degree of all the neurons except those in the input and ﬁrst layers:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.8 .
Revised: Remark 2.8 .

Original: Note that, the network width m is determined by the in-degrees of neurons except for the input and ﬁrst layers, and not necessarily relates the number of neurons in hidden layers.
Revised: Note that, the network width m is determined by the in-degrees of neurons except for the input and ﬁrst layers, and not necessarily relates the number of neurons in hidden layers.

Original: But for certain architectures e.g., FCNs, these two coincide that the minimum in-degree after the ﬁrst layer is the same as the minimum hidden layer size.
Revised: But for certain architectures e.g., FCNs, these two coincide that the minimum in-degree after the ﬁrst layer is the same as the minimum hidden layer size.

Original: We say a feedforward neural network is wide if its width m is large enough.
Revised: We say a feedforward neural network is wide if its width m is large enough.

Original: In this paper, we consider wide feedforward neural networks with a ﬁxed number of layers.
Revised: In this paper, we consider wide feedforward neural networks with a ﬁxed number of layers.

Original:  Transition to linearity of feedforward neural networks
Revised:  Transition to linearity of feedforward neural networks

Original: In this section, we show that the feedforward neural networks exhibit the phenomenon of transition to linearity, which was previously observed in speciﬁc types of neural networks.
Revised: In this section, we show that the feedforward neural networks exhibit the phenomenon of transition to linearity, which was previously observed in speciﬁc types of neural networks.

Original: Speciﬁcally, we prove that a feedforward neural network f ( w ; x ) , when considered as a function of its weight parameters w , is arbitrarily close to a linear function in the ball B ( w 0 , R ) given constant
Revised: Speciﬁcally, we prove that a feedforward neural network f ( w ; x ) , when considered as a function of its weight parameters w , is arbitrarily close to a linear function in the ball B ( w 0 , R ) given constant

Original: R > 0 , where w 0 is randomly initialized, as long as the width of the network is sufﬁciently large.
Revised: R > 0 , where w 0 is randomly initialized, as long as the width of the network is sufﬁciently large.

Original: First, we make the following assumptions on the input x and the activation functions:
Revised: First, we make the following assumptions on the input x and the activation functions:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: The input is uniformly upper bounded, i.e., k x k 1  C x for some constant C x > 0 .
Revised: The input is uniformly upper bounded, i.e., k x k 1  C x for some constant C x > 0 .

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: All the activation functions  ( · ) are twice differentiable, and there exist constants  0 ,  1 ,  2 > 

Revised: All the activation functions  ( · ) are twice differentiable, and there exist constants  0 ,  1 ,  2 >

Original: such that, for all activation functions, |  (0) |   0 and the following Lipschitz continuity and smoothness conditions are satisﬁed
Revised: such that, for all activation functions, |  (0) |   0 and the following Lipschitz continuity and smoothness conditions are satisﬁed

Original: [Equation]
Revised: [Equation]

Original: We note that the above two assumptions are very common in literature.
Revised: We note that the above two assumptions are very common in literature.

Original: Although ReLU does not satisfy Assumption 3.2 due to non-differentiability at point 0 , we believe our main claims still hold as
Revised: Although ReLU does not satisfy Assumption 3.2 due to non-differentiability at point 0 , we believe our main claims still hold as

Original: ReLU can be approximated arbitrarily closely by some differentiable function which satisﬁes our assumption.
Revised: ReLU can be approximated arbitrarily closely by some differentiable function which satisﬁes our assumption.

Original: Remark 3.3 .
Revised: Remark 3.3 .

Original: By assuming all the activation functions are twice differentiable, it is not hard to see that the feedforward neural network i.e., Eq.
Revised: By assuming all the activation functions are twice differentiable, it is not hard to see that the feedforward neural network i.e., Eq.

Original: (7) is also twice differentiable.
Revised: (7) is also twice differentiable.

Original: Taylor expansion.
Revised: Taylor expansion.

Original: To study the linearity of a general feedforward neural network, we consider its
Revised: To study the linearity of a general feedforward neural network, we consider its

Original: Taylor expansion with second order Lagrange remainder term.
Revised: Taylor expansion with second order Lagrange remainder term.

Original: Given a point w 0 , we can write the network function f ( w ) (omitting the input argument for simplicity) as
Revised: Given a point w 0 , we can write the network function f ( w ) (omitting the input argument for simplicity) as

Original: [Equation]
Revised: [Equation]

Original: 9) where ⇠ is a point on the line segment between w 0 and w .
Revised: 9) where ⇠ is a point on the line segment between w 0 and w .

Original: Above, f lin ( w ) is a linear function and
Revised: Above, f lin ( w ) is a linear function and

Original: R ( w ) is the Lagrange remainder term.
Revised: R ( w ) is the Lagrange remainder term.

Original: In the rest of the section, we will show that in a ball B ( w 0 , R ) of any constant radius R > 0 ,
Revised: In the rest of the section, we will show that in a ball B ( w 0 , R ) of any constant radius R > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where m is the network width (see Deﬁnition 2.7).
Revised: where m is the network width (see Deﬁnition 2.7).

Original: Hence, f ( w ) can be arbitrarily close to its linear approximation f lin ( w ) with sufﬁciently large m .
Revised: Hence, f ( w ) can be arbitrarily close to its linear approximation f lin ( w ) with sufﬁciently large m .

Original: Note that in Eq.
Revised: Note that in Eq.

Original: (9), we consider a single output of the network function.
Revised: (9), we consider a single output of the network function.

Original: The same analysis can be applied to multiple outputs (see Corollary C.1).
Revised: The same analysis can be applied to multiple outputs (see Corollary C.1).

Original: Remark 3.4 .
Revised: Remark 3.4 .

Original: For a general function, the remainder term R ( w ) is not expected to vanish at a ﬁnite distance from w 0 .
Revised: For a general function, the remainder term R ( w ) is not expected to vanish at a ﬁnite distance from w 0 .

Original: Hence, the transition to linearity in the ball B ( w 0 , R ) is a non-trivial property.
Revised: Hence, the transition to linearity in the ball B ( w 0 , R ) is a non-trivial property.

Original: On the other hand, the radius R can be set to be large enough to contain the whole optimization path of
Revised: On the other hand, the radius R can be set to be large enough to contain the whole optimization path of

Original: GD/SGD for various types of wide neural networks (see [13, 27], also indicated in [6, 5, 26, 12]).
Revised: GD/SGD for various types of wide neural networks (see [13, 27], also indicated in [6, 5, 26, 12]).

Original: In
Revised: In

Original: Section 4, we will see that such a ball is also large enough to cover the whole optimization path of
Revised: Section 4, we will see that such a ball is also large enough to cover the whole optimization path of

Original: GD/SGD for the general feedforward neural networks.
Revised: GD/SGD for the general feedforward neural networks.

Original: Hence, to study the optimization dynamics of wide feedforward neural networks, this ball is large enough.
Revised: Hence, to study the optimization dynamics of wide feedforward neural networks, this ball is large enough.

Original: To prove Eq.
Revised: To prove Eq.

Original: (10), we make an assumption on the width m :
Revised: (10), we make an assumption on the width m :

Original: Assumption 3.5.
Revised: Assumption 3.5.

Original: The maximum in-degree of any neuron is at most polynomial in the network width m :
Revised: The maximum in-degree of any neuron is at most polynomial in the network width m :

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a constant.
Revised: where c > 0 is a constant.

Original: This assumption puts a constraint on the neurons with large in-degrees such that the in-degrees cannot be super-polynomially large compared to m .
Revised: This assumption puts a constraint on the neurons with large in-degrees such that the in-degrees cannot be super-polynomially large compared to m .

Original: A natural question is whether this constraint is necessary, for example, do our main results still hold in cases some in-degrees are exponentially large in m ?
Revised: A natural question is whether this constraint is necessary, for example, do our main results still hold in cases some in-degrees are exponentially large in m ?

Original: While we believe the answer is positive, we need this assumption to apply the proof techniques.
Revised: While we believe the answer is positive, we need this assumption to apply the proof techniques.

Original: Speciﬁcally, we apply the tail bound for the norm of matrix Gaussian series [22], where there is a dimension factor equivalent to the number of weight parameters.
Revised: Speciﬁcally, we apply the tail bound for the norm of matrix Gaussian series [22], where there is a dimension factor equivalent to the number of weight parameters.

Original: Thus an exponentially large dimension factor would result in useless bounds.
Revised: Thus an exponentially large dimension factor would result in useless bounds.

Original: It is still an open question whether the dimension factor in the bound can be removed or moderated (see the discussion after Theorem 4.1.1 in [22]).
Revised: It is still an open question whether the dimension factor in the bound can be removed or moderated (see the discussion after Theorem 4.1.1 in [22]).

Original: With these assumptions, we are ready to present our main result:
Revised: With these assumptions, we are ready to present our main result:

Original: Theorem 3.6 (Scaling of the Hessian norm) .
Revised: Theorem 3.6 (Scaling of the Hessian norm) .

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given a ﬁxed R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initialization w 0 , each output neuron f k of a feedforward neural network satisﬁes
Revised: Given a ﬁxed R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initialization w 0 , each output neuron f k of a feedforward neural network satisﬁes

Original: [Equation]
Revised: [Equation]

Original: This theorem states that the Hessian matrix, as the second derivative with respect to weight parameters w , of any output neuron can be arbitrarily small, if the network width is sufﬁcient large.
Revised: This theorem states that the Hessian matrix, as the second derivative with respect to weight parameters w , of any output neuron can be arbitrarily small, if the network width is sufﬁcient large.

Original: Note that Eq.
Revised: Note that Eq.

Original: (11) holds for all w 2 B ( w 0 , R ) with high probability over the random initialization w 0 .
Revised: (11) holds for all w 2 B ( w 0 , R ) with high probability over the random initialization w 0 .

Original: The basic idea is that, the spectral norm of Hessian can be bounded at the center of the ball, i.e., w 0 , though probability bounds due to the randomness of w 0 .
Revised: The basic idea is that, the spectral norm of Hessian can be bounded at the center of the ball, i.e., w 0 , though probability bounds due to the randomness of w 0 .

Original: For all other points w 2 B ( w 0 , R ) , the distance k w  w 0 k , being no greater than R , controls k H ( w )  H ( w 0 )
Revised: For all other points w 2 B ( w 0 , R ) , the distance k w  w 0 k , being no greater than R , controls k H ( w )  H ( w 0 )

Original: k such that it is no larger than the order of k H ( w 0 ) k , hence k H ( w ) k keeps the same order.
Revised: k such that it is no larger than the order of k H ( w 0 ) k , hence k H ( w ) k keeps the same order.

Original: See the proof details in Subsection 3.1.
Revised: See the proof details in Subsection 3.1.

Original: Using the Taylor expansion Eq.
Revised: Using the Taylor expansion Eq.

Original: (9), we can bound the Lagrange remainder and have transition to linearity of the network:
Revised: (9), we can bound the Lagrange remainder and have transition to linearity of the network:

Original: Corollary 3.7 (Transition to linearity) .
Revised: Corollary 3.7 (Transition to linearity) .

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given a ﬁxed
Revised: Given a ﬁxed

Original: R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initialization w 0 , each f k will be closely approximated by a linear model:
Revised: R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initialization w 0 , each f k will be closely approximated by a linear model:

Original: [Equation]
Revised: [Equation]

Original: For feedforward neural networks with multiple output neurons, the property of transition to linearity holds with high probability, if the number of output neurons is bounded, i.e., d ` = O (1) .
Revised: For feedforward neural networks with multiple output neurons, the property of transition to linearity holds with high probability, if the number of output neurons is bounded, i.e., d ` = O (1) .

Original: See the result in Appendix C.
Revised: See the result in Appendix C.

Original: Furthermore, as deﬁned in Deﬁnition 2.2, each pre-activation, as a function of all related weight parameters and inputs, can be viewed as a feedforward neural network.
Revised: Furthermore, as deﬁned in Deﬁnition 2.2, each pre-activation, as a function of all related weight parameters and inputs, can be viewed as a feedforward neural network.

Original: Therefore, we can apply the same techniques used for Theorem 3.6 to show that each pre-activation can transition to linearity:
Revised: Therefore, we can apply the same techniques used for Theorem 3.6 to show that each pre-activation can transition to linearity:

Original: Theorem 3.8.
Revised: Theorem 3.8.

Original: Suppose Assumption 3.1, 3.2 and 3.5 hold.
Revised: Suppose Assumption 3.1, 3.2 and 3.5 hold.

Original: Given a ﬁxed radius R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initliazation of w 0 , any pre-activation in a feedforward neural network i.e., ˜ f
Revised: Given a ﬁxed radius R > 0 , for all w 2 B ( w 0 , R ) , with probability at least 1  exp(  ⌦ (log 2 m )) over the random initliazation of w 0 , any pre-activation in a feedforward neural network i.e., ˜ f

Original: ( ` ) k ( w ) satisﬁes
Revised: ( ` ) k ( w ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Remark 3.9 .
Revised: Remark 3.9 .

Original: Note that pre-activations in the input layer i.e., the input data and in the ﬁrst layer are constant and linear functions respectively, hence the spectral norm of their Hessian is zero.
Revised: Note that pre-activations in the input layer i.e., the input data and in the ﬁrst layer are constant and linear functions respectively, hence the spectral norm of their Hessian is zero.

Original: Experimental veriﬁcation.
Revised: Experimental veriﬁcation.

Original: To verify our theoretical result on the scaling of the Hessian norm, i.e.,
Revised: To verify our theoretical result on the scaling of the Hessian norm, i.e.,

Original: Theorem 3.6, we train a DAG network built from a 3-layer DenseNet with each weight removed i.i.d.
Revised: Theorem 3.6, we train a DAG network built from a 3-layer DenseNet with each weight removed i.i.d.

Original: with probability 1 / 2 , on 10 data points of CIFAR-2 (2-class subset of CIFAR-10 [11]) using GD.
Revised: with probability 1 / 2 , on 10 data points of CIFAR-2 (2-class subset of CIFAR-10 [11]) using GD.

Original: We compute the maximum relative change of the tangent kernel (deﬁnition is deferred to Eq.
Revised: We compute the maximum relative change of the tangent kernel (deﬁnition is deferred to Eq.

Original: (17)) during training, i.e., max t k K t  K 0 k / k K 0 k to simulate the scaling of the spectral norm of the Hessian.
Revised: (17)) during training, i.e., max t k K t  K 0 k / k K 0 k to simulate the scaling of the spectral norm of the Hessian.

Original: We observe the convergence of loss for all widths { 2 , 2 2 , ..., 2 12 } , and the scaling of the Hessian follows close to the theoretical prediction of ⇥ (1 / p m ) .
Revised: We observe the convergence of loss for all widths { 2 , 2 2 , ..., 2 12 } , and the scaling of the Hessian follows close to the theoretical prediction of ⇥ (1 / p m ) .

Original: See Figure 2.
Revised: See Figure 2.

Original: Non-linear activation on output neurons breaks transition to linearity.
Revised: Non-linear activation on output neurons breaks transition to linearity.

Original: In the above discussions, the transition to linearity of networks are under the assumption of identity activation function on every output neuron.
Revised: In the above discussions, the transition to linearity of networks are under the assumption of identity activation function on every output neuron.

Original: In fact, the activation function on output neurons is critical to the linearity of neural networks.
Revised: In fact, the activation function on output neurons is critical to the linearity of neural networks.

Original: Simply, composing a non-linear function with a linear function will break the linearity.
Revised: Simply, composing a non-linear function with a linear function will break the linearity.

Original: Consistently, as shown in [13] for FCNs, with non-linear activation function on the output, transition to linearity does not hold any more.
Revised: Consistently, as shown in [13] for FCNs, with non-linear activation function on the output, transition to linearity does not hold any more.

Original: “Bottleneck neurons” do not necessarily break transition to linearity.
Revised: “Bottleneck neurons” do not necessarily break transition to linearity.

Original: We have seen that if all neurons have sufﬁciently large in-degree, the network will transition to linearity.
Revised: We have seen that if all neurons have sufﬁciently large in-degree, the network will transition to linearity.

Original: Does transition to linearity still hold, if neurons with small in-degree exist?
Revised: Does transition to linearity still hold, if neurons with small in-degree exist?

Original: We call neurons with small in-degree bottleneck neurons , as their in-degrees are smaller than rest of neurons hence forming “bottlenecks”.
Revised: We call neurons with small in-degree bottleneck neurons , as their in-degrees are smaller than rest of neurons hence forming “bottlenecks”.

Original: As we show in Appendix E, existence of these bottleneck neurons does not break the transition to linearity, as long as the number of such neurons is signiﬁcantly smaller than that of non-bottleneck neurons.
Revised: As we show in Appendix E, existence of these bottleneck neurons does not break the transition to linearity, as long as the number of such neurons is signiﬁcantly smaller than that of non-bottleneck neurons.

Original: Figure 3 shows an example with two bottleneck neurons, whose in-degree is 1 .
Revised: Figure 3 shows an example with two bottleneck neurons, whose in-degree is 1 .

Original: This network still transitions to linearity as the number of neurons goes to inﬁnity.
Revised: This network still transitions to linearity as the number of neurons goes to inﬁnity.

Original: 3.1 Proof sketch of Theorem 3. 
Revised: 3.1 Proof sketch of Theorem 3.

Original: By Lemma K.1, the spectral norm of H f k can be bounded by the summation of the spectral norm of all the Hessian blocks, i.e., k H f k k 
Revised: By Lemma K.1, the spectral norm of H f k can be bounded by the summation of the spectral norm of all the Hessian blocks, i.e., k H f k k 

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 k H (
Revised: 2 k H (

Original: [Equation]
Revised: [Equation]

Original: 1 , ` 2 ) f k k , where H ( ` 1 , ` 2 ) f k :=
Revised: 1 , ` 2 ) f k k , where H ( ` 1 , ` 2 ) f k :=

Original: @ 2 f k @ w ( ` 1) @
Revised: @ 2 f k @ w ( ` 1) @

Original: w ( ` 2) .
Revised: w ( ` 2) .

Original: Therefore, it sufﬁces to bound the spectral norm of each block.
Revised: Therefore, it sufﬁces to bound the spectral norm of each block.

Original: Without lose of generality, we consider the block with 1  ` 1  ` 2  L .
Revised: Without lose of generality, we consider the block with 1  ` 1  ` 2  L .

Original: By the chain rule of derivatives, we can write the Hessian block into:
Revised: By the chain rule of derivatives, we can write the Hessian block into:

Original: [Equation]
Revised: [Equation]

Original: For each G L, ` 0 k , since f ( ` 0 ) i =  ⇣ ˜ f
Revised: For each G L, ` 0 k , since f ( ` 0 ) i =  ⇣ ˜ f

Original: ( ` 0 ) i ⌘ , again by the chain rule of derivatives, we have
Revised: ( ` 0 ) i ⌘ , again by the chain rule of derivatives, we have

Original: G L, ` 0 k =
Revised: G L, ` 0 k =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (14) where F S ( L ) k :=
Revised: (14) where F S ( L ) k :=

Original: { f : f 2 f S ( L ) k }
Revised: { f : f 2 f S ( L ) k }

Original: and id L,k ` 0 ,i :=
Revised: and id L,k ` 0 ,i :=

Original: { p : ⇣ f S ( L ) k ⌘ p = f
Revised: { p : ⇣ f S ( L ) k ⌘ p = f

Original: ( ` 0 ) i } .
Revised: ( ` 0 ) i } .

Original: The ﬁrst quantity on the RHS of the above equation,
Revised: The ﬁrst quantity on the RHS of the above equation,

Original: P ⇣ w ( L
Revised: P ⇣ w ( L

Original: [Equation]
Revised: [Equation]

Original: k ⌘ id L
Revised: k ⌘ id L

Original: [Equation]
Revised: [Equation]

Original: k `
Revised: k `

Original: [Equation]
Revised: [Equation]

Original: i 
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ⇣ ˜ f ( r )
Revised: ⇣ ˜ f ( r )

Original: [Equation]
Revised: [Equation]

Original: ⌘
Revised: ⌘

Original: G r, ` 0 i , is a matrix
Revised: G r, ` 0 i , is a matrix

Original: Gaussian series with respect to random variables w ( L ) k , conditioned on ﬁxed ⇣ ˜ f
Revised: Gaussian series with respect to random variables w ( L ) k , conditioned on ﬁxed ⇣ ˜ f

Original: ( r ) i ⌘
Revised: ( r ) i ⌘

Original: G r, ` 0 i for all i such that f ( r ) i 2 F S ( ` ) k .
Revised: G r, ` 0 i for all i such that f ( r ) i 2 F S ( ` ) k .

Original: We apply the tail bound for matrix Gaussian series, Theorem 4.1.1 from [22], to bound this quantity.
Revised: We apply the tail bound for matrix Gaussian series, Theorem 4.1.1 from [22], to bound this quantity.

Original: To that end, we need to bound its matrix variance, which sufﬁces to bound the spectral norm of
Revised: To that end, we need to bound its matrix variance, which sufﬁces to bound the spectral norm of

Original: P i G r, ` 0 i since  0 ( · ) is assumed to be uniformly bounded by Assumption 3.2.
Revised: P i G r, ` 0 i since  0 ( · ) is assumed to be uniformly bounded by Assumption 3.2.

Original: There is a recursive relation that the norm bound of G L, ` 0 k depends on the norm bound of G r, ` 0 i which appears in the matrix variance.
Revised: There is a recursive relation that the norm bound of G L, ` 0 k depends on the norm bound of G r, ` 0 i which appears in the matrix variance.

Original: Therefore, we can recursively apply the argument to bound each G .
Revised: Therefore, we can recursively apply the argument to bound each G .

Original: Similarly, the second quantity on the RHS of the above equation is also a matrix Gaussian series with respect to w ( L ) k , conditioned on ﬁxed 
Revised: Similarly, the second quantity on the RHS of the above equation is also a matrix Gaussian series with respect to w ( L ) k , conditioned on ﬁxed

Original: [Equation]
Revised: [Equation]

Original: for all i such that f ( ` 0 )
Revised: for all i such that f ( ` 0 )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: As  00 ( · ) is assumed to be uniformly bounded by Assumption 3.2, we use Lemma B.1 to bound its matrix variance, hence the matrix Gaussian series can be bounded.
Revised: As  00 ( · ) is assumed to be uniformly bounded by Assumption 3.2, we use Lemma B.1 to bound its matrix variance, hence the matrix Gaussian series can be bounded.

Original: Note that such tail bound does not scale with the largest in-degree of the networks, since the in-degree of f k , i.e., m ( L ) k , will be cancelled out with the scaling factor 1 /
Revised: Note that such tail bound does not scale with the largest in-degree of the networks, since the in-degree of f k , i.e., m ( L ) k , will be cancelled out with the scaling factor 1 /

Original: [Equation]
Revised: [Equation]

Original: in the bound of matrix variance.
Revised: in the bound of matrix variance.

Original: See the complete proof in Appendix B.
Revised: See the complete proof in Appendix B.

Original:  Relation to optimization
Revised:  Relation to optimization

Original: While transition to linearity is a signiﬁcant and surprising property of wide networks in its own right, it also plays an important role in building the optimization theory of wide feedforward neural networks.
Revised: While transition to linearity is a signiﬁcant and surprising property of wide networks in its own right, it also plays an important role in building the optimization theory of wide feedforward neural networks.

Original: Speciﬁcally, transition to linearity provides a path toward showing that the corresponding loss function satisﬁes the PL ⇤ condition in a ball of a certain radius, which is sufﬁcient for exponential convergence of optimization to a global minimum by gradient descent or SGD [13].
Revised: Speciﬁcally, transition to linearity provides a path toward showing that the corresponding loss function satisﬁes the PL ⇤ condition in a ball of a certain radius, which is sufﬁcient for exponential convergence of optimization to a global minimum by gradient descent or SGD [13].

Original: Consider a supervised learning task.
Revised: Consider a supervised learning task.

Original: Given training inputs and labels { ( x i , y i ) } ni =1 , we use GD/SGD to minimize the square loss:
Revised: Given training inputs and labels { ( x i , y i ) } ni =1 , we use GD/SGD to minimize the square loss:

Original: [Equation]
Revised: [Equation]

Original: 15) where f ( w ; · ) is a feedforward neural network.
Revised: 15) where f ( w ; · ) is a feedforward neural network.

Original: The loss L ( w ) is said to satisfy µ -PL ⇤ condition, a variant of the well-known Polyak- Ł ojasiewicz condition [19, 15], at point w , if
Revised: The loss L ( w ) is said to satisfy µ -PL ⇤ condition, a variant of the well-known Polyak- Ł ojasiewicz condition [19, 15], at point w , if

Original: [Equation]
Revised: [Equation]

Original: Satisfaction of this µ -PL ⇤ condition in a ball B ( w 0 , R ) with R
Revised: Satisfaction of this µ -PL ⇤ condition in a ball B ( w 0 , R ) with R

Original: =
Revised: =

Original: O
Revised: O

Original: (1 /µ ) around the starting point w 0 of GD/SGD guarantees a fast converge of the algorithm to a global minimum in this ball [13].
Revised: (1 /µ ) around the starting point w 0 of GD/SGD guarantees a fast converge of the algorithm to a global minimum in this ball [13].

Original: In the following, we use the transition to linearity of wide feedforward networks to establish the PL ⇤ condition for L ( w ) .
Revised: In the following, we use the transition to linearity of wide feedforward networks to establish the PL ⇤ condition for L ( w ) .

Original: Taking derivative on Eq.
Revised: Taking derivative on Eq.

Original: (15), we get
Revised: (15), we get

Original: [Equation]
Revised: [Equation]

Original: 16) where matrix K ( w ) , with elements
Revised: 16) where matrix K ( w ) , with elements

Original: [Equation]
Revised: [Equation]

Original: (17) is called Neural Tangent Kernel (NTK) [10], and  min ( · ) denotes the smallest eigenvalue of a matrix.
Revised: (17) is called Neural Tangent Kernel (NTK) [10], and  min ( · ) denotes the smallest eigenvalue of a matrix.

Original: Note that, by deﬁnition, the NTK matrix is always positive semi-deﬁnite, i.e.,  min ( K ( w ))
Revised: Note that, by deﬁnition, the NTK matrix is always positive semi-deﬁnite, i.e.,  min ( K ( w ))

Original:  0 .
Revised:  0 .

Original: Directly by the deﬁnition of PL ⇤ condition, at a given point w , if  min ( K ( w )) is strictly positive, then the loss function L ( w ) satisﬁes PL ⇤ condition.
Revised: Directly by the deﬁnition of PL ⇤ condition, at a given point w , if  min ( K ( w )) is strictly positive, then the loss function L ( w ) satisﬁes PL ⇤ condition.

Original: To establish convergence of GD/SGD, it is sufﬁcient to verify that PL ⇤ condition is satisﬁed in a ball
Revised: To establish convergence of GD/SGD, it is sufﬁcient to verify that PL ⇤ condition is satisﬁed in a ball

Original: B ( w 0 , R ) with R = O (1 /µ ) .
Revised: B ( w 0 , R ) with R = O (1 /µ ) .

Original: Assuming that  min ( K ( w 0 )) is bounded away from zero, transition to linearity extends the satisfaction of the PL ⇤ condition from one point w 0 to all points in B ( w 0 , R ) .
Revised: Assuming that  min ( K ( w 0 )) is bounded away from zero, transition to linearity extends the satisfaction of the PL ⇤ condition from one point w 0 to all points in B ( w 0 , R ) .

Original: PL ⇤ condition at w 0 .
Revised: PL ⇤ condition at w 0 .

Original: For certain neural networks, e.g., FCNs, CNNs and ResNets, strict positiveness of  min ( K ( w 0 )) can be shown, see for example, [6, 5].
Revised: For certain neural networks, e.g., FCNs, CNNs and ResNets, strict positiveness of  min ( K ( w 0 )) can be shown, see for example, [6, 5].

Original: We expect same holds more generally, in the case of general feedforward neural networks.
Revised: We expect same holds more generally, in the case of general feedforward neural networks.

Original: Here we show that  min ( K ( w 0 )) can be bounded from 0 for one data point under certain additional assumptions.
Revised: Here we show that  min ( K ( w 0 )) can be bounded from 0 for one data point under certain additional assumptions.

Original: Since there is only one data point,  min ( K ( w 0 ))
Revised: Since there is only one data point,  min ( K ( w 0 ))

Original: =
Revised: =

Original: K ( w 0 )
Revised: K ( w 0 )

Original: =
Revised: =

Original: kr w f ( w 0 ) k 2 .
Revised: kr w f ( w 0 ) k 2 .

Original: We also assume the following on activation functions and the input.
Revised: We also assume the following on activation functions and the input.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: The input x satisﬁes x ⇠ N (0 , I d 0 ) .
Revised: The input x satisﬁes x ⇠ N (0 , I d 0 ) .

Original: Assumption 4.2.
Revised: Assumption 4.2.

Original: The activation function is homogeneous, i.e.  ( ` ) i ( az ) = a r  ( ` ) i ( z ) , r > 0 for any constant a .
Revised: The activation function is homogeneous, i.e.  ( ` ) i ( az ) = a r  ( ` ) i ( z ) , r > 0 for any constant a .

Original: And inf ` 2 [ L  1] ,i 

Revised: And inf ` 2 [ L  1] ,i

Original: [ d ` ] E z ⇠ N (0 , 1) h  ( ` ) i ( z )i =
Revised: [ d ` ] E z ⇠ N (0 , 1) h  ( ` ) i ( z )i =

Original: C  > 0 .
Revised: C  > 0 .

Original: Remark 4.3 .
Revised: Remark 4.3 .

Original: Here for simplicity we assume the activation functions are homogeneous with the same r .
Revised: Here for simplicity we assume the activation functions are homogeneous with the same r .

Original: It is not hard to extend the result to the case that each activation function has different r .
Revised: It is not hard to extend the result to the case that each activation function has different r .

Original: Proposition 4.4.
Revised: Proposition 4.4.

Original: With Assumption 4.1 and 4.2, we have for any
Revised: With Assumption 4.1 and 4.2, we have for any

Original: k 2 [ d ` ] ,
Revised: k 2 [ d ` ] ,

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix G.
Revised: The proof can be found in Appendix G.

Original: The above proposition establishes a positive lower bound on kr w f ( w 0 ) k , hence also on  min ( K ( w 0 )) .
Revised: The above proposition establishes a positive lower bound on kr w f ( w 0 ) k , hence also on  min ( K ( w 0 )) .

Original: Using Eq.
Revised: Using Eq.

Original: (16), we get that the loss function L ( w ) satisﬁes PL ⇤ at w 0 .
Revised: (16), we get that the loss function L ( w ) satisﬁes PL ⇤ at w 0 .

Original: Extending PL ⇤ condition to B ( w 0 , R ) .
Revised: Extending PL ⇤ condition to B ( w 0 , R ) .

Original: Now we use transition to linearity to extend the satisfaction of PL ⇤ condition to the ball B ( w 0 , R ) .
Revised: Now we use transition to linearity to extend the satisfaction of PL ⇤ condition to the ball B ( w 0 , R ) .

Original: In Theorem 3.6, we see that, a feedforward neural network f ( w ) transitions to linearity, i.e., k H f ( w ) k
Revised: In Theorem 3.6, we see that, a feedforward neural network f ( w ) transitions to linearity, i.e., k H f ( w ) k

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: O
Revised: O

Original: (1 / p m ) in this ball.
Revised: (1 / p m ) in this ball.

Original: An immediate consequence is that, for any w 2 B ( w 0 , R ) ,
Revised: An immediate consequence is that, for any w 2 B ( w 0 , R ) ,

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: Since  min ( K ( w 0 )) is bound from 0 and k H f ( w ) k can be arbitrarily small as long as m is large enough, we have  min ( K ( w )) is lower bounded from 0 in the whole ball.
Revised: Since  min ( K ( w 0 )) is bound from 0 and k H f ( w ) k can be arbitrarily small as long as m is large enough, we have  min ( K ( w )) is lower bounded from 0 in the whole ball.

Original: Speciﬁcally, there is a µ > 0 such that
Revised: Speciﬁcally, there is a µ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Moreover, the radius R can be set to be O (1 /µ ) , while keeping the above inequality hold.
Revised: Moreover, the radius R can be set to be O (1 /µ ) , while keeping the above inequality hold.

Original: Then, applying the theory in [13], existence of global minima of L ( w ) and convergence of GD/SGD can be established.
Revised: Then, applying the theory in [13], existence of global minima of L ( w ) and convergence of GD/SGD can be established.

Original: For the case of multiple data points, extra techniques are needed to lower bound the minimum eigenvalue of the tangent kernel.
Revised: For the case of multiple data points, extra techniques are needed to lower bound the minimum eigenvalue of the tangent kernel.

Original: Since we focus more on the transition to linearity of feedforward neural networks in this paper, we leave it as a future work.
Revised: Since we focus more on the transition to linearity of feedforward neural networks in this paper, we leave it as a future work.

Original: Non-linear activation function on outputs and transition to linearity.
Revised: Non-linear activation function on outputs and transition to linearity.

Original: In this paper, we mainly discussed feedforward neural networks with linear activation function on output neurons.
Revised: In this paper, we mainly discussed feedforward neural networks with linear activation function on output neurons.

Original: In most of the literature also considers this setting [10, 17, 18, 6, 5, 27, 26].
Revised: In most of the literature also considers this setting [10, 17, 18, 6, 5, 27, 26].

Original: In fact, as pointed out in [13] for
Revised: In fact, as pointed out in [13] for

Original: FCNs, while this linearity of activation function on the outputs is necessary for transition to linearity, it is not required for successful optimization.
Revised: FCNs, while this linearity of activation function on the outputs is necessary for transition to linearity, it is not required for successful optimization.

Original: Speciﬁcally, simply adding a nonlinear activation function on the output layer causes the Hessian norm to be O (1) , independently of the network width.
Revised: Speciﬁcally, simply adding a nonlinear activation function on the output layer causes the Hessian norm to be O (1) , independently of the network width.

Original: Thus transition to linearity does not occur.
Revised: Thus transition to linearity does not occur.

Original: However, the corresponding square loss can still satisfy the
Revised: However, the corresponding square loss can still satisfy the

Original: PL ⇤ condition and the existence of global minimum and efﬁcient optimization can still be established.
Revised: PL ⇤ condition and the existence of global minimum and efﬁcient optimization can still be established.

Original:  Discussion and future directions
Revised:  Discussion and future directions

Original: In this work, we showed that transition to linearity arises in general feedforward neural networks with arbitrary DAG architectures, extending previous results for standard architectures [10, 12, 14].
Revised: In this work, we showed that transition to linearity arises in general feedforward neural networks with arbitrary DAG architectures, extending previous results for standard architectures [10, 12, 14].

Original: We identiﬁed the minimum in-degree of all neurons except for the input and ﬁrst layers as the key quantity to control the transition to linearity of general feedforward neural networks.
Revised: We identiﬁed the minimum in-degree of all neurons except for the input and ﬁrst layers as the key quantity to control the transition to linearity of general feedforward neural networks.

Original: We showed  the property of transition to linearity is ﬂexible to the choice of the neuron function
Revised: We showed that the property of transition to linearity is ﬂexible to the choice of the neuron function

Original: Eq.
Revised: Eq.

Original: (2), e.g., skip connections and shared weights.
Revised: E.g., skip connections and shared weights do not break the property.

Original: Hence we believe our framework can be extended to more complicated neuron functions, e.g., attention layers [8].
Revised: Therefore, we believe our framework can be extended to more complicated neuron functions, e.g., attention layers [8].

Original: For non-feedforward networks, such as RNN, recent work [1] showed they also have a constant NTK.
Revised: For non-feedforward networks, such as RNN, recent work [1] showed they also have a constant NTK.

Original: For this reason, we expect transition to linearity also to occur for of non-feedforward networks.
Revised: For this reason, we expect transition to linearity also to occur for of non-feedforward networks.

Original: Another direction of future work is better understanding of optimization for DAG networks, which requires a more delicate analysis of the NTK at initialization.
Revised: Another direction of future work is better understanding of optimization for DAG networks, which requires a more delicate analysis of the NTK at initialization.

Original: Speciﬁcally, a lower bound on the minimum eigenvalue of the NTK of the DAG networks with multiple training samples is sufﬁcient for the PL ⇤ condition to hold, hence the guarantee of the convergence of GD/SGD.
Revised: Speciﬁcally, a lower bound on the minimum eigenvalue of the NTK of the DAG networks with multiple training samples is sufﬁcient for the PL ⇤ condition to hold, hence the guarantee of the convergence of GD/SGD.

Original: Acknowledgements
Revised: Acknowledgements

Original: We are grateful for support of the NSF and the Simons Foundation for the Collaboration on the
Revised: We are grateful for support of the NSF and the Simons Foundation for the Collaboration on the

Original: Theoretical Foundations of Deep Learning 4 through awards DMS-2031883 and #814639.
Revised: Theoretical Foundations of Deep Learning 4 through awards DMS-2031883 and #814639.

Original: We also acknowledge NSF support through IIS-1815697 and the TILOS institute (NSF CCF-2112665).
Revised: We also acknowledge NSF support through IIS-1815697 and the TILOS institute (NSF CCF-2112665).

Original: We thank Nvidia for the donation of GPUs.
Revised: We thank Nvidia for the donation of GPUs.

Original: This work used the Extreme Science and Engineering
Revised: This work used the Extreme Science and Engineering

Original: Discovery Environment (XSEDE, [21]), which is supported by National Science Foundation grant number ACI-1548562 and allocation TG-CIS210104.
Revised: Discovery Environment (XSEDE, [21]), which is supported by National Science Foundation grant number ACI-1548562 and allocation TG-CIS210104.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] This is a paper with theoretical results hence the assumptions constitute the core limitations.
Revised: [Yes] This is a paper with theoretical results hence the assumptions constitute the core limitations.

Original: We have the discussion for each assumption.
Revised: We have the discussion for each assumption.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] The work presents mathematical results related to deep learning and is not expected to have any negative societal impact within its scope.
Revised: [No] The work presents mathematical results related to deep learning and is not expected to have any negative societal impact within its scope.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption 3.1, 3.2 and 3.5.
Revised: [Yes] See Assumption 3.1, 3.2 and 3.5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We give a proof sketch in Section 3.1 and the complete proofs can be found in appendix.
Revised: [Yes] We give a proof sketch in Section 3.1 and the complete proofs can be found in appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]