Original: The Neural Covariance SDE: Shaped Inﬁnite Depth-and-Width Networks at Initialization
Revised: The Neural Covariance SDE: Shaped Inﬁnite Depth-and-Width Networks at Initialization

Original: Abstract
Revised: Abstract

Original: The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix deﬁned by the penultimate layer.
Revised: The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix deﬁned by the penultimate layer.

Original: In this work, we study the distribution of this random matrix.
Revised: In this work, we study the distribution of this random matrix.

Original: Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate.
Revised: Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate.

Original: However, the current inﬁnitewidth-style understanding of this shaping method is unsatisfactory for large depth: inﬁnite-width analyses ignore the microscopic ﬂuctuations from layer to layer, but these ﬂuctuations accumulate over many layers.
Revised: However, the current inﬁnitewidth-style understanding of this shaping method is unsatisfactory for large depth: inﬁnite-width analyses ignore the microscopic ﬂuctuations from layer to layer, but these ﬂuctuations accumulate over many layers.

Original: To overcome this shortcoming, we study the random covariance matrix in the shaped inﬁnite-depth-and-width limit.
Revised: To overcome this shortcoming, we study the random covariance matrix in the shaped inﬁnite-depth-and-width limit.

Original: We identify the precise scaling of the activation function necessary to arrive at a non-trivial limit, and show that the random covariance matrix is governed by a stochastic differential equation (SDE) that we call the Neural Covariance SDE.
Revised: We identify the precise scaling of the activation function necessary to arrive at a non-trivial limit, and show that the random covariance matrix is governed by a stochastic differential equation (SDE) that we call the Neural Covariance SDE.

Original: Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of ﬁnite networks.
Revised: Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of ﬁnite networks.

Original: Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function.
Revised: Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function.

Original:  Introduction
Revised:  Introduction

Original: Of the many milestones in deep learning theory the precise characterization of the inﬁnite-width limit of neural networks at initialization as a Gaussian process with a non-random covariance matrix [1, 2] was a turning point.
Revised: Of the many milestones in deep learning theory the precise characterization of the inﬁnite-width limit of neural networks at initialization as a Gaussian process with a non-random covariance matrix [1, 2] was a turning point.

Original: The so-called Neural Network Gaussian process (NNGP) theory laid the mathematical foundation to study various limiting training dynamics under gradient descent [3–12].
Revised: The so-called Neural Network Gaussian process (NNGP) theory laid the mathematical foundation to study various limiting training dynamics under gradient descent [3–12].

Original: The Neural Tangent Kernel (NTK) limit formed the foundation for a rush of theoretical work, including advances in our understanding of generalization for wide networks [13–15].
Revised: The Neural Tangent Kernel (NTK) limit formed the foundation for a rush of theoretical work, including advances in our understanding of generalization for wide networks [13–15].

Original: Besides the NTK limit, the inﬁnite-width mean-ﬁeld limit was developed [16–19], where the different parameterization demonstrates beneﬁts for feature learning and hyperparameter tuning [20–22].
Revised: Besides the NTK limit, the inﬁnite-width mean-ﬁeld limit was developed [16–19], where the different parameterization demonstrates beneﬁts for feature learning and hyperparameter tuning [20–22].

Original: Fundamentally, the inﬁnite-width paradigm derives results from the assumption that the depth of the network is held ﬁxed while the widths of all layers grow to inﬁnity.
Revised: Fundamentally, the inﬁnite-width paradigm derives results from the assumption that the depth of the network is held ﬁxed while the widths of all layers grow to inﬁnity.

Original: Unfortunately, this assumption can be problematic for modeling real-world networks, as the microscopic ﬂuctuations from layer to layer are neglected in this limit (see Figure 1).
Revised: Unfortunately, this assumption can be problematic for modeling real-world networks, as the microscopic ﬂuctuations from layer to layer are neglected in this limit (see Figure 1).

Original: In particular, inﬁnite-width predictions are shown to be poor approximations of real networks unless the depth is much less than the width [23, 24].
Revised: In particular, inﬁnite-width predictions are shown to be poor approximations of real networks unless the depth is much less than the width [23, 24].

Original: Impressive achievements of deep networks with billions of parameters crystallize the importance of understanding extremely large, deep neural networks (DNNs).
Revised: Impressive achievements of deep networks with billions of parameters crystallize the importance of understanding extremely large, deep neural networks (DNNs).

Original: An alternative to the inﬁnite-width paradigm is the inﬁnite-depth-and-width paradigm.
Revised: An alternative to the inﬁnite-width paradigm is the inﬁnite-depth-and-width paradigm.

Original: In this setting, both the network depth d and the width n of each layer are simultaneously scaled to inﬁnity, while their relative ratio d/n remains ﬁxed [23, 25–29].
Revised: In this setting, both the network depth d and the width n of each layer are simultaneously scaled to inﬁnity, while their relative ratio d/n remains ﬁxed [23, 25–29].

Original: Recent work also explores using d/n as an effective perturbation parameter [30–
Revised: Recent work also explores using d/n as an effective perturbation parameter [30–

Original: [Figure]
Revised: [Figure]

Original: Shaped ReLU DNNs
Revised: Shaped ReLU DNNs

Original: [Figure]
Revised: [Figure]

Original: see Deﬁnition 3.
Revised: see Deﬁnition 3.

Original: [Figure]
Revised: [Figure]

Original: = − 1 , DE step size 1e − 2 .
Revised: = − 1 , DE step size 1e − 2 .

Original: Densities from kernel density estimation.
Revised: Densities from kernel density estimation.

Original: 33] or to study concentration bounds in terms of d/n [5, 34].
Revised: 33] or to study concentration bounds in terms of d/n [5, 34].

Original: This limit has the distinct advantage of being incredibly accurate at predicting the output distribution for ﬁnite size networks at initialization [27] — a signiﬁcant improvement over the NNGP theory.
Revised: This limit has the distinct advantage of being incredibly accurate at predicting the output distribution for ﬁnite size networks at initialization [27] — a signiﬁcant improvement over the NNGP theory.

Revised: Furthermore, it has also been shown that there is feature learning in this limit [23], in contrast to the linear regime of inﬁnite-width limits [8].

Original: Considering the mathematical success of the NNGP techniques, the inﬁnite-depth-and-width limit hints at the possibility of developing an accurate theory for training and generalization.
Revised: Considering the mathematical success of the NNGP techniques, the inﬁnite-depth-and-width limit hints at the possibility of developing an accurate theory for training and generalization.

Original: An immediate issue of the inﬁnite-depth limit is that this limit predicts that network output becomes degenerate as depth increases: on initialization the network becomes a constant function sending all inputs to the same (random) output [35, 36, 33].
Revised: An immediate issue of the inﬁnite-depth limit is that this limit predicts that network output becomes degenerate as depth increases: on initialization the network becomes a constant function sending all inputs to the same (random) output [35, 36, 33].

Original: While degenerate outputs are not necessarily an issue in theory, it poses a more serious problem in practice: degenerate correlations imply a “sharp” input–output Jacobian, and therefore exploding gradients [37, 25].
Revised: While degenerate outputs are not necessarily an issue in theory, it poses a more serious problem in practice: degenerate correlations imply a “sharp” input–output Jacobian, and therefore exploding gradients [37, 25].

Original: Intuitively, the output is not very sensitive to changes in the input, hence the gradient must be very large in the earlier layers.
Revised: Intuitively, the output is not very sensitive to changes in the input, hence the gradient must be very large in the earlier layers.

Revised: A promising new attack on this problem is to modify the activation function (“shaping”) to reduce to the effect of degeneracy [38, 39].

Revised: In this prior work, extensive experiments show that shaping the activation signiﬁcantly improves training speed without the need for normalization layers .

Revised: This method has been proven effective for problems as large as standard ResNets on ImageNet data.

Original: The authors designed several criteria including reducing estimated output correlation, and numerically  optimized the shape of activation functions for improved training results.
Revised: The authors designed several criteria including reducing estimated output correlation, and numerically [Table] optimized the shape of activation functions for improved training results.

Original: However, their deterministic estimation of output correlation using the inﬁnite-width limit leads to a poor approximation of real networks, as the additional randomness has both non-zero mean and heavy skew (see Figure 1 right column).
Revised: However, their deterministic estimation of output correlation using the inﬁnite-width limit leads to a poor approximation of real networks, as the additional randomness has both non-zero mean and heavy skew (see Figureright column).

Original: Furthermore, numerically searching for the activation shape obscures the picture on how shaping should depend on the network depth and width.[Table]
Revised: Furthermore, numerically searching for the activation shape obscures the picture on how shaping should depend on the network depth and width.

Original: In this paper
Revised: In this paper

Original: [Table]
Revised: [Table]

Original: we address these problems by providing a precise theory of shaped inﬁnite-depthand-width networks, extending both the NNGP theories and the activation shaping techniques. 
Revised: we address these problems by providing a precise theory of shaped inﬁnite-depthand-width networks, extending both the NNGP theories and the activation shaping techniques. In particular

Original: In particular, we prescribe an exact scaling of the activation function shape as a function of network width n that leads to a non-trivial  limit.
Revised: [Table] we prescribe an exact scaling of the activation function shape as a function of network width n that leads to a non-trivial nonlinear limit.

Original: By keeping track of microscopic O ( n − 1 / 2 ) random ﬂuctuations in each layer of the network, we show that the cumulative effect is described by a stochastic differential equation (SDE) in the limit.
Revised: By keeping track of microscopic O ( n − 1 / 2 ) random ﬂuctuations in each layer of the network, we show that the cumulative effect is described by a stochastic differential equation (SDE) in the limit.

Original: In contrast to existing inﬁnite-width theory, we are able to characterize the random distribution of the output covariance, which matches closely to simulations of real networks.
Revised: In contrast to existing inﬁnite-width theory, we are able to characterize the random distribution of the output covariance, which matches closely to simulations of real networks.

Original: In a similar spirit to how the NNGP theory laid the foundation for studying training and generalization in the inﬁnite-width limit, we also see this work as building the mathematical tools for an inﬁnite-depth-and-width theory of training and generalization [Table]
Revised: In a similar spirit to how the NNGP theory laid the foundation for studying training and generalization in the inﬁnite-width limit, we also see this work as building the mathematical tools for an inﬁnite-depth-and-width theory of training and generalization.

Original: 1.1 Contributions
Revised: 1.1 Contributions

Original: Similar to the NNGP approach [Table] we use the fact that the output is Gaussian conditional on the penultimate layer.
Revised: Similar to the NNGP approach, we use the fact that the output is Gaussian conditional on the penultimate layer.

Original: However, unlike in the inﬁnite-width paradigm, the covariance matrix is no longer deterministic in the inﬁnite-depth-and-width limit.
Revised: However, unlike in the inﬁnite-width paradigm, the covariance matrix is no longer deterministic in the inﬁnite-depth-and-width limit.

Original: Our focus in this paper is to study this random covariance matrix.
Revised: Our focus in this paper is to study this random covariance matrix.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: We introduce the tool of stochastic √ n -expansions and convergence to SDEs for analyzing the distribution of covariances in DNNs.
Revised: We introduce the tool of stochastic √ n -expansions and convergence to SDEs for analyzing the distribution of covariances in DNNs.

Original: For unshaped ReLU-like activations, we show that the norm of each layer evolves according to geometric Brownian motion and correlations evolve according to a discrete Markov process.
Revised: For unshaped ReLU-like activations, we show that the norm of each layer evolves according to geometric Brownian motion and correlations evolve according to a discrete Markov process.

Original: See left column of Figure 1 and Section 2. 3.
Revised: See left column of Figure 1 and Section 2. 3.

Original: For both ReLU-like and a large class of smooth activation functions, we derive the Neural Covariance SDE characterizing the distribution of the shaped inﬁnite-depth-and-width limit.
Revised: For both ReLU-like and a large class of smooth activation functions, we derive the Neural Covariance SDE characterizing the distribution of the shaped inﬁnite-depth-and-width limit.

Original: See right column of Figure 1 and Section 3. 4.
Revised: See right column of Figure 1 and Section 3. 4.

Original: We show our prescribed shape scaling is exact, as other rates of scaling leads to either degenerate or linear network limits.
Revised: We show our prescribed shape scaling is exact, as other rates of scaling leads to either degenerate or linear network limits.

Original: See Proposition 3.4 and Proposition 3.10.
Revised: See Proposition 3.4 and Proposition 3.10.

Original: For smooth activations, we derive an if-and-only-if condition for exploding/vanishing norms based on properties of the activation function.
Revised: For smooth activations, we derive an if-and-only-if condition for exploding/vanishing norms based on properties of the activation function.

Original: See Proposition 3.7 and Section 4. 6.
Revised: See Proposition 3.7 and Section 4. 6.

Original: We provide simulations to verify theoretical predictions and help interpret properties of real DNNs.
Revised: We provide simulations to verify theoretical predictions and help interpret properties of real DNNs.

Original: See Figures 1 and 2 and supplemental simulations in Appendix F.
Revised: See Figures 1 and 2 and supplemental simulations in Appendix F.

Original:  Limits for Unshaped ReLU-Like Activations
Revised:  Limits for Unshaped ReLU-Like Activations

Original: Using the notation in Table 1 , the output of a fully connected feedforward network with d hidden layers of width n on input x α is deﬁned by vectors of pre-activations z αℓ and post-activations φ αℓ :
Revised: Using the notation in Table 1 , the output of a fully connected feedforward network with d hidden layers of width n on input x α is deﬁned by vectors of pre-activations z αℓ and post-activations φ αℓ :

Original: [Equation]
Revised: [Equation]

Original: Note that factors of √ cn − 1 are equivalent to intializing according to the so-called He initialization [38].
Revised: Note that factors of √ cn − 1 are equivalent to intializing according to the so-called He initialization [40].

Original: We use Greek indices α, β, . . .
Revised: We use Greek indices α, β, . . .

Original: to denote multiple different inputs.
Revised: to denote multiple different inputs.

Original: Note that while our results are all stated for ﬁxed width n in each layer, they can be generalized to layer width n ℓ in the limit where all n ℓ → ∞ with P dℓ =1 n ℓ − 1 replacing the role of the depth-to-width ratio d/n [25].
Revised: Note that while our results are all stated for ﬁxed width n in each layer, they can be generalized to layer width n ℓ in the limit where all n ℓ → ∞ with P dℓ =1 n ℓ − 1 replacing the role of the depth-to-width ratio d/n [25].

Original: In this section, we analyze ReLU-like activations by which we mean activations which are linear on the negative and positive numbers given respectively by two slopes s + and s − :
Revised: In this section, we analyze ReLU-like activations by which we mean activations which are linear on the negative and positive numbers given respectively by two slopes s + and s − :

Original: [Equation]
Revised: [Equation]

Original: These are precisely the positive homogeneous functions: φ ( ax ) = | a | φ [Equation] x [Equation]
Revised: These are precisely the positive homogeneous functions: φ ( ax ) = | a | φ ( x ) ∀ x, a ∈ R .

Original: ∀ x,

Original: [Equation]

Original: ∈ R .

Original: 2.1 SDE Limits of Markov Chains
Revised: 2.1 SDE Limits of Markov Chains

Original: We brieﬂy review the main type of SDE convergence principle used in our main results (see Proposition A.6 for a more precise version).
Revised: We brieﬂy review the main type of SDE convergence principle used in our main results (see Proposition A.6 for a more precise version).

Original: Let X t , t ∈ R + , be a continuous time diffusion process obeying an SDE with drift b and variance σ 2 as given in (2.3).
Revised: Let X t , t ∈ R + , be a continuous time diffusion process obeying an SDE with drift b and variance σ 2 as given in (2.3).

Original: Suppose that for each n ∈ N , Y nℓ is a discrete time Markov chain ℓ ∈ N whose increments obey (2.3) in terms of the same functions b, σ 2 : 
Revised: Suppose that for each n ∈ N , Y nℓ is a discrete time Markov chain ℓ ∈ N whose increments obey (2.3) in terms of the same functions b, σ 2 : dX t = b ( X t ) dt + σ ( X t ) dB t

Original: [Equation]
Revised: [Equation]

Original: where ξ ℓ are independent variables with E ( ξ ℓ )
Revised: where ξ ℓ are independent variables with E ( ξ ℓ )

Original: = 0 , Var ( ξ ℓ )
Revised: = 0 , Var ( ξ ℓ )

Original: = 1 .
Revised: = 1 .

Original: With this setup, under technical conditions described precisely in Appendix A, we have convergence of Y ℓ at ℓ =
Revised: With this setup, under technical conditions described precisely in Appendix A, we have convergence of Y ℓ at ℓ =

Original: ⌊ tn ⌋ to X t , or more precisely: with X nt :
Revised: ⌊ tn ⌋ to X t , or more precisely: with X nt :

Original: = Y n ⌊ tn ⌋ we have X n → X as n → ∞ in the Skorohod topology.
Revised: = Y n ⌊ tn ⌋ we have X n → X as n → ∞ in the Skorohod topology.

Original: In our applications, n is always the width (i.e., number of neurons in each layer) which may appear implicitly and ℓ is always the layer number.
Revised: In our applications, n is always the width (i.e., number of neurons in each layer) which may appear implicitly and ℓ is always the layer number.

Original: 2.2 A Simple SDE: Geometric Brownian Motion Describes | φ αℓ |
Revised: 2.2 A Simple SDE: Geometric Brownian Motion Describes | φ αℓ |

Original: To motivate our approach of SDE limits, we illustrate the method using the example of the squared norm of the ℓ -th layer, | φ αℓ | 2 , where we recall φ αℓ = φ ( z αℓ ) .
Revised: To motivate our approach of SDE limits, we illustrate the method using the example of the squared norm of the ℓ -th layer, | φ αℓ | 2 , where we recall φ αℓ = φ ( z αℓ ) .

Original: For a single ﬁxed input x α and a ReLUlike activation φ , the norm of the post-activation neurons | φ αℓ | 2 forms a Markov chain in the layer number ℓ .
Revised: For a single ﬁxed input x α and a ReLUlike activation φ , the norm of the post-activation neurons | φ αℓ | 2 forms a Markov chain in the layer number ℓ .

Original: We use the fact that a matrix with iid Gaussian entries applied to any unit vector gives a
Revised: We use the fact that a matrix with iid Gaussian entries applied to any unit vector gives a

Original: Gaussian vector of iid N (0 , 1) entries.
Revised: Gaussian vector of iid N (0 , 1) entries.

Original: Hence, in each layer, we can deﬁne the Gaussian vector g α as follows, and use (2.1) with the positive homogeneity of φ to write the Markov chain update rule:
Revised: Hence, in each layer, we can deﬁne the Gaussian vector g α as follows, and use (2.1) with the positive homogeneity of φ to write the Markov chain update rule:

Original: [Equation]
Revised: [Equation]

Original:  where 
Revised: , where g α := W ℓ φ αℓ

Original: [Equation]
Revised: [Equation]

Original: At this point, the inﬁnite-width approach applies the law of large numbers (LLN) to conclude lim n →∞
Revised: φ [Equation] d = N

Original: [Equation]
Revised: [Equation]

Original: | φ
Revised: I n ) . [Equation]

Original: [Equation]
Revised: [Equation]

Revised: At this point, the inﬁnite-width approach applies the law of large numbers (LLN) to conclude lim n →∞ (cid:12)(cid:12)

Revised: φ αℓ +1 (cid:12)(cid:12) 2 = | φ αℓ | 2 E [ cφ 2 ( g )]

Original:  | 2 · 1 a.s. by deﬁnition of c .
Revised: = | φ αℓ | 2 · 1 a.s. by deﬁnition of c .

Original: However, the LLN cannot be applied when depth d is diverging with n , as the cumulative effect of the ﬂuctuations over d layers does not vanish!
Revised: However, the LLN cannot be applied when depth d is diverging with n , as the cumulative effect of the ﬂuctuations over d layers does not vanish!

Original: Instead, we keep track of the O (1 / √ n ) ﬂuctuations in each layer by introducing the zero mean ﬁnite variance random variable R ααℓ := 1 √ n P ni =
Revised: Instead, we keep track of the O (1 / √ n ) ﬂuctuations in each layer by introducing the zero mean ﬁnite variance random variable R ααℓ := 1 √ n P ni =

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: cφ ( g αi ) 2 − 1 (cid:1) .

Original: This allows us to rewrite this Markov chain update rule as
Revised: This allows us to rewrite this Markov chain update rule as

Original: [Equation]
Revised: [Equation]

Original: which allows us to see that the Markov chain Y nℓ
Revised: which allows us to see that the Markov chain Y nℓ

Original: = cn | φ αℓ [Equation] 2 is now in the form of (2.3) with
Revised: = cn | φ αℓ | 2 is now in the form of (2.3) with

Original: Y n 0 = 1 n in | x α | 2 , b ( Y ) ≡ 0 , σ 2 ( Y ) =
Revised: Y n 0 = 1 n in | x α | 2 , b ( Y ) ≡ 0 , σ 2 ( Y ) =

Original: Var ( R ααℓ )
Revised: Var ( R ααℓ )

Original: Y 2 =
Revised: Y 2 =

Original: Var ( cφ ( g ) 2 )
Revised: Var ( cφ ( g ) 2 )

Original: Y 2 .
Revised: Y 2 .

Original: Consequently, we have that the squared norm Markov chain converges to a geometric Brownian motion dX t = σX t dB t , or more precisely
Revised: Consequently, we have that the squared norm Markov chain converges to a geometric Brownian motion dX t = σX t dB t , or more precisely

Original: [Equation]
Revised: [Equation]

Original: where the convergence is in the Skorohod topology (see Appendix A).
Revised: where the convergence is in the Skorohod topology (see Appendix A).

Original: When φ is the ReLU function ( s + = 1 , s − = 0 ), we have c = 2 and σ 2 = 5 , which recovers known results in [25, 27–29].
Revised: When φ is the ReLU function ( s + = 1 , s − = 0 ), we have c = 2 and σ 2 = 5 , which recovers known results in [25, 27–29].

Original: We remark again this simple Markov chain example illustrates the main technique we use in later sections to establish SDE convergence for shaped networks in Section 3.
Revised: We remark again this simple Markov chain example illustrates the main technique we use in later sections to establish SDE convergence for shaped networks in Section 3.

Original: 2.3 Non-SDE Markov Chains: the Gram Matrix D φ αℓ , φ βℓ E and Correlation ρ αβℓ
Revised: 2.3 Non-SDE Markov Chains: the Gram Matrix D φ αℓ , φ βℓ E and Correlation ρ αβℓ

Original: We can generalize Section 2.2 to a collection of m inputs { x α } mα =1 by looking at the entire Gram matrix [ ⟨ φ αℓ , φ βℓ ⟩ ] mα,β =1 , where we again recall φ αℓ = φ ( z αℓ ) .
Revised: We can generalize Section 2.2 to a collection of m inputs { x α } mα =1 by looking at the entire Gram matrix [ ⟨ φ αℓ , φ βℓ ⟩ ] mα,β =1 , where we again recall φ αℓ = φ ( z αℓ ) .

Original: We note that the convergence of
Revised: We note that the convergence of

Original: Markov chains to SDEs in (2.3) can be generalized to Y nℓ ∈ R N by considering Cov ( ξ ℓ )
Revised: Markov chains to SDEs in (2.3) can be generalized to Y nℓ ∈ R N by considering Cov ( ξ ℓ )

Original: =
Revised: =

Original: I N , b : R N → R N , and σ : R N → R N × N .
Revised: I N , b : R N → R N , and σ : R N → R N × N .

Original: The Gram matrix is of particular interest because the neurons in any layer are conditionally Gaussian when conditioned on the previous layer , with covariance matrix proportional to the Gram matrix:
Revised: The Gram matrix is of particular interest because the neurons in any layer are conditionally Gaussian when conditioned on the previous layer , with covariance matrix proportional to the Gram matrix:

Original: [Equation]
Revised: [Equation]

Original: where F ℓ denotes the sigma-algebra generated by the ℓ -th layer [ z αℓ ] mα =1 , and ⊗ denotes the Kronecker product (here indicating conditionally independent entries in each vector).
Revised: where F ℓ denotes the sigma-algebra generated by the ℓ -th layer [ z αℓ ] mα =1 , and ⊗ denotes the Kronecker product (here indicating conditionally independent entries in each vector).

Original: With this property in mind, we will introduce E ℓ [ · ] := E [ · |F ℓ ] to denote the conditional expectation, and
Revised: With this property in mind, we will introduce E ℓ [ · ] := E [ · |F ℓ ] to denote the conditional expectation, and

Original: Var ℓ ( · ) , Cov ℓ ( · ) similarly to denote the conditional variance and covariance.
Revised: Var ℓ ( · ) , Cov ℓ ( · ) similarly to denote the conditional variance and covariance.

Original: If we deﬁne g α as in (2.4), we see that the g α are all marginally N (0 , I n ) .
Revised: If we deﬁne g α as in (2.4), we see that the g α are all marginally N (0 , I n ) .

Original: Similar to (2.4), we can write the update rule for the α, β -entry of the Gram matrix:
Revised: Similar to (2.4), we can write the update rule for the α, β -entry of the Gram matrix:

Original: [Equation]
Revised: [Equation]

Original: βi
Revised: βi

Original: [Equation]
Revised: [Equation]

Original: Just as we did in (2.5), we can deﬁne R αβℓ := 1 √ n P ni =1 cφ ( g αi ) φ ( g βi ) − E ℓ [ cφ ( g αi ) φ ( g βi )] and write
Revised: Just as we did in (2.5), we can deﬁne R αβℓ := 1 √ n P ni =1 cφ ( g αi ) φ ( g βi ) − E ℓ [ cφ ( g αi ) φ ( g βi )] and write

Original: [Equation]
Revised: [Equation]

Original: where R αβℓ are mean zero with covariance Cov ℓ [ R αβℓ , R γδℓ ]
Revised: where R αβℓ are mean zero with covariance Cov ℓ [ R αβℓ , R γδℓ ]

Original: = Cov
Revised: = Cov

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: cφ ( g
Revised: cφ ( g

Original: [Equation]
Revised: [Equation]

Original: ( g
Revised: ( g

Original: [Equation]
Revised: [Equation]

Original: (By the Central Limit Theorem, R αβℓ will be approximately Gaussian for large n .)
Revised: (By the Central Limit Theorem, R αβℓ will be approximately Gaussian for large n .)

Original: However, unlike the simple single-data-point case from Section 2 . 2 , we do not have convergence to a continuous time SDE.
Revised: However, unlike the simple single-data-point case from Section 2 . 2 , we do not have convergence to a continuous time SDE.

Original: This is because the differences ⟨ φ αℓ +1 , φ βℓ +1 ⟩ − ⟨ φ αℓ , φ βℓ ⟩ ↛ 0 as n → ∞ .
Revised: This is because the differences ⟨ φ αℓ +1 , φ βℓ +1 ⟩ − ⟨ φ αℓ , φ βℓ ⟩ ↛ 0 as n → ∞ .

Original: Instead, (2.9) is a discrete recursion update with additive noise of the form Y nℓ
Revised: Instead, (2.9) is a discrete recursion update with additive noise of the form Y nℓ

Original: +

Revised: +

Original: = f
Revised: = f

Original: ( Y nℓ )
Revised: ( Y nℓ )

Original: + 1 √ n ξ for some function f , and consequently Y nℓ
Revised: + 1 √ n ξ for some function f , and consequently Y nℓ

Original: +1 − Y nℓ does not vanish as n → ∞ .
Revised: +1 − Y nℓ does not vanish as n → ∞ .

Original: For a clarifying example, we can consider the one-dimensional Markov chain of hidden layer correlations.
Revised: For a clarifying example, we can consider the one-dimensional Markov chain of hidden layer correlations.

Original: More precisely, we can deﬁne ρ αβℓ = ⟨ φ αℓ , φ βℓ ⟩ / | φ
Revised: More precisely, we can deﬁne ρ αβℓ = ⟨ φ αℓ , φ βℓ ⟩ / | φ

Original: αℓ
Revised: αℓ

Original: || φ βℓ | , which we observe can be extracted from the entries of the Gram matrix.
Revised: || φ βℓ | , which we observe can be extracted from the entries of the Gram matrix.

Original: In fact, we can write down an approximate recursion update for ρ αβℓ (see Appendix B and Proposition B.8 for details):
Revised: In fact, we can write down an approximate recursion update for ρ αβℓ (see Appendix B and Proposition B.8 for details):

Original: [Equation]
Revised: [Equation]

Original: where K 1 ( ρ ) :
Revised: where K 1 ( ρ ) :

Original: =
Revised: =

Original: E [ φ ( g ) φ ( gρ + w p 1 − ρ 2 )] for g, w iid N (0 , 1) random variables, and ξ ℓ are iid
Revised: E [ φ ( g ) φ ( gρ + w p 1 − ρ 2 )] for g, w iid N (0 , 1) random variables, and ξ ℓ are iid

Original: N (0 , 1) .
Revised: N (0 , 1) .

Original: For the ReLU case, c = 2 and cK 1 ( ρ ) =
Revised: For the ReLU case, c = 2 and cK 1 ( ρ ) =

Original: ( p 1 − ρ 2 + ρ arccos( − ρ ))
Revised: ( p 1 − ρ 2 + ρ arccos( − ρ ))

Original: /π was ﬁrst calculated in [39].
Revised: /π was ﬁrst calculated in [41].

Original: In fact, we can observe that as n → ∞ , ρ αβ ⌊ tn ⌋ converges to the ﬁxed point of cK 1 ( · ) at ρ =for all t > 0 .
Revised: In fact, we can observe that as n → ∞ , ρ αβ ⌊ tn ⌋ converges to the ﬁxed point of cK 1 ( · ) at ρ =for all t > 0 .

Original: We note this limiting behaviour cannot be described by an SDE , as the solution must jump from the initial condition to the ﬁxed point at t = 0 .
Revised: We note this limiting behaviour cannot be described by an SDE , as the solution must jump from the initial condition to the ﬁxed point at t = 0 .

Original: Despite not having an SDE limit, we observe that the approximate Markov chain (2.10) already provides a much better approximation to ﬁnite size networks compared to the inﬁnite-width theory (see left column of Figure 1).
Revised: Despite not having an SDE limit, we observe that the approximate Markov chain (2.10) already provides a much better approximation to ﬁnite size networks compared to the inﬁnite-width theory (see left column of Figure 1).

Original: This is because the inﬁnite-width approach discards the terms in (2.10) that vanish as n → ∞ and consider only the update ρ αβ ℓ +1 = cK 1 ( ρ αβ ℓ ) .
Revised: This is because the inﬁnite-width approach discards the terms in (2.10) that vanish as n → ∞ and consider only the update ρ αβ ℓ +1 = cK 1 ( ρ αβ ℓ ) .

Original: Analysis of this deterministic equation leads to the prediction that ρ αβℓ = 1 − O ( ℓ − 2 ) for ℓ ≫ 1 (see (4.8) in [33] and a new bound in Appendix E).
Revised: Analysis of this deterministic equation leads to the prediction that ρ αβℓ = 1 − O ( ℓ − 2 ) for ℓ ≫ 1 (see (4.8) in [33] and a new bound in Appendix E).

Original: Furthermore, we observe that in this case, the microscopic O ( n − 1 ) and O ( n − 1 / 2 ) terms in (2.10) accumulate to macroscopic differences!
Revised: Furthermore, we observe that in this case, the microscopic O ( n − 1 ) and O ( n − 1 / 2 ) terms in (2.10) accumulate to macroscopic differences!

Original: For the examples in Figure 1, we see their net effect is that ρ αβℓ → 1 faster than the inﬁnite-width prediction.
Revised: For the examples in Figure 1, we see their net effect is that ρ αβℓ → 1 faster than the inﬁnite-width prediction.

Original: Heuristically, the reason for this discrepancy is due to σ ReLU ( ρ ) → 0 as ρ → 1 .
Revised: Heuristically, the reason for this discrepancy is due to σ ReLU ( ρ ) → 0 as ρ → 1 .

Original: This means that the randomness can push ρ αβℓ closer to 1 , but becomes “trapped” when ρ αβℓ is close to 1 because σ ReLU is so small here.
Revised: This means that the randomness can push ρ αβℓ closer to 1 , but becomes “trapped” when ρ αβℓ is close to 1 because σ ReLU is so small here.

Original: In the next section, we will see that we are just one step away from achieving limiting SDEs.
Revised: In the next section, we will see that we are just one step away from achieving limiting SDEs.

Original:  Neural Covariance SDEs: Shaped Inﬁnite-Depth-and-Width Limit
Revised:  Neural Covariance SDEs: Shaped Inﬁnite-Depth-and-Width Limit

Original: In this section, we follow the ideas of [40, 41] to reshape the activation function φ .
Revised: In this section, we follow the ideas of [38, 39] to reshape the activation function φ .

Original: Reshaping means to replace the base activation function φ in (2.1) with φ s that depends on width n .
Revised: Reshaping means to replace the base activation function φ in (2.1) with φ s that depends on width n .

Original: We will also replace the normalizing constant c = (cid:0) E φ s ( g ) 2 (cid:1) − 1 for g ∼ N (0 , 1) .
Revised: We will also replace the normalizing constant c = (cid:0) E φ s ( g ) 2 (cid:1) − 1 for g ∼ N (0 , 1) .

Original: Speciﬁcally, we will choose φ s to depend on n such that in the limit as n → ∞ , we have that φ is approximately an identity function,
Revised: Speciﬁcally, we will choose φ s to depend on n such that in the limit as n → ∞ , we have that φ is approximately an identity function,

Original: φ s → Id. Recalling from (2.7) that the output is conditionally Gaussian with covariance determined by the Gram matrix [ ⟨ φ αℓ , φ βℓ ⟩ ] mα,β =1 , therefore we recover a complete characterization by describing the random covariance matrix.
Revised: φ s → Id. Recalling from (2.7) that the output is conditionally Gaussian with covariance determined by the Gram matrix [ ⟨ φ αℓ , φ βℓ ⟩ ] mα,β =1 , therefore we recover a complete characterization by describing the random covariance matrix.

Original: 3.1 Neural Covariance SDE for Shaped ReLU-Like Activations
Revised: 3.1 Neural Covariance SDE for Shaped ReLU-Like Activations

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: We shape the ReLU-like activation φ s ( x ) :
Revised: We shape the ReLU-like activation φ s ( x ) :

Original: =
Revised: =

Original: s + max( x, 0) + s − min( x, 0) , by setting the slopes to depend on n according to s ± :
Revised: s + max( x, 0) + s − min( x, 0) , by setting the slopes to depend on n according to s ± :

Original: = 1 + c ± √ n for some given constants c + , c − ∈ R .
Revised: = 1 + c ± √ n for some given constants c + , c − ∈ R .

Original: We will also set c =
Revised: We will also set c =

Original: [Equation]
Revised: [Equation]

Original: for g
Revised: for g

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We will show that with shaping of Deﬁnition 3.1, one gets non-trivial SDEs that describe the covariance (Theorem 3.2) and correlations (Theorem 3.3) of the network.
Revised: We will show that with shaping of Deﬁnition 3.1, one gets non-trivial SDEs that describe the covariance (Theorem 3.2) and correlations (Theorem 3.3) of the network.

Original: The precise scaling is shown to be the critical scaling for a non-trivial limit in Proposition 3.4.
Revised: The precise scaling is shown to be the critical scaling for a non-trivial limit in Proposition 3.4.

Original: All proofs for results in this section appear in Appendix C.
Revised: All proofs for results in this section appear in Appendix C.

Original: Remark.
Revised: Remark.

Original: Note that in the statement of our theorems, we abuse notation and use the same letter to denote the pre-limit Markov chain and the limiting SDE.
Revised: Note that in the statement of our theorems, we abuse notation and use the same letter to denote the pre-limit Markov chain and the limiting SDE.

Original: For example, in Theorem 3.2 we use V ℓ for the covariance at layer ℓ and V t to denote the limiting SDE at time t .
Revised: For example, in Theorem 3.2 we use V ℓ for the covariance at layer ℓ and V t to denote the limiting SDE at time t .

Original: Theorem 3.2 (Covariance SDE, ReLU) .
Revised: Theorem 3.2 (Covariance SDE, ReLU) .

Original: Let V αβℓ :
Revised: Let V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ , and deﬁne V ℓ :=
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .

Original: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges in distribution in the Skorohod topology of D R + , R m ( m +1) / 2 to the solution of the SDE
Revised: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges in distribution in the Skorohod topology of D R + , R m ( m +1) / 2 to the solution of the SDE

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ν ( ρ ) :
Revised: where ν ( ρ ) :

Original: = ( c + − c − ) 2 2 π (cid:16)p
Revised: = ( c + − c − ) 2 2 π (cid:16)p

Original: [Equation]
Revised: [Equation]

Original: b ( V t ) =
Revised: b ( V t ) =

Original: (cid:20) ν
Revised: (cid:20) ν

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) q
Revised: (cid:17) q

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: (cid:21) 1 ≤ α ≤
Revised: (cid:21) 1 ≤ α ≤

Original: [Equation]
Revised: [Equation]

Original: ≤ m , and Σ( V t )
Revised: ≤ m , and Σ( V t )

Original: =
Revised: =

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: δ .
Revised: δ .

Original: Furthermore, the output distribution can be described conditional on V T evaluated at ﬁnal time T
Revised: Furthermore, the output distribution can be described conditional on V T evaluated at ﬁnal time T

Original: [Equation]
Revised: [Equation]

Original: Here we remark that ν (1) = 0 , and therefore the drift component of diagonal entries ( V ααt ) are zero, as they are geometric Brownian motion.
Revised: Here we remark that ν (1) = 0 , and therefore the drift component of diagonal entries ( V ααt ) are zero, as they are geometric Brownian motion.

Original: However, we emphasize that the m -point joint output distribution is not characterized by the marginal for each of the pairs, as the output z α out is not
Revised: However, we emphasize that the m -point joint output distribution is not characterized by the marginal for each of the pairs, as the output z α out is not

Original: Gaussian.
Revised: Gaussian.

Original: In particular, we observe the diffusion matrix entry corresponding to V αβt , V γδt involves other processes V αγt , V βδt , V αδt , V βγt !
Revised: In particular, we observe the diffusion matrix entry corresponding to V αβt , V γδt involves other processes V αγt , V βδt , V αδt , V βγt !

Original: This implies that the Neural Covariance SDE limit cannot be described by a kernel, unlike stacking random features or NNGP.
Revised: This implies that the Neural Covariance SDE limit cannot be described by a kernel, unlike stacking random features or NNGP.

Original: That being said, it is still instructive to study the marginal for a pair of data points.
Revised: That being said, it is still instructive to study the marginal for a pair of data points.

Original: More speciﬁcally, it turns out in the generalized ReLU case, we can derive the marginal SDE for the correlation process.
Revised: More speciﬁcally, it turns out in the generalized ReLU case, we can derive the marginal SDE for the correlation process.

Original: Theorem 3.3 (Correlation SDE, ReLU) .
Revised: Theorem 3.3 (Correlation SDE, ReLU) .

Original: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ
Revised: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ βℓ | , where φ αℓ :=
Revised: || φ βℓ | , where φ αℓ :=

Original: φ s ( z αℓ ) .
Revised: φ s ( z αℓ ) .

Original: In the limit as n → ∞ and s ± = 1 + c ± √ n , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R
Revised: In the limit as n → ∞ and s ± = 1 + c ± √ n , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To help interpret the SDE, we observe that µ and σ are entirely independent of the activation function.
Revised: To help interpret the SDE, we observe that µ and σ are entirely independent of the activation function.

Original: In other words, these terms will be present in this limit even for linear networks.
Revised: In other words, these terms will be present in this limit even for linear networks.

Original: At the same time, ν describes the inﬂuence of the shaped activation function in this limit.
Revised: At the same time, ν describes the inﬂuence of the shaped activation function in this limit.

Original: [41] has derived a related ordinary differential equation (ODE) of dρ t =
Revised: [39] has derived a related ordinary differential equation (ODE) of dρ t =

Original: ν ( ρ t ) dt in the sequential limit of n → ∞ then d → ∞ , where the activation is shaped depending on depth.
Revised: ν ( ρ t ) dt in the sequential limit of n → ∞ then d → ∞ , where the activation is shaped depending on depth.

Original: Here we also note that ν ( ρ ) is closely related to the J 1 function derived in [39].
Revised: Here we also note that ν ( ρ ) is closely related to the J 1 function derived in [41].

Original: See Appendix C.3 for the m -point joint version of the correlation SDE, and Appendix F for an empirical measure of convergence in the Kolmogorov–
Revised: See Appendix C.3 for the m -point joint version of the correlation SDE, and Appendix F for an empirical measure of convergence in the Kolmogorov–

Original: Smirnov distance.
Revised: Smirnov distance.

Original: One immediate consequence of the correlation SDE is that we can show the n − 1 / 2 scaling in Deﬁnition 3.1 is the only case where the limit is neither degenerate nor a linear network.
Revised: One immediate consequence of the correlation SDE is that we can show the n − 1 / 2 scaling in Deﬁnition 3.1 is the only case where the limit is neither degenerate nor a linear network.

Original: Proposition 3.4 (Critical Exponent, ReLU) .
Revised: Proposition 3.4 (Critical Exponent, ReLU) .

Original: Let ρ αβℓ :=
Revised: Let ρ αβℓ :=

Original: ⟨ φ
Revised: ⟨ φ

Original: [Equation]
Revised: [Equation]

Original: ,φ
Revised: ,φ

Original: [Equation]
Revised: [Equation]

Original: ⟩ | φ αℓ || φ βℓ | , where φ αℓ :=
Revised: ⟩ | φ αℓ || φ βℓ | , where φ αℓ :=

Original: φ s ( z αℓ ) .
Revised: φ s ( z αℓ ) .

Original: Consider the limit n → ∞ and s ± = 1 + c ± n p for some p ≥ 0 .
Revised: Consider the limit n → ∞ and s ± = 1 + c ± n p for some p ≥ 0 .

Original: Then depending on the value of p , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution w.r.t.
Revised: Then depending on the value of p , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution w.r.t.

Original: the Skorohod topology of D R + , R to
Revised: the Skorohod topology of D R + , R to

Original: (i) the degenerate limit: ρ αβt = 1 for all t > 0 , if 0 ≤ p < 12 , and c + ̸ = c − , (ii) the critical limit: the SDE from Theorem 3.3, if p = 12 , (iii) the linear network limit: if p > 12 , the following SDE, with µ, σ as deﬁned in (3.5) ,
Revised: (i) the degenerate limit: ρ αβt = 1 for all t > 0 , if 0 ≤ p < 12 , and c + ̸ = c − , (ii) the critical limit: the SDE from Theorem 3.3, if p = 12 , (iii) the linear network limit: if p > 12 , the following SDE, with µ, σ as deﬁned in (3.5) ,

Original: [Equation]
Revised: [Equation]

Original: Here we remark that the unshaped network case ( p = 0 ) is contained by the above in case (i).
Revised: Here we remark that the unshaped network case ( p = 0 ) is contained by the above in case (i).

Original: At the same time, we observe that case (iii) is equivalent to the correlation SDE in Theorem 3.3 except with ν = 0 .
Revised: At the same time, we observe that case (iii) is equivalent to the correlation SDE in Theorem 3.3 except with ν = 0 .

Original: In particular, we observe this limit is also reached when c + = c − , which implies φ s ( x )
Revised: In particular, we observe this limit is also reached when c + = c − , which implies φ s ( x )

Original: =
Revised: =

Original: s + x is linear, which is the reason we call this the linear network limit.
Revised: s + x is linear, which is the reason we call this the linear network limit.

Original: Furthermore, without much additional work, the same argument also implies the joint covariance SDE also loses the drift component, i.e., dV t = Σ( V t ) 1 / 2 dB t .
Revised: Furthermore, without much additional work, the same argument also implies the joint covariance SDE also loses the drift component, i.e., dV t = Σ( V t ) 1 / 2 dB t .

Original: 3.2 Neural Covariance SDE for Shaped Smooth Activations
Revised: 3.2 Neural Covariance SDE for Shaped Smooth Activations

Original: In this section, we consider smooth activation functions and derive a similar covariance SDE.
Revised: In this section, we consider smooth activation functions and derive a similar covariance SDE.

Original: All the proofs for results in this section can be found in Appendix D.
Revised: All the proofs for results in this section can be found in Appendix D.

Original: Assumption 3.5.
Revised: Assumption 3.5.

Original: φ ∈ C 4 ( R ) , φ (0) = 0 , φ ′ (0)
Revised: φ ∈ C 4 ( R ) , φ (0) = 0 , φ ′ (0)

Original: = 1 , and | φ (4) ( x ) | ≤ C (1+ | x | p ) for some C, p > 0 .
Revised: = 1 , and | φ (4) ( x ) | ≤ C (1+ | x | p ) for some C, p > 0 .

Original: We note that for any non-constant function σ ∈ C 1 ( R ) and x 0 ∈ R such that σ ′ ( x 0 ) ̸ = 0 , we can always deﬁne φ ( x ) :
Revised: We note that for any non-constant function σ ∈ C 1 ( R ) and x 0 ∈ R such that σ ′ ( x 0 ) ̸ = 0 , we can always deﬁne φ ( x ) :

Original: = σ ( x + x 0 ) − σ ( x 0 ) σ ′ ( x 0 ) such that it satisﬁes φ (0) = 0 , φ ′ (0)
Revised: = σ ( x + x 0 ) − σ ( x 0 ) σ ′ ( x 0 ) such that it satisﬁes φ (0) = 0 , φ ′ (0)

Original: = 1 .
Revised: = 1 .

Original: The choice of xwill be discussed further in Section 4.
Revised: The choice of xwill be discussed further in Section 4.

Original: The fourth derivative growth condition is used to control the
Revised: The fourth derivative growth condition is used to control the

Original: Taylor remainder term in expectation, but any control over the remainder will sufﬁce.
Revised: Taylor remainder term in expectation, but any control over the remainder will sufﬁce.

Original: Following the ideas of [40], we consider the following shaping of a smooth activation function φ .
Revised: Following the ideas of [38], we consider the following shaping of a smooth activation function φ .

Original: Deﬁnition 3.6.
Revised: Deﬁnition 3.6.

Original: For some constant a > 0 , we set φ s ( x ) :=
Revised: For some constant a > 0 , we set φ s ( x ) :=

Original: sφ (cid:0) xs (cid:1) with s = a √ n , and c = (cid:0)
Revised: sφ (cid:0) xs (cid:1) with s = a √ n , and c = (cid:0)

Original: E φ s ( g ) 2 (cid:1) − 1 for g ∼ N (0 , 1) .
Revised: E φ s ( g ) 2 (cid:1) − 1 for g ∼ N (0 , 1) .

Revised: Observe that in the limit n → ∞ , we will achieve that φ s → Id as desired.

Revised: We also observe that the shaping factor s outside the activation cancels out with the next layer’s 1 s factor, therefore it is equivalent shape the entire network.

Revised: More precisely, if we view z out as an input-output map f : R n in → R n out of an unshaped network, then shaping the smooth activation functions is equivalent to the modiﬁcation sf

Revised: [Equation]

Original: In this regime, we can similarly characterize the joint output distribution, however the limiting SDEs are not always well behaved.
Revised: In this regime, we can similarly characterize the joint output distribution, however the limiting SDEs are not always well behaved.

Original: In particular, they can have ﬁnite time explosions as described by the
Revised: In particular, they can have ﬁnite time explosions as described by the

Original: Feller test for explosions [42, Theorem 5.5.29].
Revised: Feller test for explosions [42, Theorem 5.5.29].

Original: Here the SDE in Proposition 3.7 is exactly the V ααt marginal of the Neural Covariance SDE, with the parameter b determined by the activation function φ and controls whether or not ﬁnite time explosions happen (see (4.1)).
Revised: Here the SDE in Proposition 3.7 is exactly the V ααt marginal of the Neural Covariance SDE, with the parameter b determined by the activation function φ and controls whether or not ﬁnite time explosions happen (see (4.1)).

Original: Proposition 3.7 (Finite Time Explosion) .
Revised: Proposition 3.7 (Finite Time Explosion) .

Original: Let X t ∈ R
Revised: Let X t ∈ R

Original: + be a solution to the following SDE
Revised: + be a solution to the following SDE

Original: [Equation]
Revised: [Equation]

Original: Let τ ∗ =
Revised: Let τ ∗ =

Original: sup M> 0 inf { t : X t ≥ M or X t ≤ M − 1 } be the explosion time, and we say X t has a ﬁnite time explosion if τ ∗ < ∞ .
Revised: sup M> 0 inf { t : X t ≥ M or X t ≤ M − 1 } be the explosion time, and we say X t has a ﬁnite time explosion if τ ∗ < ∞ .

Original: For this equation, P [ τ ∗ = ∞ ]
Revised: For this equation, P [ τ ∗ = ∞ ]

Original: = 1 if and only if b ≤ 0 .
Revised: = 1 if and only if b ≤ 0 .

Original: Technically speaking, the main culprit behind ﬁnite time explosions is the non-Lipschitzness of the drift coefﬁcient.
Revised: Technically speaking, the main culprit behind ﬁnite time explosions is the non-Lipschitzness of the drift coefﬁcient.

Original: This issue requires us to weaken the sense of convergence in this section; the ordinary convergence in the Skorohod topology is in general not true when the diffusion has ﬁnite time explosions.
Revised: This issue requires us to weaken the sense of convergence in this section; the ordinary convergence in the Skorohod topology is in general not true when the diffusion has ﬁnite time explosions.

Original: A weakened type of convergence is the best we can hope for.
Revised: A weakened type of convergence is the best we can hope for.

Original: To this goal, we introduce the following deﬁnition.
Revised: To this goal, we introduce the following deﬁnition.

Original: Deﬁnition 3.8.
Revised: Deﬁnition 3.8.

Original: We say a sequence of processes X n converge locally to X in the Skorohod topology if for any r > 0 , we deﬁne the following stopping times
Revised: We say a sequence of processes X n converge locally to X in the Skorohod topology if for any r > 0 , we deﬁne the following stopping times

Original: [Equation]
Revised: [Equation]

Original: and we have that X nt
Revised: and we have that X nt

Original: ∧ τ n converge to X t ∧ τ in the Skorohod topology.
Revised: ∧ τ n converge to X t ∧ τ in the Skorohod topology.

Original: This weakened sense of convergence essentially constrains the processes X n , X in a bounded set by adding an absorbing boundary condition.
Revised: This weakened sense of convergence essentially constrains the processes X n , X in a bounded set by adding an absorbing boundary condition.

Original: Not only do these stopping times rule out explosions, the drift coefﬁcient is now also Lipschitz on a compact set.
Revised: Not only do these stopping times rule out explosions, the drift coefﬁcient is now also Lipschitz on a compact set.

Original: With this notion of convergence, we can now state a precise Neural Covariance SDE result for general smooth activation functions.
Revised: With this notion of convergence, we can now state a precise Neural Covariance SDE result for general smooth activation functions.

Original: Theorem 3.9 (Covariance SDE, Smooth) .
Revised: Theorem 3.9 (Covariance SDE, Smooth) .

Original: Let φ satisfy Assumption 3.5, V αβℓ :
Revised: Let φ satisfy Assumption 3.5, V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) , and deﬁne V ℓ :=
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .

Original: Then, with s = a √ n as in Deﬁnition 3.6, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /
Revised: Then, with s = a √ n as in Deﬁnition 3.6, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /

Original: [Equation]
Revised: [Equation]

Original: where Σ( V t ) is the same as Theorem 3.2 and
Revised: where Σ( V t ) is the same as Theorem 3.2 and

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if V T is ﬁnite, then the output distribution can be described conditional on V T as
Revised: Furthermore, if V T is ﬁnite, then the output distribution can be described conditional on V T as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and otherwise the distribution of [ z α out ] mα =1 is undeﬁned.
Revised: and otherwise the distribution of [ z α out ] mα =1 is undeﬁned.

Original: We also have a similar critical scaling result for general smooth activations.
Revised: We also have a similar critical scaling result for general smooth activations.

Original: Proposition 3.10 (Critical Exponent, Smooth) .
Revised: Proposition 3.10 (Critical Exponent, Smooth) .

Original: Let φ satisfy Assumption 3.5, V αβℓ :
Revised: Let φ satisfy Assumption 3.5, V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) with s = an p for some p > 0 , and deﬁne V ℓ :=
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) with s = an p for some p > 0 , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector.
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector.

Original: Then in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution w.r.t.
Revised: Then in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution w.r.t.

Original: the Skorohod topology of D R + , R m ( m +1) / 2 to V , which depending on the value of p is
Revised: the Skorohod topology of D R + , R m ( m +1) / 2 to V , which depending on the value of p is

Original: [Equation]
Revised: [Figure]

Original: for all t > 0 and 1 ≤ α ≤ β ≤ m ,
Revised: Centre

Original: [Figure]
Revised: [Figure]

Revised: 1e

Original: M a g n i t u d e S a m p l e
Revised: M a g n i t u d e S a m p l e

Original: P a t h s
Revised: P a t h s

Original: V [Figure] Stable Center [Figure]
Revised: V (b) Stable Center x 0 = log     100 Layer 0.

Original: log
Revised: M a g n i t u d e S a m p l e P a t h s V (i) the degenerate limit: if

Original: [Figure]
Revised: [Equation]

Original: [Figure] Layer 0.
Revised: p

Original: [Figure]
Revised: [Equation]

Original: M a g n i t u d e S a m p l e P a t h s V (ii) the critical limit: the solution of the SDE from Theorem 3.9, if p = 12 , (iii) the linear network limit: the stopped solution to the SDE dV t = Σ( V t ) dB t with coefﬁcient Σ deﬁned in Theorem 3.3, if p > 12 .
Revised: for all t > 0 and 1 ≤ α ≤ β ≤ m , (ii) the critical limit: the solution of the SDE from Theorem 3.9, if p = 12 , (iii) the linear network limit: the stopped solution to the SDE dV t = Σ( V t ) dB t with coefﬁcient Σ deﬁned in Theorem 3.3, if p > 12 .

Original: Here we observe that in case (i) when 34 φ ′′ (0) 2 + φ ′′′ (0) ≤ 0 , we also have a constant (in time) correlation ρ αβt similar to the ReLU case in Proposition 3.4, however in this case ρ αβt is not necessarily equal to 1 .
Revised: Here we observe that in case (i) when 34 φ ′′ (0) 2 + φ ′′′ (0) ≤ 0 , we also have a constant (in time) correlation ρ αβt similar to the ReLU case in Proposition 3.4, however in this case ρ αβt is not necessarily equal to 1 .

Original: At the same time, the linear network limit in case (iii) also has the same covariance SDE as Proposition 3.4.
Revised: At the same time, the linear network limit in case (iii) also has the same covariance SDE as Proposition 3.4.

Original:  Consequences, Discussion, and Future Directions
Revised:  Consequences, Discussion, and Future Directions

Original: So far, we have derived the Neural Covariance SDE.
Revised: So far, we have derived the Neural Covariance SDE.

Original: Analysis of this SDE reveals important behaviour of the network on initialization.
Revised: Analysis of this SDE reveals important behaviour of the network on initialization.

Original: Here we lay out one concrete example and provide some discussion and future directions.
Revised: Here we lay out one concrete example and provide some discussion and future directions.

Original: Exploding and Vanishing Norms.
Revised: Exploding and Vanishing Norms.

Revised: Here we consider the behaviour of shaping smooth activation functions, as it is done in the experiments of [38].

Revised: While the authors here avoided exploding and vanishing norms by numerically optimizing shaping parameters, we can actually describe the precise behaviour a priori with the Neural Covariance SDE.

Original: Recall the shaping parameter a from Deﬁnition 3.6.
Revised: Recall the shaping parameter a from Deﬁnition 3.6.

Original: Let V t be the solution to the SDE in (3.9).
Revised: Let V t be the solution to the SDE in (3.9).

Original: We can write down the marginal SDE for V ααt as
Revised: We can write down the marginal SDE for V ααt as

Original: [Equation]
Revised: [Equation]

Original: which implies by Proposition 3.7 that V t has a ﬁnite time explosion (with non-zero probability) if and only if 34 φ ′′ (0) 2 + φ ′′′ (0) > 0 .
Revised: which implies by Proposition 3.7 that V t has a ﬁnite time explosion (with non-zero probability) if and only if 3 4 φ ′′ (0) 2 + φ ′′′ (0) > 0 .

Original: This criterion can be used to help choose how activation functions should be centered for shaping; below are two examples.
Revised: This criterion can be used to help choose how activation functions should be centered for shaping; below are two examples.

Original: Example 4.1 (Sigmoid and tanh at x 0 = 0 ) .
Revised: Example 4.1 (Sigmoid and tanh at x 0 = 0 ) .

Original: We start with the sigmoid activation σ ( x ) = 1 1+ e − x
Revised: We start with the sigmoid activation σ ( x ) = 1 1+ e − x

Original: then we can deﬁne φ ( x ) :
Revised: then we can deﬁne φ ( x ) :

Original: = 4 σ ( x ) − 2 to satisfy Assumption 3.5, which leads to φ ′′ (0) = 0 , φ ′′′ (0) = − 12 , and therefore leads to a stable network.
Revised: = 4 σ ( x ) − 2 to satisfy Assumption 3.5, which leads to φ ′′ (0) = 0 , φ ′′′ (0) = − 12 , and therefore leads to a stable network.

Original: It turns out φ ( x ) :=
Revised: It turns out φ ( x ) :=

Original: tanh( x ) already satisﬁes Assumption 3.5, which leads to φ ′′ (0)
Revised: tanh( x ) already satisﬁes Assumption 3.5, which leads to φ ′′ (0)

Original: =
Revised: =

Original: 0 , φ ′′′ (0) =
Revised: 0 , φ ′′′ (0) =

Original: − 2 , and therefore is also stable.
Revised: − 2 , and therefore is also stable.

Original: More generally, if σ behaves like a cumulative distribution function for a symmetric unimodal density, we will have that φ ′′ (0) = 0 and φ ′′′ (0) < 0 as desired.
Revised: More generally, if σ behaves like a cumulative distribution function for a symmetric unimodal density, we will have that φ ′′ (0) = 0 and φ ′′′ (0) < 0 as desired.

Original: Example 4.2 (Soft Plus at General x 0 ∈ R ) .
Revised: Example 4.2 (Soft Plus at General x 0 ∈ R ) .

Original: Let us consider x 0 ∈ R and σ ( x )
Revised: Let us consider x 0 ∈ R and σ ( x )

Original: =
Revised: =

Original: log(

Revised: log(

Original: + e x + x 0 ) , which implies φ ( x ) :
Revised: + e x + x 0 ) , which implies φ ( x ) :

Original: = (1 + e − x 0 ) log 

Revised: = (1 + e − x 0 ) log

Original: + e x + x 0 1+ e xsatisﬁes Assumption 3.5.
Revised: + e x + x 0 1+ e xsatisﬁes Assumption 3.5.

Original: This gives us φ ′′ (0) =
Revised: This gives us φ ′′ (0) =

Original: + e x 0 , φ ′′′ (0) =
Revised: + e x 0 , φ ′′′ (0) =

Original: 1 − e x 0 (1+ e x 0 ) 2 , and therefore 34 φ ′′ (0) 

Revised: 1 − e x 0 (1+ e x 0 ) 2 , and therefore 34 φ ′′ (0)

Original: + φ ′′′ (0) =
Revised: + φ ′′′ (0) =

Original: 1 (1+ e x 0 )(cid:0) 54 − e x 0 (cid:1) .
Revised: 1 (1+ e x 0 )(cid:0) 54 − e x 0 (cid:1) .

Original: In other words, the shaped network is stable if and only if x 0 ≥ log 54 (see Figure 2).
Revised: In other words, the shaped network is stable if and only if x 0 ≥ log 54 (see Figure 2).

Revised: We note that the authors of [38] numerically found a shift of x 0 ≈ 0 .

Revised: 41 , which is in the stable regime of x 0 ≥ log 54 ≈ 0 .

Original: Relationship to Edge of Chaos.
Revised: Relationship to Edge of Chaos.

Original: The ﬁnite time explosion example above resembles the Edge of
Revised: The ﬁnite time explosion example above resembles the Edge of

Original: Chaos (EOC) analysis of gradient stability [43, 35, 44, 45], where the weight and bias variance at initialization determines a stability criterion.
Revised: Chaos (EOC) analysis of gradient stability [43, 35, 44, 45], where the weight and bias variance at initialization determines a stability criterion.

Original: However, we note that the EOC regime is sufﬁciently different that the results are not directly comparable.
Revised: However, we note that the EOC regime is sufﬁciently different that the results are not directly comparable.

Original: More precisely, the EOC analysis is in the sequential limit of inﬁnite-width and then inﬁnite-depth, which also leaves the activation function unchanged.
Revised: More precisely, the EOC analysis is in the sequential limit of inﬁnite-width and then inﬁnite-depth, which also leaves the activation function unchanged.

Original: Under very weak assumptions, the variance (diagonal of V t ) will not explode in this regime; instead, the gradient can explode due to the covariance (off diagonals).
Revised: Under very weak assumptions, the variance (diagonal of V t ) will not explode in this regime; instead, the gradient can explode due to the covariance (off diagonals).

Original: On the other hand, our ﬁnite explosion result is in the joint limit of depth and width, where the variance (diagonal of
Revised: On the other hand, our ﬁnite explosion result is in the joint limit of depth and width, where the variance (diagonal of

Original: V t ) can explode instead.
Revised: V t ) can explode instead.

Original: Posterior Inference.
Revised: Posterior Inference.

Original: Similar to the NNGP setting, we can use the Neural Covariance SDE to generate a prior over functions f : R n in → R n out .
Revised: Similar to the NNGP setting, we can use the Neural Covariance SDE to generate a prior over functions f : R n in → R n out .

Original: Consequently, an interesting future direction would be to study the posterior distribution, i.e. the output z m +1 out conditioned on x m +1 and a training dataset ( x α , z α out ) mα
Revised: Consequently, an interesting future direction would be to study the posterior distribution, i.e. the output z m +1 out conditioned on x m +1 and a training dataset ( x α , z α out ) mα

Original: =1 .
Revised: =1 .

Original: However, to our best knowledge, it is not straightforward to explicitly compute or sample from the conditional distributions for this SDE structure.
Revised: However, to our best knowledge, it is not straightforward to explicitly compute or sample from the conditional distributions for this SDE structure.

Original: It would be desirable to extend existing approaches in the perturbative regime [30, 31] to our setting.
Revised: It would be desirable to extend existing approaches in the perturbative regime [30, 31] to our setting.

Original: Extension to Other Architectures.
Revised: Extension to Other Architectures.

Original: The key step to deriving the covariance SDE is the conditional
Revised: The key step to deriving the covariance SDE is the conditional

Original: Gaussian distribution in (2.7), which directly leads to a Markov chain.
Revised: Gaussian distribution in (2.7), which directly leads to a Markov chain.

Original: It follows immediately that
Revised: It follows immediately that

Original: ResNets [46] admit a similar conditional structure.
Revised: ResNets [46] admit a similar conditional structure.

Original: With a bit more work for convolutional networks, we can obtain z αℓ +1 |F ℓ ∼ N (0 , A ( V ℓ ) ⊗ I n ) where A is an afﬁne transformation and V ℓ is the previous layer’s
Revised: With a bit more work for convolutional networks, we can obtain z αℓ +1 |F ℓ ∼ N (0 , A ( V ℓ ) ⊗ I n ) where A is an afﬁne transformation and V ℓ is the previous layer’s

Original: Gram matrix [47].
Revised: Gram matrix [47].

Original: We note that recurrent networks will not lead to a Markov chain or SDE limit, as the weight matrix is reused from layer to layer.
Revised: We note that recurrent networks will not lead to a Markov chain or SDE limit, as the weight matrix is reused from layer to layer.

Original: Simulating SDEs.
Revised: Simulating SDEs.

Original: Both the Markov chains and SDEs predict neural networks at initialization very well (see Figure 1), but the SDE is signiﬁcantly faster to simulate.
Revised: Both the Markov chains and SDEs predict neural networks at initialization very well (see Figure 1), but the SDE is signiﬁcantly faster to simulate.

Original: In particular, we can view the
Revised: In particular, we can view the

Original: Markov chain as an approximate Euler discretization of the SDE, but with a very small step size n − 1 .
Revised: Markov chain as an approximate Euler discretization of the SDE, but with a very small step size n − 1 .

Original: In contrast, to simulate the SDE we should only need a step size that is small on the scale of depth-to-width ratio T =
Revised: In contrast, to simulate the SDE we should only need a step size that is small on the scale of depth-to-width ratio T =

Original: d/n , which is independent of width n .
Revised: d/n , which is independent of width n .

Original: Therefore, practitioners using the shaping techniques of [40, 41] can now simulate the covariance SDEs at a low computational cost to signiﬁcantly improve estimates of the output correlation (see Figure 1 and additional simulations in Appendix F).
Revised: Therefore, practitioners using the shaping techniques of [38, 39] can now simulate the covariance SDEs at a low computational cost to signiﬁcantly improve estimates of the output correlation (see Figure 1 and additional simulations in Appendix F).

Original: Analytical Tractability of SDEs.
Revised: Analytical Tractability of SDEs.

Original: Besides numerical tractability, the SDEs are also far more tractable to analyze.
Revised: Besides numerical tractability, the SDEs are also far more tractable to analyze.

Original: For example, in the one input case, we arrive at geometric Brownian motion (2.6), which is known to have a log-normal distribution at ﬁxed times.
Revised: For example, in the one input case, we arrive at geometric Brownian motion (2.6), which is known to have a log-normal distribution at ﬁxed times.

Original: Similarly, our ﬁnite time explosions hinge on the fact we identiﬁed an SDE limit.
Revised: Similarly, our ﬁnite time explosions hinge on the fact we identiﬁed an SDE limit.

Original: In the same way that NNGP theory played a major role in the inﬁnite-width regime, the Neural Covariance SDEs and the techniques developed here also serve as a mathematical foundation for studying training and generalization.
Revised: In the same way that NNGP theory played a major role in the inﬁnite-width regime, the Neural Covariance SDEs and the techniques developed here also serve as a mathematical foundation for studying training and generalization.

Original: Acknowledgement
Revised: Acknowledgement

Original: We would like to thank Sinho Chewi, James Foster, Boris Hanin, Jeffrey Negrea, Nuri Mert Vural,
Revised: We would like to thank Sinho Chewi, James Foster, Boris Hanin, Jeffrey Negrea, Nuri Mert Vural,

Original: Guodong Zhang, Matthew S. Zhang, and Yuchong Zhang for helpful discussions and draft feedback.
Revised: Guodong Zhang, Matthew S. Zhang, and Yuchong Zhang for helpful discussions and draft feedback.

Original: We would like to thank Sam Buchanan and Souﬁane Hayou for pointing out a gap in the proof of
Revised: We would like to thank Sam Buchanan and Souﬁane Hayou for pointing out a gap in the proof of

Original: Proposition B.8.
Revised: Proposition B.8.

Original: ML is supported by Ontario Graduate Scholarship and the Vector Institute.
Revised: ML is supported by Ontario Graduate Scholarship and the Vector Institute.

Original: MN is supported by an NSERC Discovery Grant.
Revised: MN is supported by an NSERC Discovery Grant.

Original: DMR is supported in part by Canada CIFAR AI
Revised: DMR is supported in part by Canada CIFAR AI

Original: Chair funding through the Vector Institute, an NSERC Discovery Grant, Ontario Early Researcher
Revised: Chair funding through the Vector Institute, an NSERC Discovery Grant, Ontario Early Researcher

Original: Award, a stipend provided by the Charles Simonyi Endowment, and a New Frontiers in Research
Revised: Award, a stipend provided by the Charles Simonyi Endowment, and a New Frontiers in Research

Original: Exploration Grant.
Revised: Exploration Grant.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] The contributions list directly references the theorems and sections corresponding to them.
Revised: [Yes] The contributions list directly references the theorems and sections corresponding to them.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] In our introduction, we discussed that our theory is only at initialization, and does not describe training dynamics and generalization.
Revised: [Yes] In our introduction, we discussed that our theory is only at initialization, and does not describe training dynamics and generalization.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] The only assumption required is in Assumption 3.5.
Revised: [Yes] The only assumption required is in Assumption 3.5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Please see the relevant sections in Appendices A, C and D. 3.
Revised: [Yes] Please see the relevant sections in Appendices A, C and D. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] All of our simulations (at initialization only) are contained the ﬁle Correlation.ipynb .
Revised: [Yes] All of our simulations (at initialization only) are contained the ﬁle Correlation.ipynb .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A]
Revised: [N/A]

Original: Our experiments only required simulations at initialization, and therefore no training.
Revised: Our experiments only required simulations at initialization, and therefore no training.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: We plotted the histogram and full density of the samples instead.
Revised: We plotted the histogram and full density of the samples instead.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: Our simulations were small enough that it did not require GPUs.
Revised: Our simulations were small enough that it did not require GPUs.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Background on Markov Chain Convergence to SDEs
Revised: A Background on Markov Chain Convergence to SDEs

Original: In this section we brieﬂy review the background and technical results required to characterize the convergence of a Markov chain to an SDE.
Revised: In this section we brieﬂy review the background and technical results required to characterize the convergence of a Markov chain to an SDE.

Original: Majority of the content in this section are based on [48– 50].
Revised: Majority of the content in this section are based on [48– 50].

Original: To start we ﬁrst introduce the Skorohod J 1 -topology [48, Appendix 5].
Revised: To start we ﬁrst introduce the Skorohod J 1 -topology [48, Appendix 5].

Original: Let S be a complete separable metric space, and D R + ,S be the space of càdlàg functions (right continuous with left limits) from R + → S .
Revised: Let S be a complete separable metric space, and D R + ,S be the space of càdlàg functions (right continuous with left limits) from R + → S .

Original: Here we write x n ul −→ x to denote locally uniform convergence (i.e., uniform on compact subsets of R + ).
Revised: Here we write x n ul −→ x to denote locally uniform convergence (i.e., uniform on compact subsets of R + ).

Original: We also consider bijections λ on R +
Revised: We also consider bijections λ on R +

Original: so that λ is strictly increasing with λ 0 = 0 .
Revised: so that λ is strictly increasing with λ 0 = 0 .

Original: We can now deﬁne Skorohod convergence x n s −→ x on D R + ,S if there exists a sequence of bijections λ n satisfying the above conditions and
Revised: We can now deﬁne Skorohod convergence x n s −→ x on D R + ,S if there exists a sequence of bijections λ n satisfying the above conditions and

Original: [Equation]
Revised: [Equation]

Original: The most important result is that D R + ,S equipped with the above sense of convergence is indeed a well behaved probability space, which we state below.
Revised: The most important result is that D R + ,S equipped with the above sense of convergence is indeed a well behaved probability space, which we state below.

Original: Theorem A.1 (Theorem A5.3, [48]) .
Revised: Theorem A.1 (Theorem A5.3, [48]) .

Original: For any separable complete metric space S , there exists a topology T on D R + ,S such that
Revised: For any separable complete metric space S , there exists a topology T on D R + ,S such that

Original: (i) T induces the Skorohod convergence x n s −→ x , (ii) D R + ,S is Polish (separable completely metrizable topological space) under T , (iii) T generates the Borel σ -ﬁeld generated by the evaluation maps π t , t ≥ 0 , where π t ( x ) = x t .
Revised: (i) T induces the Skorohod convergence x n s −→ x , (ii) D R + ,S is Polish (separable completely metrizable topological space) under T , (iii) T generates the Borel σ -ﬁeld generated by the evaluation maps π t , t ≥ 0 , where π t ( x ) = x t .

Original: We also need to deﬁne Feller semi-groups.
Revised: We also need to deﬁne Feller semi-groups.

Original: To start we let S be a locally compact separable metric space and C 0 :
Revised: To start we let S be a locally compact separable metric space and C 0 :

Original: = C 0 ( S ) be the space of continuous functions that vanishes at inﬁnity, and we equip
Revised: = C 0 ( S ) be the space of continuous functions that vanishes at inﬁnity, and we equip

Original: C 0 with the sup norm to make it a Banach space.
Revised: C 0 with the sup norm to make it a Banach space.

Original: T : C 

Revised: T : C

Original: → C 0 is a positive contraction operator if for all 0 ≤ f ≤ 1 we have 0 ≤ Tf ≤ 1 .
Revised: → C 0 is a positive contraction operator if for all 0 ≤ f ≤ 1 we have 0 ≤ Tf ≤ 1 .

Original: A semi-group of such operators ( T t ) on C 0 is called a
Revised: A semi-group of such operators ( T t ) on C 0 is called a

Original: Feller semi-group if it additionally satisﬁes
Revised: Feller semi-group if it additionally satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Let D ⊂ C 0 and A : D → C 0 , and we say that ( A, D ) is a generator of ( T t ) if D is the maximal set such that for all f ∈ D , we have that
Revised: Let D ⊂ C 0 and A : D → C 0 , and we say that ( A, D ) is a generator of ( T t ) if D is the maximal set such that for all f ∈ D , we have that

Original: [Equation]
Revised: [Equation]

Original: An operator A with domain D on a Banach space B is said to be closed , if its graph G = { ( f, Af ) | f ∈ D} is a closed subset of B × B .
Revised: An operator A with domain D on a Banach space B is said to be closed , if its graph G = { ( f, Af ) | f ∈ D} is a closed subset of B × B .

Original: If the closure of G is the graph of an operator¯
Revised: If the closure of G is the graph of an operator

Original:  A , we say ¯ A is the closure of A .
Revised: ¯ A , we say ¯ A is the closure of A .

Original: Finally, we will deﬁne a linear subspace D ⊂ D as a core of
Revised: Finally, we will deﬁne a linear subspace D ⊂ D as a core of

Original: A if the closure of A | D is A .
Revised: A if the closure of A | D is A .

Original: If ( A, D ) is a generator of a Feller semigroup, every dense invariant subspace D ⊂ D is a core of A [48, Proposition 17.9].
Revised: If ( A, D ) is a generator of a Feller semigroup, every dense invariant subspace D ⊂ D is a core of A [48, Proposition 17.9].

Original: In particular, we will work with the core C ∞of smooth functions vanishing at inﬁnity.
Revised: In particular, we will work with the core C ∞of smooth functions vanishing at inﬁnity.

Original: We will state a sufﬁcient condition required for an semi-group to be Feller based on its generator.
Revised: We will state a sufﬁcient condition required for an semi-group to be Feller based on its generator.

Original: Theorem A.2 (Section 8, Theorem 2.5, [49]) .
Revised: Theorem A.2 (Section 8, Theorem 2.5, [49]) .

Original: Let a ij ∈ C 2 ( R d ) with ∂ k ∂ ℓ a ij be bounded for all i, j, k, ℓ ∈ [ d ] .
Revised: Let a ij ∈ C 2 ( R d ) with ∂ k ∂ ℓ a ij be bounded for all i, j, k, ℓ ∈ [ d ] .

Original: Further let b : R d → R d be Lipschitz.
Revised: Further let b : R d → R d be Lipschitz.

Original: Then the generator deﬁned by
Revised: Then the generator deﬁned by

Original: [Equation]
Revised: [Equation]

Original: generates a Feller 
Revised: generates a Feller semi-group on C 0 .

Original: We will next state a set of equivalent criterion for convergence of Feller processes.
Revised: We will next state a set of equivalent criterion for convergence of Feller processes.

Original: Theorem A.3 (Theorem 17.25, [48]) .
Revised: Theorem A.3 (Theorem 17.25, [48]) .

Original: Let X, X 1 , X 2 , X 3 , · · · be Feller processes in S with semigroups ( T t ) , ( T n,t ) and generators ( A, D ) , ( A n , D n ) , respectively, and ﬁx a core D for A .
Revised: Let X, X 1 , X 2 , X 3 , · · · be Feller processes in S with semigroups ( T t ) , ( T n,t ) and generators ( A, D ) , ( A n , D n ) , respectively, and ﬁx a core D for A .

Original: Then these conditions are equivalent:
Revised: Then these conditions are equivalent:

Original: (i) for any f ∈ D , there exists some f n ∈ D n with f n → f and A n f n → Af , (ii) T n,t → T t strongly for each t > 0 , (iii) T n,t f
Revised: (i) for any f ∈ D , there exists some f n ∈ D n with f n → f and A n f n → Af , (ii) T n,t → T t strongly for each t > 0 , (iii) T n,t f

Original: → T t f for every f ∈ C 0 , uniformly for bounded t > 0 , (iv) X n 0 d −→ X 0 in S ⇒ X n d −→ X in the Skohorod topology of D R + ,S .
Revised: → T t f for every f ∈ C 0 , uniformly for bounded t > 0 , (iv) X n 0 d −→ X 0 in S ⇒ X n d −→ X in the Skohorod topology of D R + ,S .

Original: Once again, we note that it is common to choose the core D = C ∞ 0 , and that checking condition (i) is sufﬁcient for convergence in the Skorohod topology.
Revised: Once again, we note that it is common to choose the core D = C ∞ 0 , and that checking condition (i) is sufﬁcient for convergence in the Skorohod topology.

Original: This is translated to the Markov chain setting by the next theorem.
Revised: This is translated to the Markov chain setting by the next theorem.

Original: Theorem A.4 (Theorem 17.28, [48]) .
Revised: Theorem A.4 (Theorem 17.28, [48]) .

Original: Let Y 1 , Y 2 , Y 3 , · · · be discrete time Markov chains in S with transition operators
Revised: Let Y 1 , Y 2 , Y 3 , · · · be discrete time Markov chains in S with transition operators

Original: U 1 , U 2 , U 3 , · · · , and let X be a Feller process with semi-group ( T t ) and generator A .
Revised: U 1 , U 2 , U 3 , · · · , and let X be a Feller process with semi-group ( T t ) and generator A .

Original: Fix a core D for A , and let 0 < h n → 0 .
Revised: Fix a core D for A , and let 0 < h n → 0 .

Original: Then conditions ( i ) − ( iv ) of Theorem A. remain equivalent for the operators and processes
Revised: Then conditions ( i ) − ( iv ) of Theorem A. remain equivalent for the operators and processes

Original: [Equation]
Revised: [Equation]

Original: It remains to check that the generators
Revised: It remains to check that the generators

Original: A n converges to A with respect to the core D = C ∞ 0 , and we will use a criterion from [50].
Revised: A n converges to A with respect to the core D = C ∞ 0 , and we will use a criterion from [50].

Original: Here we will ﬁrst let Π n ( x, dy ) be the Markov transition kernel of
Revised: Here we will ﬁrst let Π n ( x, dy ) be the Markov transition kernel of

Original: Y n , and deﬁne
Revised: Y n , and deﬁne

Original: [Equation]
Revised: [Equation]

Original: Lemma A.5 (Lemma 11.2.1, [50]) .
Revised: Lemma A.5 (Lemma 11.2.1, [50]) .

Original: The following two conditions are equivalent:
Revised: The following two conditions are equivalent:

Original: (i) For any R > 0 , ϵ > 

Revised: (i) For any R > 0 , ϵ >

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: (ii) For each f ∈ C ∞ 

Revised: (ii) For each f ∈ C ∞

Original: ( R d ) , we have that
Revised: ( R d ) , we have that

Original: [Equation]
Revised: [Equation]

Original: uniformly on compact sets of R d , where Ais deﬁned as (A.4).
Revised: uniformly on compact sets of R d , where A

Original: Finally [Equation] we summarize the above results in a user friendly form for our applications.
Revised: Finally, we summarize the above results in a user friendly form for our applications.

Original: Proposition A.6 (Convergence of Markov Chains to SDE) .
Revised: Proposition A.6 (Convergence of Markov Chains to SDE) .

Original: Let Y n be a discrete time Markov chain on R N deﬁned by the following update for p, δ >
Revised: Let Y n be a discrete time Markov chain on R N deﬁned by the following update for p, δ >

Original: [Equation]
Revised: [Equation]

Original: where ξ nℓ ∈ R N are iid random variables with zero mean, identity covariance, and moments uniformly bounded in n .
Revised: where ξ nℓ ∈ R N are iid random variables with zero mean, identity covariance, and moments uniformly bounded in n .

Original: Furthermore, ω nℓ are also iid random variables such that E [ b b n ( Y nℓ , ω nℓ ) |
Revised: Furthermore, ω nℓ are also iid random variables such that E [ b b n ( Y nℓ , ω nℓ ) |

Original: Y nℓ =
Revised: Y nℓ =

Original: y ]
Revised: y ]

Original: = b n ( y ) and b b n ( y, ω nℓ ) has uniformly bounded moments in n .
Revised: = b n ( y ) and b b n ( y, ω nℓ ) has uniformly bounded moments in n .

Original: Finally, σ n is a deterministic function, and the remainder terms in O ( n − 2 p − δ ) have uniformly bounded moments in n .
Revised: Finally, σ n is a deterministic function, and the remainder terms in O ( n − 2 p − δ ) have uniformly bounded moments in n .

Original: Suppose b n , σ n are uniformly Lipschitz functions in n and converges to b, σ uniformly on compact sets, then in the limit as n → ∞ , the process X nt =
Revised: Suppose b n , σ n are uniformly Lipschitz functions in n and converges to b, σ uniformly on compact sets, then in the limit as n → ∞ , the process X nt =

Original: Y n ⌊ tn 2 p ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R N
Revised: Y n ⌊ tn 2 p ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R N

Original: [Equation]
Revised: [Equation]

Original: Suppose otherwise b n , σ n are only locally Lipschitz (but still uniform in n ), then X n converges locally to X in the same topology (see Deﬁnition 3.8).
Revised: Suppose otherwise b n , σ n are only locally Lipschitz (but still uniform in n ), then X n converges locally to X in the same topology (see Deﬁnition 3.8).

Original: More precisely, for any ﬁxed r > 0 , we consider the stopping times
Revised: More precisely, for any ﬁxed r > 0 , we consider the stopping times

Original: [Equation]
Revised: [Equation]

Original: then the stopped process X nt
Revised: then the stopped process X nt

Original: ∧ τ n converges in distribution to the stopped solution X t ∧ τ of the above
Revised: ∧ τ n converges in distribution to the stopped solution X t ∧ τ of the above

Original: SDE in the same topology.
Revised: SDE in the same topology.

Original: Proof.
Revised: Proof.

Original: We will essentially check the criterion of Theorem A.4 directly for the metric space S
Revised: We will essentially check the criterion of Theorem A.4 directly for the metric space S

Original: = R N if b, σ is globally Lipschitz, and S = B (0 , r ) otherwise.
Revised: = R N if b, σ is globally Lipschitz, and S = B (0 , r ) otherwise.

Original: In both of these cases, b, σ are Lipschitz on
Revised: In both of these cases, b, σ are Lipschitz on

Original: S , therefore the limiting process (either X t or X t ∧ τ ) is Feller in S by Theorem A.2.
Revised: S , therefore the limiting process (either X t or X t ∧ τ ) is Feller in S by Theorem A.2.

Original: In the equivalent criteria of Theorem A.3, we will use the implication of (i) ⇒ (iv) to get convergence of X n to X in the Skorohod topology of D R + ,S .
Revised: In the equivalent criteria of Theorem A.3, we will use the implication of (i) ⇒ (iv) to get convergence of X n to X in the Skorohod topology of D R + ,S .

Original: More precisely, it is sufﬁcient to choose h n = 1 n 2 p as the natural time scale, and check 1 h n A n f → Af for any f ∈ C ∞ 0 .
Revised: More precisely, it is sufﬁcient to choose h n = 1 n 2 p as the natural time scale, and check 1 h n A n f → Af for any f ∈ C ∞ 0 .

Original: Given Lemma A.5, it is sufﬁcient to check the convergence of the coefﬁcients and ∆ ϵn .
Revised: Given Lemma A.5, it is sufﬁcient to check the convergence of the coefﬁcients and ∆ ϵn .

Original: We start with ∆ ϵn ( x ) .
Revised: We start with ∆ ϵn ( x ) .

Original: Given that the randomness in the Markov chain have bounded moments (uniform in n ), then by a Markov inequality we have that for any q >Π n( [Equation]
Revised: Given that the randomness in the Markov chain have bounded moments (uniform in n ), then by a Markov inequality we have that for any q >Π n

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)b

Original: [Equation]
Revised: [Equation]

Original: ) therefore choosing q > 1 we have sup | x |≤ R ∆ ϵn ( x ) =
Revised: therefore choosing q > 1 we have sup | x |≤ R ∆ ϵn ( x ) =

Original: O ( n − 2 p ( q − 1) )
Revised: O ( n − 2 p ( q − 1) )

Original: → 0 for any ﬁxed ϵ .
Revised: → 0 for any ﬁxed ϵ .

Original: We can rewrite b n ( x ) as
Revised: We can rewrite b n ( x ) as

Original: [Equation]
Revised: [Equation]

Original: since ξ nℓ has zero mean and the remainder terms have bounded moments (uniform in n ), which also gives the desired convergence of sup x ≤| R | | b n ( x ) − b ( x ) | → 0 .
Revised: since ξ nℓ has zero mean and the remainder terms have bounded moments (uniform in n ), which also gives the desired convergence of sup x ≤| R | | b n ( x ) − b ( x ) | → 0 .

Original: Similarly we can rewrite a n ( x ) as
Revised: Similarly we can rewrite a n ( x ) as

Original: [Equation]
Revised: [Equation]

Original:  where we note the drift’s randomness contributes the higher order n − 2 p term and therefore also vanishes in the limit.
Revised: ) where we note the drift’s randomness contributes the higher order n − 2 p term and therefore also vanishes in the limit.

Original: This implies sup x ≤| R | ∥ a n ( x ) − a ( x ) ∥ op → 0 , which gives us the desired result.
Revised: This implies sup x ≤| R | ∥ a n ( x ) − a ( x ) ∥ op → 0 , which gives us the desired result.

Original: B Unshaped ReLU Markov Chain
Revised: B Unshaped ReLU Markov Chain

Original: In this section, we will derive the Markov chain update (2.10) with explicit coefﬁcients.
Revised: In this section, we will derive the Markov chain update (2.10) with explicit coefﬁcients.

Original: For the rest of this section, we will adopt the following notation.
Revised: For the rest of this section, we will adopt the following notation.

Original: Let φ ( x ) :
Revised: Let φ ( x ) :

Original: =
Revised: =

Original: max( x, 0) be the ReLU activation function.
Revised: max( x, 0) be the ReLU activation function.

Original: Let f ( x ) = 1 √ 2 π e − x 2 / 2 be the density of a standard Gaussian, and let F ( x )
Revised: Let f ( x ) = 1 √ 2 π e − x 2 / 2 be the density of a standard Gaussian, and let F ( x )

Original: =
Revised: =

Original: R x −∞ f ( t ) dt be the cumulative distribution function (CDF).
Revised: R x −∞ f ( t ) dt be the cumulative distribution function (CDF).

Original: Lemma B.1 (Gaussian Integration-by-Parts with Indicator Function) .
Revised: Lemma B.1 (Gaussian Integration-by-Parts with Indicator Function) .

Original: For g ∼ N (0 , 1) and h is weakly differentiable, we have that
Revised: For g ∼ N (0 , 1) and h is weakly differentiable, we have that

Original: [Equation]
Revised: [Equation]

Original: where f is the standard Gaussian density.
Revised: where f is the standard Gaussian density.

Original: Proof.
Revised: Proof.

Original: We start by writing the expectation as an integral
Revised: We start by writing the expectation as an integral

Original: [Equation]
Revised: [Equation]

Original: Here by observing that f ′ ( x ) = − xf
Revised: Here by observing that f ′ ( x ) = − xf

Original: ( x ) , we can use integration by parts for u = h ( x ) , dv = xf ( x ) dx to get du
Revised: ( x ) , we can use integration by parts for u = h ( x ) , dv = xf ( x ) dx to get du

Original: = h ′ ( x ) dx, v = − f ( x ) , and therefore Z∞
Revised: = h ′ ( x ) dx, v = − f ( x ) , and therefore Z

Original: [Equation]
Revised: [Equation]

Original: Finally we recover the desired result using symmetry of f ( − a ) =
Revised: Finally we recover the desired result using symmetry of f ( − a ) =

Original: f ( a ) .
Revised: f ( a ) .

Original: We will note the special case of a = 0 to get
Revised: We will note the special case of a = 0 to get

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2 (Gaussian Density Substitution) .
Revised: Lemma B.2 (Gaussian Density Substitution) .

Original: Let g ∼ N (0 , 1) , ρ ∈ [0 , 1] , q = p 1 − ρ 2 , then we have that
Revised: Let g ∼ N (0 , 1) , ρ ∈ [0 , 1] , q = p 1 − ρ 2 , then we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will again write the expectation as an integral
Revised: We will again write the expectation as an integral

Original: [Equation]
Revised: [Equation]

Original: Here observe that
Revised: Here observe that

Original: [Equation]
Revised: [Equation]

Original: at this point, we can complete the square to write
Revised: at this point, we can complete the square to write

Original: [Equation]
Revised: [Equation]

Original: This implies that we have
Revised: This implies that we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we can use the substitution y
Revised: Finally [Equation] we can use the substitution y

Original: = x
Revised: = x

Original: + aρq , dy = 1 q dx to get 
Revised: + aρq , dy = 1 q dx to get Z

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: We will start by calculating simpler quantities.
Revised: We will start by calculating simpler quantities.

Original: Lemma B.3 (Moments) .
Revised: Lemma B.3 (Moments) .

Original: Let g ∼ N (0 , 1) , then
Revised: Let g ∼ N (0 , 1) , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the second and fourth moments, we simply observe that g 2 is symmetric and φ is exactly half of of the integral.
Revised: For the second and fourth moments, we simply observe that g 2 is symmetric and φ is exactly half of of the integral.

Original: For the ﬁrst integral we will use Gaussian integration-by-parts with h ( g ) =
Revised: For the ﬁrst integral we will use Gaussian integration-by-parts with h ( g ) =

Original: to get
Revised: to get

Original: [Equation]
Revised: [Equation]

Original: which is the desire result.
Revised: which is the desire result.

Original: We will also recall the following result from [39]
Revised: We will also recall the following result from [41]

Original: Lemma B.4 ( ¯ J 0 , ¯ J 1 , ¯ J 2 ) .
Revised: Lemma B.4 ( ¯ J 0 , ¯ J 1 , ¯ J 2 ) .

Original: Let ρ ∈ [0 , 1] , q = p 1 − ρ 2 and let ρ, w ∼ N (0 , 1) be independent.
Revised: Let ρ ∈ [0 , 1] , q = p 1 − ρ 2 and let ρ, w ∼ N (0 , 1) be independent.

Original: Then we have that
Revised: Then we have that

Original: [Equation]
Revised: [Equation]

Original: We will need to compute the following quantity.
Revised: We will need to compute the following quantity.

Original: Lemma B.5 ( ¯ J 3 , 1 ) .
Revised: Lemma B.5 ( ¯ J 3 , 1 ) .

Original: Let ρ ∈ [0 , 1] , q = p 1 − ρ 2 and let ρ, w ∼ N (0 , 1) be independent.
Revised: Let ρ ∈ [0 , 1] , q = p 1 − ρ 2 and let ρ, w ∼ N (0 , 1) be independent.

Original: Then we have that
Revised: Then we have that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: B.14)

Original: Proof.
Revised: Proof.

Original: We start by using Gaussian integration-by-parts with h ( g ) =
Revised: We start by using Gaussian integration-by-parts with h ( g ) =

Original: E g g 2 φ ( ρg + qw ) where we use E g [ · ] := E [ · | g ] to denote conditional expectation
Revised: E g g 2 φ ( ρg + qw ) where we use E g [ · ] := E [ · | g ] to denote conditional expectation

Original: [Equation]
Revised: [Equation]

Original: Here we observe that E g g 1 ρq + qw> 0 = gF ( ρg/q ) , and we can again set this to the new h ( g ) and use integration-by-parts to write
Revised: Here we observe that E g g 1 ρq + qw> 0 = gF ( ρg/q ) , and we can again set this to the new h ( g ) and use integration-by-parts to write

Original: [Equation]
Revised: [Equation]

Original: At this point we can use the substitution formula from Lemma B.2 to write
Revised: At this point we can use the substitution formula from Lemma B.2 to write

Original: [Equation]
Revised: [Equation]

Original: Putting this together, we have
Revised: Putting this together, we have

Original: [Equation]
Revised: [Equation]

Original: which is the desired result after simplifying.
Revised: which is the desired result after simplifying.

Original: We will now recall the ReLU-like activations for s =
Revised: We will now recall the ReLU-like activations for s =

Original: ( s + , s − ) ∈ R
Revised: ( s + , s − ) ∈ R

Original: [Equation]
Revised: [Equation]

Original: 19) where φ ( x ) :=
Revised: [Equation] where φ ( x ) :

Original:  max( x, 0) is the usual ReLU activation.
Revised: = max( x, 0) is the usual ReLU activation.

Original: We will compute several basic moments ﬁrst.
Revised: We will compute several basic moments ﬁrst.

Original: Lemma B.6 (Moments, c, M 2 ) .
Revised: Lemma B.6 (Moments, c, M 2 ) .

Original: Let g ∼ N (0 , 1) , we have that
Revised: Let g ∼ N (0 , 1) , we have that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, this implies the normalizing constant is c =
Revised: Furthermore, this implies the normalizing constant is c =

Original: [Equation]
Revised: [Equation]

Original: s 2+

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: To start we ﬁrst recall the Gaussian integration by parts calculation
Revised: To start we ﬁrst recall the Gaussian integration by parts calculation

Original: [Equation]
Revised: [Equation]

Original: then the ﬁrst moment follows immediately from rewriting in terms of φ
Revised: then the ﬁrst moment follows immediately from rewriting in terms of φ

Original: [Equation]
Revised: [Equation]

Original: For the second moment, we will also rewrite in terms of φ
Revised: For the second moment, we will also rewrite in terms of φ

Original: [Equation]
Revised: [Equation]

Original: where we used that φ ( g ) φ ( − g ) = 0 almost surely and g d = − g [Equation] and the desire result follows from
Revised: where we used that φ ( g ) φ ( − g ) = 0 almost surely and g d = − g , and the desire result follows from

Original: Gaussian integration by parts
Revised: Gaussian integration by parts

Original: [Equation]
Revised: [Equation]

Original: For the fourth moment [Equation] we will similarly observe that all mixed moments φ ( g ) p φ ( − g ) r = 0 almost surely whenever p, r > 0 , which allows us to write[Equation]
Revised: For the fourth moment, we will similarly observe that all mixed moments φ ( g ) p φ ( − g ) r = 0 almost surely whenever p, r > 0 , which allows us to write

Original: [Equation]
Revised: [Equation]

Original: and the desire result follows from the Gaussian integration by parts calculation
Revised: and the desire result follows from the Gaussian integration by parts calculation

Original: [Equation]
Revised: [Equation]

Original: We will also convert the ¯ J k,ℓ formulas to K k,ℓ formulas, i.e. the following quantities
Revised: We will also convert the ¯ J k,ℓ formulas to K k,ℓ formulas, i.e. the following quantities

Original: [Equation]
Revised: [Equation]

Original:  where g, w ∼ N (0 , 1) and we deﬁne ˆ g = ρg + qw with q = p 1 − ρ 2 .
Revised: 27) where g, w ∼ N (0 , 1) and we deﬁne ˆ g = ρg + qw with q = p 1 − ρ 2 .

Original: We will also use the short hand notation to write ¯ J p :
Revised: We will also use the short hand notation to write ¯ J p :

Original: =
Revised: =

Original: ¯ J p,p , K p :=
Revised: ¯ J p,p , K p :=

Original: K p,p .
Revised: K p,p .

Original: Lemma B.7 ( K 1 , K 2 , K 3 , 1 ) .
Revised: Lemma B.7 ( K 1 , K 2 , K 3 , 1 ) .

Original: Let ρ ∈ [ − 1 , 1] , q = p [Equation] [Equation] g 
Revised: Let ρ ∈ [ − 1 , 1] , q = p 1 − ρ 2 , g, w ∼ N (0 , 1) , and ˆ g = ρg + qw .

Original: [Equation]

Original: ∼ N (

Original: [Equation]

Original: , and ˆ g = ρg + qw .

Original: Then we have the following formulas
Revised: Then we have the following formulas

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Before we start, we will make several observations.
Revised: Before we start, we will make several observations.

Original: Using the fact that ( g, w ) d = ( ± g, ± w ) , we have the following equality in distribution relations
Revised: Using the fact that ( g, w ) d = ( ± g, ± w ) , we have the following equality in distribution relations

Original: [Equation]
Revised: [Equation]

Original: In particular, we note that the two Gaussian random variable ( g, − ˆ g ) have correlation − ρ .
Revised: In particular [Equation] we note that the two Gaussian random variable ( g, − ˆ g ) have correlation − ρ .

Original: This allows us to simplify K
Revised: This allows us to simplify K

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: With K
Revised: With K

Original: [Equation]
Revised: [Equation]

Original: we will additionally make use of the fact that φ ( g ) φ ( − g ) = 0 almost surely to write
Revised: we will additionally make use of the fact that φ ( g ) φ ( − g ) = 0 almost surely to write

Original: [Equation]
Revised: [Equation]

Original: K 3 , 1 follows from a similar calculation
Revised: K 3 , 1 follows from a similar calculation

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] we to get to state the desired formulas for the approximate Markov chain.
Revised: Finally, we to get to state the desired formulas for the approximate Markov chain.

Original: Here we will make introduce several deﬁnitions ﬁrst.
Revised: Here we will make introduce several deﬁnitions ﬁrst.

Original: In the event that | φ αℓ | = 0 or | φ βℓ | = 0 , the formula ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ
Revised: In the event that | φ αℓ | = 0 or | φ βℓ | = 0 , the formula ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ αℓ | is undeﬁned.
Revised: || φ αℓ | is undeﬁned.

Original: We will remedy this by introducing an additional point e in the state space R ∪ { e } , and set ρ αβℓ =
Revised: We will remedy this by introducing an additional point e in the state space R ∪ { e } , and set ρ αβℓ =

Original: e in this event.
Revised: e in this event.

Original: We note that once ρ αβℓ =
Revised: We note that once ρ αβℓ =

Original: e , then the next step ρ αβℓ +

Revised: e , then the next step ρ αβℓ +

Original: =
Revised: =

Original: e as well since either z αℓ +1 , z βℓ +1 = 0 .
Revised: e as well since either z αℓ +1 , z βℓ +1 = 0 .

Original: For all x ∈ R we will deﬁne the distance | x − e | = ∞ .
Revised: For all x ∈ R we will deﬁne the distance | x − e | = ∞ .

Original: Consequently, R ∪ { e } is a Polish space (complete separable metric space), and therefore it’s a well behaved probability space (e.g. admits conditional densities).
Revised: Consequently, R ∪ { e } is a Polish space (complete separable metric space), and therefore it’s a well behaved probability space (e.g. admits conditional densities).

Revised: For a random variable X , we write

Revised: X =

Revised: O ( n p ) if all moments of n − p X are bounded by a constant independent of n .

Original: We will also deﬁne the bounded Lipschitz function norm as
Revised: We will also deﬁne the bounded Lipschitz function norm as

Original: [Equation]
Revised: [Equation]

Original: which induces the bounded Lipschitz distance for probability measures
Revised: which induces the bounded Lipschitz distance for probability measures

Original: [Equation]
Revised: [Equation]

Original: Proposition B.8 (Unshaped ReLU Correlation) .
Revised: Proposition B.8 (Unshaped ReLU Correlation) .

Original: Let ρ αβℓ := ⟨ φ α ℓ ,φ β ℓ ⟩ | φ αℓ
Revised: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ αℓ | when deﬁned, and e when either | φ αℓ | , | φ αℓ | = 0 .
Revised: || φ αℓ | when deﬁned, and e when either | φ αℓ | , | φ αℓ | = 0 .

Original: Let us also deﬁne the approximate Markov chain
Revised: Let us also deﬁne the approximate Markov chain

Original: [Equation]
Revised: [Equation]

Original: where z ℓ are iid N (0 , 1) and
Revised: where z ℓ are iid N (0 , 1) and

Original: [Equation]
Revised: [Equation]

Original: where we write K · = K · ( ρ αβℓ ) , and the formulas for K 1 , K 2 , K 3 , 1 , c, M 2 are calculated in
Revised: where we write K · = K · ( ρ αβℓ ) , and the formulas for K 1 , K 2 , K 3 , 1 , c, M 2 are calculated in

Original: Lemma B.6 and Lemma B.7.
Revised: Lemma B.6 and Lemma B.7.

Original: Let Π( x, dy ) , P ( x, dy ) be the Markov transition kernels of ρ αβℓ and p ℓ respectively, then 
Revised: Let Π( x, dy ) , P ( x, dy ) be the Markov transition kernels of ρ αβℓ and p ℓ respectively, then d

Original: [Equation]
Revised: [Equation]

Revised: for all x ∈ R ∪ { e }

Revised: [Equation]

Revised: B.37)

Original: Remark B.9 .
Revised: Remark B.9 .

Original: The inﬁnite-width ( n → ∞ ) approximation of the Markov chain corresponds to the update q ℓ +1 = cK 1 ( q ℓ ) , and this is an O ( n − 1 / 2 ) approximation to the chain { ρ αβℓ } .
Revised: The inﬁnite-width ( n → ∞ ) approximation of the Markov chain corresponds to the update q ℓ +1 = cK 1 ( q ℓ ) , and this is an O ( n − 1 / 2 ) approximation to the chain { ρ αβℓ } .

Original: On the other hand, the { p ℓ } chain we propose is an improved approximation up to the zero mean terms up to
Revised: On the other hand, the { p ℓ } chain we propose is an improved approximation up to the zero mean terms up to

Original: O ( n − 1 / 2 ) , and the expected value of non-zero mean terms up to O ( n − 1 ) .
Revised: O ( n − 1 / 2 ) , and the expected value of non-zero mean terms up to O ( n − 1 ) .

Original: In the SDE limit of
Revised: In the SDE limit of

Original: Proposition A.6, these are exactly the terms that do not vanish, which leads us to speculate that this approximation is sufﬁciently close when studying the inﬁnite-depth-and-width limit.
Revised: Proposition A.6, these are exactly the terms that do not vanish, which leads us to speculate that this approximation is sufﬁciently close when studying the inﬁnite-depth-and-width limit.

Original: We will also note that O ( n − 1 ) error in the result arise from replacing the O ( n − 1 / 2 ) with a Gaussian due to Berry–Esseen, and the O ( n − 1 ) term with its expectation, as these are the dominant error terms in the approximation.
Revised: We will also note that O ( n − 1 ) error in the result arise from replacing the O ( n − 1 / 2 ) with a Gaussian due to Berry–Esseen, and the O ( n − 1 ) term with its expectation, as these are the dominant error terms in the approximation.

Original: Proof.
Revised: Proof.

Original: We start by deﬁning the notations
Revised: We start by deﬁning the notations

Original: [Equation]
Revised: [Equation]

Original:  and using positive homogeneity we can write φ s ( p
Revised: ) and using positive homogeneity we can write φ s ( p

Original: [Equation]
Revised: [Equation]

Original:  | [Equation] ) , which gives us 
Revised: p cn | φ αℓ | φ s ( g αℓ ) , which gives us ⟨ φ αℓ +1 , φ βℓ +1 ⟩

Original: [Equation]
Revised: [Equation]

Original: Now consider the same case for R ααℓ and R ββℓ with K 1 (1) =
Revised: Now consider the same case for R ααℓ and R ββℓ with K 1 (1) =

Original: c − 1 , we also getρ αβℓ +
Revised: c − 1 , we also get

Original: [Equation]

Original: = 8<

Original: [Equation]
Revised: [Equation]

Original: if | φ αℓ +1 | , | φ βℓ +1 | > 0 , e , otherwise.

Original: (B.40)

Original: We observe that whenever | φ ℓα | > 0 , we have that 1 + 1 √ n R ααℓ = | φ αℓ +1 | 2 | φ αℓ | 2 ≥ 0 .
Revised: We observe that whenever | φ ℓα | > 0 , we have that 1 + 1 √ n R ααℓ = | φ αℓ +1 | 2 | φ αℓ | 2 ≥ 0 .

Original: Therefore the event
Revised: Therefore the event

Original: E := { R ααℓ , R ββℓ ≤ −√ n } is the same as { ρ αβℓ +

Revised: E := { R ααℓ , R ββℓ ≤ −√ n } is the same as { ρ αβℓ +

Original: =
Revised: =

Original: e } , which is equivalent to when z αℓ +1 or z βℓ +1 has only non-positive entries.
Revised: e } , which is equivalent to when z αℓ +1 or z βℓ +1 has only non-positive entries.

Original: When conditioned on the previous layer, all the entries are independent, this event has probability Π( x, { e } )
Revised: When conditioned on the previous layer, all the entries are independent, this event has probability Π( x, { e } )

Original: =
Revised: =

Original: O (2 − n ) .
Revised: O (2 − n ) .

Original: We will see later that modifying this Markov chain to remove this event will incur only a "minor cost" of O (2 − n ) .
Revised: We will see later that modifying this Markov chain to remove this event will incur only a "minor cost" of O (2 − n ) .

Revised: Let us ﬁx any realization of R αα , R ββ , R αβ outside of the event E (i.e. by viewing it as a map

Original:  Taylor expansion with respect to 1 / √ n about 0{+
+} (Taylor expansion done using SymPy [51] Python package)
Revised: R αα : Ω → R from the probability space for some ﬁxed ω ∈ Ω ), we can compute the Taylor expansion with respect to 1 / √ n about {+
+} (Taylor expansion done using SymPy [51] Python package)

Original: [Equation]
Revised: [Equation]

Original: where  .
Revised: where we recall the X = O ( n − 3 / 2 ) notation denotes a random variable (the Taylor remainder term) where all moments of n 3 / 2 X are bounded by a constant independent of n .

Original: We can simplify these terms further by computing the mean and variance of the expansion (without conditioning on E c ).
Revised: We can simplify these terms further by computing the mean and variance of the expansion (without conditioning on E c ).

Original: More speciﬁcally, each of the R ααℓ , R ββℓ , R αβℓ have zero mean and covariance
Revised: More speciﬁcally, each of the R ααℓ , R ββℓ , R αβℓ have zero mean and covariance

Original: [Equation]
Revised: [Equation]

Original: where we recall Cov ℓ is the conditional covariance given the sigma-algebra F ℓ generated by the ℓ -th layer [ z αℓ ] mα =1 .
Revised: where we recall Cov ℓ is the conditional covariance given the sigma-algebra F ℓ generated by the ℓ -th layer [ z αℓ ] mα =1 .

Original: We can now recover the desired result by calculating the drift and variance coefﬁcients using SymPy [51] again
Revised: We can now recover the desired result by calculating the drift and variance coefﬁcients using SymPy [51] again

Original: [Equation]
Revised: [Equation]

Original: where we recall E ℓ [ ·
Revised: where we recall E ℓ [ ·

Original: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra F ℓ generated by the ℓ -th layer [ z αℓ ] mα =1 .
Revised: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra F ℓ generated by the ℓ -th layer [ z αℓ ] mα =1 .

Original: This allows us to write (considering the well deﬁned case)
Revised: This allows us to write (considering the well deﬁned case)

Original: [Equation]
Revised: [Equation]

Original: where ξ ℓ is has zero mean and unit variance (when not conditioned on ρ αβℓ +

Revised: where ξ ℓ is has zero mean and unit variance (when not conditioned on ρ αβℓ +

Original: = e ), and η ( ρ αβℓ ) has zero mean.
Revised: = e ), and η ( ρ αβℓ ) has zero mean.

Original: Observe that there are three differences between { ρ αβℓ } and the approximate chain { p ℓ } :
Revised: Observe that there are three differences between { ρ αβℓ } and the approximate chain { p ℓ } :

Original: ρ αβℓ +1 =
Revised: ρ αβℓ +1 =

Original: e with probability O (2 − n ) , 2.
Revised: e with probability O (2 − n ) , 2.

Original: ξ ℓ is replaced by z ℓ ∼ N (0 , 1) ,3.
Revised: ξ ℓ is replaced by z ℓ ∼ N (0 , 1) ,

Original: η ( ρ αβℓ ) and the higher order O ( n − 3 / 2 ) terms in the Taylor expansion are removed.
Revised: η ( ρ αβℓ ) and the higher order O ( n − 3 / 2 ) terms in the Taylor expansion are removed.

Original: To complete the proof, we will need to control these differences in terms of the bounded Lipschitz distance on the Markov transition kernels.
Revised: To complete the proof, we will need to control these differences in terms of the bounded Lipschitz distance on the Markov transition kernels.

Original: To this goal, we let h be such that ∥ h ∥ BL ≤ 1 , hence it must be both bounded by 1 and at worst 1 -Lipschitz.
Revised: To this goal, we let h be such that ∥ h ∥ BL ≤ 1 , hence it must be both bounded by 1 and at worst 1 -Lipschitz.

Original: We will ﬁrst condition on E c to write the
Revised: We will ﬁrst condition on E c to write the

Original: Taylor expansion, and then “uncondition” to recover the original distribution, both at a cost of an
Revised: Taylor expansion, and then “uncondition” to recover the original distribution, both at a cost of an

Original: O (2 − n ) error term.
Revised: O (2 − n ) error term.

Original: More precisely, we will write
Revised: More precisely, we will write

Original: [Equation]
Revised: [Equation]

Original: B

Original: [Equation]

Original: [Equation]

Original: where we recall E ℓ [ ·
Revised: where we recall E ℓ [ ·

Original: ] = E [ · |F ℓ ] [Equation] and we deﬁne P ℓ ( E ) :
Revised: ] = E [ · |F ℓ ] , and we deﬁne P ℓ ( E ) :

Original: = E ℓ 1 E .
Revised: = E ℓ 1 E .

Original: At this point we observe that we can now “uncondition” the Taylor expansion by essentially doing the same trick [Equation] or more precisely observe that
Revised: At this point we observe that we can now “uncondition” the Taylor expansion by essentially doing the same trick, or more precisely observe that

Original: [Equation]
Revised: [Equation]

Original: therefore we can write
Revised: therefore we can write

Original: [Equation]
Revised: [Equation]

Original: Since h is 1 -Lipschitz, we have that h ( x + y ) ≤ h ( x ) + | y | , and therefore we can write
Revised: Since h is 1 -Lipschitz, we have that h ( x + y ) ≤ h ( x ) + | y | , and therefore we can write

Original: [Equation]
Revised: [Equation]

Original: Observe that the ﬁrst term is exactly the transition kernel of p ℓ applied to h , i.e. E ℓ h ( p ℓ +1 )
Revised: Observe that the ﬁrst term is exactly the transition kernel of p ℓ applied to h , i.e. E ℓ h ( p ℓ +1 )

Original: =
Revised: =

Original: R h ( y ) P ( p ℓ , dy ) , which means it’s sufﬁcient to control the leftover terms at order O ( n − 1 ) for a chosen coupling of ξ ℓ and z ℓ .
Revised: R h ( y ) P ( p ℓ , dy ) , which means it’s sufﬁcient to control the leftover terms at order O ( n − 1 ) for a chosen coupling of ξ ℓ and z ℓ .

Original: Since clearly E ℓ η ( ρ αβℓ )
Revised: Since clearly E ℓ η ( ρ αβℓ )

Original: =
Revised: =

Original: O (1) as it does not depend on n , we just need to show E ℓ | ξ ℓ − z ℓ | =
Revised: O (1) as it does not depend on n , we just need to show E ℓ | ξ ℓ − z ℓ | =

Original: O ( n − 1 / 2 ) .
Revised: O ( n − 1 / 2 ) .

Original: Observe that by deﬁnition, we have ξ ℓ =
Revised: Observe that by deﬁnition, we have ξ ℓ =

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: n X
Revised: n X

Original: [Equation]
Revised: [Equation]

Original: where the terms of the sum are iid with zero mean and unit variance (since each neuron is independent conditioned on the previous layer).
Revised: where the terms of the sum are iid with zero mean and unit variance (since each neuron is independent conditioned on the previous layer).

Original: Therefore, we can invoke a standard L 1 Berry–Esseen bound, e.g. Theorem 4.2 of [ chen2011normal ].
Revised: Therefore, we can invoke a standard L 1 Berry–Esseen bound, e.g. Theorem 4.2 of [ chen2011normal ].

Original: In this case, we let F be the CDF of ξ ℓ and G be the CDF of z ℓ , and by duality of L 1 (equation 4.6 of [ chen2011normal ]) we have that
Revised: In this case, we let F be the CDF of ξ ℓ and G be the CDF of z ℓ , and by duality of L 1 (equation 4.6 of [ chen2011normal ]) we have that

Original: [Equation]
Revised: [Equation]

Original: where the inf is over all couplings of ξ ℓ , z ℓ .
Revised: where the inf is over all couplings of ξ ℓ , z ℓ .

Original: Finally since the above results do not depend on the choice of the test function h , so we have that
Revised: Finally since the above results do not depend on the choice of the test function h , so we have that

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: C Proofs for ReLU Shaping Results
Revised: C Proofs for ReLU Shaping Results

Original: In this section, we ﬁrst recall the ReLU-like activation function for s =
Revised: In this section, we ﬁrst recall the ReLU-like activation function for s =

Original: ( s + , s − ) ∈ R 2 deﬁned as
Revised: ( s + , s − ) ∈ R 2 deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where φ ( x ) :
Revised: where φ ( x ) :

Original: = max( x, 0) is the usual ReLU activation.
Revised: = max( x, 0) is the usual ReLU activation.

Original: We will also recall the deﬁnitions
Revised: We will also recall the deﬁnitions

Original: [Equation]
Revised: [Equation]

Original: where g, w are iid N (0 , 1) and we deﬁne ˆ g = ρg + qw with q = p 1 − ρ 2 .
Revised: where g, w are iid N (0 , 1) and we deﬁne ˆ g = ρg + qw with q = p 1 − ρ 2 .

Original: We will also use the short hand notation to write ¯ J p :
Revised: We will also use the short hand notation to write ¯ J p :

Original: =
Revised: =

Original: ¯ J p,p , K p :=
Revised: ¯ J p,p , K p :=

Original: K p,p .
Revised: K p,p .

Original: Here we recall from [39]
Revised: Here we recall from [41]

Original: [Equation]
Revised: [Equation]

Original: We will also recall from Lemma B.6 the following moment calculations
Revised: We will also recall from Lemma B.6 the following moment calculations

Original: [Equation]
Revised: [Equation]

Original: In the shaped case, we will calculate a Taylor expansion for the function cK 1 ( ρ ) .
Revised: In the shaped case, we will calculate a Taylor expansion for the function cK 1 ( ρ ) .

Original: Lemma C.1 (Shaping Correlation Function Expansion) .
Revised: Lemma C.1 (Shaping Correlation Function Expansion) .

Original: Let s ±
Revised: Let s ±

Original: = 1 + c ± √ n , then
Revised: = 1 + c ± √ n , then

Original: [Equation]
Revised: [Equation]

Original: where ν ( ρ ) =
Revised: where ν ( ρ ) =

Original: ( c + − c − ) 2 2 π
Revised: ( c + − c − ) 2 2 π

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by consider plugging in the formula from (C.4) to get
Revised: We start by consider plugging in the formula from (C.4) to get

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that arccos( − ρ )
Revised: where we used the fact that arccos( − ρ )

Original: = π − arccos( ρ ) .
Revised: = π − arccos( ρ ) .

Original: After substituting s ± = 1+ c ± √ n , we can use SymPy [51] to Taylor expand with respect to the variable x = n − 1 / 2 about x 0 = 0 and get
Revised: After substituting s ± = 1+ c ± √ n , we can use SymPy [51] to Taylor expand with respect to the variable x = n − 1 / 2 about x 0 = 0 and get

Original: [Equation]
Revised: [Equation]

Original: where we used the simplify function on the coefﬁcients to reduce the size of the expression.
Revised: where we used the simplify function on the coefﬁcients to reduce the size of the expression.

Original: We can further simplify to get
Revised: We can further simplify to get

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: We will also need an approximation result for fourth moments.
Revised: We will also need an approximation result for fourth moments.

Original: Lemma C.2 (Fourth Moment Approximation) .
Revised: Lemma C.2 (Fourth Moment Approximation) .

Original: Let g α , g β , g γ , g δ ∈ R be jointly Gaussian such that
Revised: Let g α , g β , g γ , g δ ∈ R be jointly Gaussian such that

Original: [Equation]
Revised: [Equation]

Original: and similarly for other pairs of α, β, γ, δ .
Revised: and similarly for other pairs of α, β, γ, δ .

Original: Then
Revised: Then

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where the constant in the O ( · ) notation is universal.
Revised: where the constant in the O ( · ) notation is universal.

Original: Proof.
Revised: Proof.

Original: We start by writing
Revised: We start by writing

Original: [Equation]
Revised: [Equation]

Original: and this allows us to write
Revised: and this allows us to write

Original: [Equation]
Revised: [Equation]

Original: δ + O ( n − 1 / 2 ) .
Revised: δ + O ( n − 1 / 2 ) .

Original: (C.12)
Revised: (C.12)

Original: Then by Isserlis’ Theorem, we can write
Revised: Then by Isserlis’ Theorem, we can write

Original: [Equation]
Revised: [Equation]

Original: which gives us the desired result.
Revised: which gives us the desired result.

Original: We will also calculate a useful covariance.
Revised: We will also calculate a useful covariance.

Original: Lemma C.3 (Covariance of R αβ ) .
Revised: Lemma C.3 (Covariance of R αβ ) .

Original: Let g α , g β , g γ , g δ ∈ R n be jointly Gaussian vectors such that (cid:20)
Revised: Let g α , g β , g γ , g δ ∈ R n be jointly Gaussian vectors such that (cid:20)

Original: [Equation]
Revised: [Equation]

Original: and similarly for other pairs of α, β, γ, δ .
Revised: and similarly for other pairs of α, β, γ, δ .

Original: If we also deﬁne
Revised: If we also deﬁne

Original: [Equation]
Revised: [Equation]

Original: then we have the following covariance formula:
Revised: then we have the following covariance formula:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst observe that since each entry of the sum in R αβ are iid and zero mean, it is sufﬁcient to just compute the covariance a single term.
Revised: We ﬁrst observe that since each entry of the sum in R αβ are iid and zero mean, it is sufﬁcient to just compute the covariance a single term.

Original: In other words
Revised: In other words

Original: [Equation]
Revised: [Equation]

Original: Since c = 1 + O ( n − 1 / 2 ) and K 1 ( ρ ) =
Revised: Since c = 1 + O ( n − 1 / 2 ) and K 1 ( ρ ) =

Original: ρ + O ( n − 1 ) from Lemma C.1, we can further write this as
Revised: ρ + O ( n − 1 ) from Lemma C.1, we can further write this as

Original: [Equation]
Revised: [Equation]

Original: and we can use the fourth moment approximation Lemma C.2 to get
Revised: and we can use the fourth moment approximation Lemma C.2 to get

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: C.1 Proof of Theorem 3.2 (Covariance SDE, ReLU)
Revised: C.1 Proof of Theorem 3.2 (Covariance SDE, ReLU)

Original: We start by restating the theorem.
Revised: We start by restating the theorem.

Original: Theorem C.4 (Covariance SDE, ReLU) .
Revised: Theorem C.4 (Covariance SDE, ReLU) .

Original: Let V αβℓ :
Revised: Let V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ , and deﬁne V ℓ :=
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .

Original: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /dV
Revised: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /dV

Original: t =
Revised: t =

Original: b ( V t ) dt + Σ( V t ) 1 / 

Revised: b ( V t ) dt + Σ( V t ) 1 /

Original: dB t
Revised: dB t

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: n in ⟨ x α , x β ⟩ (cid:21) 1 ≤ α ≤ β ≤ m
Revised: n in ⟨ x α , x β ⟩ (cid:21) 1 ≤ α ≤ β ≤ m

Original: [Equation]
Revised: [Equation]

Original: (C.20) where we denote ν ( ρ ) :
Revised: (C.20) where we denote ν ( ρ ) :

Original: = ( c + − c − ) 2 2 π (cid:16)p
Revised: = ( c + − c − ) 2 2 π (cid:16)p

Original: [Equation]
Revised: [Equation]

Original: − ρ
Revised: − ρ

Original: [Equation]
Revised: [Equation]

Original: − ρ
Revised: − ρ

Original: [Equation]
Revised: [Equation]

Original: ρ (cid:17) , ρ αβt :=
Revised: ρ (cid:17) , ρ αβt :=

Original: V αβt √
Revised: V αβt √

Original: V ααt V ββt and write b ( V t ) =
Revised: V ααt V ββt and write b ( V t ) =

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the output distribution can be described conditional on V T evaluated at ﬁnal time T
Revised: Furthermore, the output distribution can be described conditional on V T evaluated at ﬁnal time T

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by recalling the deﬁnitions
Revised: We start by recalling the deﬁnitions

Original: [Equation]
Revised: [Equation]

Original: At this point, we can deﬁne
Revised: At this point, we can deﬁne

Original: [Equation]
Revised: [Equation]

Original: and observe that
Revised: and observe that

Original: [Equation]
Revised: [Equation]

Original: where F ℓ is the sigma-algebra generated by [ z αℓ ] mα =1 , ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ
Revised: where F ℓ is the sigma-algebra generated by [ z αℓ ] mα =1 , ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ

Original: αℓ || φ βℓ | , and ⊗ denotes the Kronecker product.
Revised: αℓ || φ βℓ | , and ⊗ denotes the Kronecker product.

Original: Then we can use positive homogeneity (i.e. φ s ( cx ) =
Revised: Then we can use positive homogeneity (i.e. φ s ( cx ) =

Original: | c | φ s ( x ) ) to write
Revised: | c | φ s ( x ) ) to write

Original: [Equation]
Revised: [Equation]

Original: where we deﬁned R αβℓ := 1 √ n P ni =
Revised: where we deﬁned R αβℓ := 1 √ n P ni =

Original: [Equation]
Revised: [Equation]

Original: Next we use the expansion of cK 1 ( ρ αβℓ ) from Lemma C.1 to write
Revised: Next we use the expansion of cK 1 ( ρ αβℓ ) from Lemma C.1 to write

Original: [Equation]
Revised: [Equation]

Original: ) which essentially recovers the Markov chain form we want from Proposition A.6, where the drift is
Revised: ) which essentially recovers the Markov chain form we want from Proposition A.6, where the drift is

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: It remains to simply compute the covariance conditioned on previous layer.
Revised: It remains to simply compute the covariance conditioned on previous layer.

Original: To this end, we will use
Revised: To this end, we will use

Original: Lemma C.3 to write
Revised: Lemma C.3 to write

Original: [Equation]
Revised: [Equation]

Original: where we recall E ℓ [ ·
Revised: where we recall E ℓ [ ·

Original: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .
Revised: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .

Original: By setting σ = Σ 1 / 2 , we then recover the desired SDE via Proposition A.6 on the
Revised: By setting σ = Σ 1 / 2 , we then recover the desired SDE via Proposition A.6 on the

Original: Markov chain of V αβℓ .
Revised: Markov chain of V αβℓ .

Original: C.2 Proof of Theorem 3.3 (Correlation SDE, ReLU)
Revised: C.2 Proof of Theorem 3.3 (Correlation SDE, ReLU)

Original: We start by restating the theorem.
Revised: We start by restating the theorem.

Original: Theorem C.5 (Correlation SDE, ReLU) .
Revised: Theorem C.5 (Correlation SDE, ReLU) .

Original: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ
Revised: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ βℓ | , where φ αℓ :=
Revised: || φ βℓ | , where φ αℓ :=

Original: φ s ( z αℓ ) .
Revised: φ s ( z αℓ ) .

Original: In the limit as n → ∞ and s ± = 1 + c ± √ n , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R
Revised: In the limit as n → ∞ and s ± = 1 + c ± √ n , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: While it is possible to obtain this result as a consequence of Theorem 3.2 via Itô’s Lemma, we will show an alternative derivation by extending the steps of Proposition B.8, where we can directly compute the Taylor expansion in the event E := {| φ αℓ +1 |
Revised: While it is possible to obtain this result as a consequence of Theorem 3.2 via Itô’s Lemma, we will show an alternative derivation by extending the steps of Proposition B.8, where we can directly compute the Taylor expansion in the event E := {| φ αℓ +1 |

Original: [Equation]
Revised: [Equation]

Original: | φ βℓ +1 | >
Revised: | φ βℓ +1 | >

Original: [Equation]
Revised: [Equation]

Original: where (unconditioned on E ) ξ ℓ are iid with mean zero variance one and
Revised: where (unconditioned on E ) ξ ℓ are iid with mean zero variance one and

Original: [Equation]
Revised: [Equation]

Original: where we replaced µ ReLU , σ ReLU with µ, σ as we will be shaping the activation function, and we recall E ℓ [ · ]
Revised: where we replaced µ ReLU , σ ReLU with µ, σ as we will be shaping the activation function, and we recall E ℓ [ · ]

Original: = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .
Revised: = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .

Original: We note that the undeﬁned event E occurs only when z αℓ +1 or z βℓ +1 has all negative entries, which occurs with probability O (2 − n ) .
Revised: We note that the undeﬁned event E occurs only when z αℓ +1 or z βℓ +1 has all negative entries, which occurs with probability O (2 − n ) .

Original: Since all the terms of interest have ﬁnite moments, we can proceed by removing this event E in a similar fashion as Proposition B.8.
Revised: Since all the terms of interest have ﬁnite moments, we can proceed by removing this event E in a similar fashion as Proposition B.8.

Original: Using the expansion of cK 1 ( ρ ) from Lemma C.1, we can now write
Revised: Using the expansion of cK 1 ( ρ ) from Lemma C.1, we can now write

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we also have that by Lemma C.1 and Lemma C.
Revised: Furthermore, we also have that by Lemma C.1 and Lemma C.

Original: [Equation]
Revised: [Equation]

Original: which gives us the desired formula of
Revised: which gives us the desired formula of

Original: [Equation]
Revised: [Equation]

Original: Finally, we can recover the desired SDE via Proposition A.6.
Revised: Finally, we can recover the desired SDE via Proposition A.6.

Original: C.3 Joint Correlation SDE
Revised: C.3 Joint Correlation SDE

Original: In this section, we will extend Theorem 3.3 to a general joint process over all the possible pairs of correlations.
Revised: In this section, we will extend Theorem 3.3 to a general joint process over all the possible pairs of correlations.

Original: Theorem C.6 (Joint Correlation SDE) .
Revised: Theorem C.6 (Joint Correlation SDE) .

Original: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ
Revised: Let ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ βℓ | , and deﬁne ρ ℓ :=
Revised: || φ βℓ | , and deﬁne ρ ℓ :=

Original: [ ρ αβℓ ] 1 ≤ α ≤ β
Revised: [ ρ αβℓ ] 1 ≤ α ≤ β

Original: = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .
Revised: = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .

Original: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process ρ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +
Revised: Then, with s ± = 1 + c ± √ n as in Deﬁnition 3.1, in the limit as n → ∞ , dn → T , the interpolated process ρ ⌊ tn ⌋ converges in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the coefﬁcients are deﬁned by
Revised: where the coefﬁcients are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: with ν, µ deﬁned as in Theorem 3.3.
Revised: with ν, µ deﬁned as in Theorem 3.3.

Original: Proof.
Revised: Proof.

Original: It’s sufﬁcient to just compute the covariance matrix Σ for the random terms of the Markov chain (B.41), which reduces down to
Revised: It’s sufﬁcient to just compute the covariance matrix Σ for the random terms of the Markov chain (B.41), which reduces down to

Original: [Equation]
Revised: [Equation]

Original: where we recall E ℓ [ ·
Revised: where we recall E ℓ [ ·

Original: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 , and we write K αβ:= K 1 ( ρ αβℓ ) .
Revised: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 , and we write K αβ:= K 1 ( ρ αβℓ ) .

Original: Using Lemma C.1 and Lemma C.3, we can calculate this explicitly as
Revised: Using Lemma C.1 and Lemma C.3, we can calculate this explicitly as

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: C.4 Proof for Proposition 3.4 (Critical Exponent, ReLU)
Revised: C.4 Proof for Proposition 3.4 (Critical Exponent, ReLU)

Original: We start by restating the proposition.
Revised: We start by restating the proposition.

Original: Proposition C.7 (Critical Exponent, ReLU) .
Revised: Proposition C.7 (Critical Exponent, ReLU) .

Original: Let ρ αβ ℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ
Revised: Let ρ αβ ℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ αℓ

Original: || φ βℓ | , where φ α ℓ := φ s ( z α ℓ ) .
Revised: || φ βℓ | , where φ α ℓ := φ s ( z α ℓ ) .

Original: Consider the limit n → ∞ and s ± = 1 + c ± n p for some p ≥ 0 .
Revised: Consider the limit n → ∞ and s ± = 1 + c ± n p for some p ≥ 0 .

Original: Then depending on the value of p , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution w.r.t.
Revised: Then depending on the value of p , the interpolated process ρ αβ ⌊ tn ⌋ converges in distribution w.r.t.

Original: the Skorohod topology of D R + , R to
Revised: the Skorohod topology of D R + , R to

Original: (i) the degenerate limit: ρ αβt = 1 for all t > 0 , if 0 ≤ p < 12 , and c + ̸ = c − , (ii) the critical limit: the SDE from Theorem 3.3, if p = 12 , (iii) the linear network limit: if p > 12 , the following SDE,
Revised: (i) the degenerate limit: ρ αβt = 1 for all t > 0 , if 0 ≤ p < 12 , and c + ̸ = c − , (ii) the critical limit: the SDE from Theorem 3.3, if p = 12 , (iii) the linear network limit: if p > 12 , the following SDE,

Original: with µ, σ as deﬁned in (3.5) ,
Revised: with µ, σ as deﬁned in (3.5) ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Case (ii) follows from Theorem 3.3, therefore it is sufﬁcient to only consider cases (i) and (iii).
Revised: Case (ii) follows from Theorem 3.3, therefore it is sufﬁcient to only consider cases (i) and (iii).

Original: In the case that p = 0 , we can recover the following recursion in the limit as n → ∞
Revised: In the case that p = 0 , we can recover the following recursion in the limit as n → ∞

Original: [Equation]
Revised: [Equation]

Original: which matches the inﬁnite-width limit, and it is known that ρ αβℓ → 1 as ℓ → 1 (see also Appendix E for an upper bound).
Revised: which matches the inﬁnite-width limit, and it is known that ρ αβℓ → 1 as ℓ → 1 (see also Appendix E for an upper bound).

Original: Next we will recall the result of Lemma C.1 and observe that we can simply replace √ n with n p to recover the expansion
Revised: Next we will recall the result of Lemma C.1 and observe that we can simply replace √ n with n p to recover the expansion

Original: [Equation]
Revised: [Equation]

Original: This gives us the following Markov chain from the proof of Theorem 3.
Revised: This gives us the following Markov chain from the proof of Theorem 3.

Original: [Equation]
Revised: [Equation]

Original: In the case that 0 < p < 1 / 2 , we can consider the time step size h n = n − 2 p instead of n − 1 and apply Proposition A.6, where we recover the ODE
Revised: In the case that 0 < p < 1 / 2 , we can consider the time step size h n = n − 2 p instead of n − 1 and apply Proposition A.6, where we recover the ODE

Original: [Equation]
Revised: [Equation]

Original: but on the time scale of ˆ ρ αβ,ns =
Revised: but on the time scale of ˆ ρ αβ,ns =

Original: ˆ ρ αβ ⌊ sn 2 p ⌋ .
Revised: ˆ ρ αβ ⌊ sn 2 p ⌋ .

Original: Converting it back to the time scale of ρ αβ,nt =
Revised: Converting it back to the time scale of ρ αβ,nt =

Original: ρ αβ ⌊ tn ⌋ implies that we have
Revised: ρ αβ ⌊ tn ⌋ implies that we have

Original: [Equation]
Revised: [Equation]

Original: for all
Revised: for all

Original: [Equation]
Revised: [Equation]

Original: > 0 .
Revised: > 0 .

Original: [Equation]
Revised: [Equation]

Original: And since ν ( ρ ) > 0 for all ρ < 1 and that ν ( ρ ) = C (1 − ρ ) 3 / 2 + O ((1 − ρ ) 5 / 2 ) as ρ → 1 , we have that ˆ ρ αβ ∞ = 1 as desired.
Revised: And since ν ( ρ ) > 0 for all ρ < 1 and that ν ( ρ ) = C (1 − ρ ) 3 / 2 + O ((1 − ρ ) 5 / 2 ) as ρ → 1 , we have that ˆ ρ αβ ∞ = 1 as desired.

Original: In the case p > 12 , we have that since ν is deterministic, we observe the drift term used in Proposition A.6 in the limit as n → ∞ is
Revised: In the case p > 12 , we have that since ν is deterministic, we observe the drift term used in Proposition A.6 in the limit as n → ∞ is

Original: [Equation]
Revised: [Equation]

Original: which would simply recover the desired SDE with drift µ only.
Revised: which would simply recover the desired SDE with drift µ only.

Original: D Proofs for Smooth Shaping Results
Revised: D Proofs for Smooth Shaping Results

Original: In this section, we consider smooth activation functions φ satisfying Assumption 3.5, that is φ ∈
Revised: In this section, we consider smooth activation functions φ satisfying Assumption 3.5, that is φ ∈

Original: C 4 , φ (0) = 0 , φ ′ (0)
Revised: C 4 , φ (0) = 0 , φ ′ (0)

Original: = 1 , and that | φ (4) ( x ) | ≤ C (1 + | x | p ) for some C, p > 0 .
Revised: = 1 , and that | φ (4) ( x ) | ≤ C (1 + | x | p ) for some C, p > 0 .

Original: We recall the shaping we consider for activations of this type is via the following deﬁnition for s >
Revised: We recall the shaping we consider for activations of this type is via the following deﬁnition for s >

Original: [Equation]
Revised: [Equation]

Original: so that lim s →∞ φ s ( x ) = x .
Revised: so that lim s →∞ φ s ( x ) = x .

Original: Before we start, we will calculate the behaviour of the normalizing constant c up an error order of s 3 .
Revised: Before we start, we will calculate the behaviour of the normalizing constant c up an error order of s 3 .

Original: Lemma D.1.
Revised: Lemma D.1.

Original: Let φ s be deﬁned as above with φ satisfying Assumption 3.5.
Revised: Let φ s be deﬁned as above with φ satisfying Assumption 3.5.

Original: Then if g ∼ N (0 , 1) , we have that
Revised: Then if g ∼ N (0 , 1) , we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will ﬁrst Taylor expand φ s ( g ) about g =
Revised: We will ﬁrst Taylor expand φ s ( g ) about g =

Original: [Equation]
Revised: [Equation]

Original: where we note by Assumption 3.5 the remainder term is at most polynomial in g .
Revised: where we note by Assumption 3.5 the remainder term is at most polynomial in g .

Original: Therefore the second moment satisﬁes
Revised: Therefore the second moment satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where O ( s − 3 ) is bounded due to Gaussians have all bounded moments.
Revised: where O ( s − 3 ) is bounded due to Gaussians have all bounded moments.

Original: Therefore, for s > 0 sufﬁciently small, we have the following expansion
Revised: Therefore, for s > 0 sufﬁciently small, we have the following expansion

Original: [Equation]
Revised: [Equation]

Original: where b = 34 φ ′′ (0) 2 + φ ′′′ (0) , which is the desired result.
Revised: where b = 34 φ ′′ (0) 2 + φ ′′′ (0) , which is the desired result.

Original: D.1 Proof of Theorem 3.9 (Covariance SDE, Smooth)
Revised: D.1 Proof of Theorem 3.9 (Covariance SDE, Smooth)

Original: We start by restating the theorem.
Revised: We start by restating the theorem.

Original: Theorem D.2 (Covariance SDE, Smooth) .
Revised: Theorem D.2 (Covariance SDE, Smooth) .

Original: Let φ satisfy Assumption 3.5, V αβℓ :
Revised: Let φ satisfy Assumption 3.5, V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) , and deﬁne V ℓ :=
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector in R m ( m +1) / 2 .

Original: Then, with s = a √ n as in Deﬁnition 3.6, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /
Revised: Then, with s = a √ n as in Deﬁnition 3.6, in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution to the solution of the following SDE in the Skorohod topology of D R + , R m ( m +1) /

Original: [Equation]
Revised: [Equation]

Original: where Σ( V t ) is the same as Theorem 3.2 and
Revised: where Σ( V t ) is the same as Theorem 3.2 and

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if V T is ﬁnite, then the output distribution can be described conditional on V T as
Revised: Furthermore, if V T is ﬁnite, then the output distribution can be described conditional on V T as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and otherwise the distribution of [ z α out ] mα =1 is undeﬁned.
Revised: and otherwise the distribution of [ z α out ] mα =1 is undeﬁned.

Original: Proof.
Revised: Proof.

Original: We start by deﬁning g αℓ := W ℓ φ αℓ | φ αℓ | , and observe that
Revised: We start by deﬁning g αℓ := W ℓ φ αℓ | φ αℓ | , and observe that

Original: [Equation]
Revised: [Equation]

Original: where F ℓ is the sigma-algebra generated by the ℓ -th layer [ z αℓ ] mα =1 , ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ
Revised: where F ℓ is the sigma-algebra generated by the ℓ -th layer [ z αℓ ] mα =1 , ρ αβℓ := ⟨ φ αℓ ,φ βℓ ⟩ | φ

Original: αℓ || φ βℓ | , and ⊗ denotes the Kronecker product.
Revised: αℓ || φ βℓ | , and ⊗ denotes the Kronecker product.

Original: We can then write the Taylor expansion for φ s about 0 as
Revised: We can then write the Taylor expansion for φ s about 0 as

Original: [Equation]
Revised: [Equation]

Original: where R 3 ( · ) is the Taylor remainder term, which has polynomial growth by Assumption 3.5.
Revised: where R 3 ( · ) is the Taylor remainder term, which has polynomial growth by Assumption 3.5.

Original: By using the fact that φ (0) = 0 , φ ′ (0) = 1 and observing that the derivatives of φ s satisﬁes φ ( k ) s (0) =
Revised: By using the fact that φ (0) = 0 , φ ′ (0) = 1 and observing that the derivatives of φ s satisﬁes φ ( k ) s (0) =

Original: φ ( k ) (0) s k − 1 , we can further write
Revised: φ ( k ) (0) s k − 1 , we can further write

Original: [Equation]
Revised: [Equation]

Original: where the remainder term is at most polynomial in g αℓ,i .
Revised: where the remainder term is at most polynomial in g αℓ,i .

Original: Then we can compute the inner product with the same expansion as c
Revised: Then we can compute the inner product with the same expansion as c

Original: [Equation]
Revised: [Equation]

Original: and we will proceed by analyzing the product terms separately.
Revised: and we will proceed by analyzing the product terms separately.

Original: We start with the terms of order
Revised: We start with the terms of order

Original: O ( s 0 ) ﬁrst, which are
Revised: O ( s 0 ) ﬁrst, which are

Original: [Equation]
Revised: [Equation]

Original: where we used the deﬁnitions V αβℓ :
Revised: where we used the deﬁnitions V αβℓ :

Original: = cn ⟨ φ αℓ , φ βℓ ⟩ and R αβℓ := 1 √ n P ni =1 g αℓ,i g βℓ,i − ρ αβℓ .
Revised: = cn ⟨ φ αℓ , φ βℓ ⟩ and R αβℓ := 1 √ n P ni =1 g αℓ,i g βℓ,i − ρ αβℓ .

Original: For the ﬁrst order terms, i.e., terms of order O ( s − 1 ) , we have the terms
Revised: For the ﬁrst order terms, i.e., terms of order O ( s − 1 ) , we have the terms

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne b R αβℓ : = P ni =1 g αℓ,i g βℓ,i(cid:18)p
Revised: where we deﬁne b R αβℓ := 1 √ n P ni =1 g αℓ,i g βℓ,i

Original: [Equation]
Revised: [Equation]

Original: g 
Revised: g αℓ

Original: [Equation]
Revised: [Equation]

Original: gβℓ
Revised: g

Original: [Equation]
Revised: [Equation]

Original: and observe this random variable has zero mean and a ﬁnite variance.
Revised: and observe this random variable has zero mean and a ﬁnite variance.

Original: Therefore in view of Proposition A.6, this term cannot contribute to the drift due to having zero mean, nor can this term contribute to the diffusion term due to s = a √ n leading to the term being order 1 n .
Revised: Therefore in view of Proposition A.6, this term cannot contribute to the drift due to having zero mean, nor can this term contribute to the diffusion term due to s = a √ n leading to the term being order 1 n .

Original: In other words, the effect of this term will vanish in the limit as n → ∞ .
Revised: In other words, the effect of this term will vanish in the limit as n → ∞ .

Original: We then turn our attention to the second order terms, i.e., terms of order O ( s − 2 )
Revised: We then turn our attention to the second order terms, i.e., terms of order O ( s − 2 )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since this term is order s − 2 = 1 a 2 n , it can only contribute to the drift term, and in view of Proposition A.6, we only need to compute its mean.
Revised: Since this term is order s − 2 = 1 a 2 n , it can only contribute to the drift term, and in view of Proposition A.6, we only need to compute its mean.

Original: To this goal, we will simply invoke Isserlis’ Theorem and calculate
Revised: To this goal, we will simply invoke Isserlis’ Theorem and calculate

Original: [Equation]
Revised: [Equation]

Original: where we recall E ℓ [ ·
Revised: where we recall E ℓ [ ·

Original: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .
Revised: ] = E [ · |F ℓ ] is the conditional expectation given the sigma-algebra generated by { z αℓ } mα =1 .

Original: This allows us to compute the conditional expectation E ℓ for the terms of order s − 2 as
Revised: This allows us to compute the conditional expectation E ℓ for the terms of order s − 2 as

Original: [Equation]
Revised: [Equation]

Original: Putting these terms together with the fact that c = 1 − bs 2 + O ( s − 3 ) with b = 34 φ ′′ (0) 2 + φ ′′′ (0) , we can write the update rule for V αβℓ as
Revised: Putting these terms together with the fact that c = 1 − bs 2 + O ( s − 3 ) with b = 34 φ ′′ (0) 2 + φ ′′′ (0) , we can write the update rule for V αβℓ as

Original: [Equation]
Revised: [Equation]

Original: At this point, we have fully recovered the drift term, and we observe the covariance structure is the same as Lemma C.3 in the limit as n → ∞ .
Revised: At this point, we have fully recovered the drift term, and we observe the covariance structure is the same as Lemma C.3 in the limit as n → ∞ .

Original: Therefore we can invoke Proposition A.6 to recover the desired SDE.
Revised: Therefore we can invoke Proposition A.6 to recover the desired SDE.

Original: D.2 Proof of Proposition 3.10 (Critical Exponent, Smooth)
Revised: D.2 Proof of Proposition 3.10 (Critical Exponent, Smooth)

Original: We will restate and prove the proposition.
Revised: We will restate and prove the proposition.

Original: Proposition D.3 (Critical Exponent, Smooth) .
Revised: Proposition D.3 (Critical Exponent, Smooth) .

Original: Let φ satisfy Assumption 3.5, V αβℓ :
Revised: Let φ satisfy Assumption 3.5, V αβℓ :

Original: = cn ⟨ φ α ℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) with s = an p for some p > 0 , and deﬁne V ℓ :=
Revised: = cn ⟨ φ α ℓ , φ βℓ ⟩ where φ αℓ = φ s ( z αℓ ) with s = an p for some p > 0 , and deﬁne V ℓ :=

Original: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector.
Revised: [ V αβℓ ] 1 ≤ α ≤ β = m to be the upper triangular entries thought of as a vector.

Original: Then in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution w.r.t.
Revised: Then in the limit as n → ∞ , dn → T , the interpolated process V ⌊ tn ⌋ converges locally in distribution w.r.t.

Original: the Skorohod topology of D R + , R m ( m +1) / 2 to V , which depending on the value of p is
Revised: the Skorohod topology of D R + , R m ( m +1) / 2 to V , which depending on the value of p is

Original: [Equation]
Revised: [Equation]

Original: for all t > 0 and 1 ≤ α ≤ β ≤ m , (ii) the critical limit: the solution of the SDE from Theorem 3.9, if p = 12 , (iii) the linear network limit: the stopped solution to the SDE dV t = Σ( V t ) dB t with coefﬁcient Σ deﬁned in Theorem 3.3, if p > 12 .
Revised: for all t > 0 and 1 ≤ α ≤ β ≤ m , (ii) the critical limit: the solution of the SDE from Theorem 3.9, if p = 12 , (iii) the linear network limit: the stopped solution to the SDE dV t = Σ( V t ) dB t with coefﬁcient Σ deﬁned in Theorem 3.3, if p > 12 .

Original: Proof.
Revised: Proof.

Original: Similar to the proof of Theorem 3.
Revised: Similar to the proof of Theorem 3.

Original: [Equation]
Revised: [Equation]

Original: we will borrow the same notation and write down the
Revised: we will borrow the same notation and write down the

Original: Markov chain update and consider the time scale depending on the value of p .
Revised: Markov chain update and consider the time scale depending on the value of p .

Original: In case (i) where 0 < p <
Revised: In case (i) where 0 < p <

Original: [Equation]
Revised: [Equation]

Original: we will consider the time scale h n = 1 s 2 = 1 a 2 n 2 p and observe that based on the Taylor expansion of φ s about
Revised: we will consider the time scale h n = 1 s 2 = 1 a 2 n 2 p and observe that based on the Taylor expansion of φ s about

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne R ααℓ := 1 √ n P ni
Revised: where we deﬁne R ααℓ := 1 √ n P ni

Original: =1 ( g αℓ,i ) 2 − 1 , b R ααℓ := 1 √ n P ni =1 ( g αℓ,i ) 3 , e R ααℓ := 1 √ n P ni =1 ( g αℓ,i ) 4 − 3 and observe they all have zero mean and ﬁnite variance.
Revised: =1 ( g αℓ,i ) 2 − 1 , b R ααℓ := 1 √ n P ni =1 ( g αℓ,i ) 3 , e R ααℓ := 1 √ n P ni =1 ( g αℓ,i ) 4 − 3 and observe they all have zero mean and ﬁnite variance.

Original: In view of the time scale s − 2 for Proposition A.6, it is then only important to keep track of the expected value of the s − 2 terms and the covariance of the s − 1 terms.
Revised: In view of the time scale s − 2 for Proposition A.6, it is then only important to keep track of the expected value of the s − 2 terms and the covariance of the s − 1 terms.

Original: However, since there is no terms on the order of s − 1 , we essentially have
Revised: However, since there is no terms on the order of s − 1 , we essentially have

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that c = 1 − bs 2 + O ( s − 3 ) for b
Revised: where we used the fact that c = 1 − bs 2 + O ( s − 3 ) for b

Original: = φ ′′′ (0) + 34 φ ′′ (0) 2 from Lemma D.1.
Revised: = φ ′′′ (0) + 34 φ ′′ (0) 2 from Lemma D.1.

Original: Hence, we have that U αα,nt :=
Revised: Hence, we have that U αα,nt :=

Original: [Equation]
Revised: [Equation]

Original: αα ⌊ ts 2 ⌋ converging to the ODE via Proposition A.
Revised: αα ⌊ ts 2 ⌋ converging to the ODE via Proposition A.

Original: [Equation]
Revised: [Equation]

Original: where we observe if b > 0 this ODE is “mean avoiding” as it will drift towards 0 or ∞ .
Revised: where we observe if b > 0 this ODE is “mean avoiding” as it will drift towards 0 or ∞ .

Original: And since the V t time scale is on the order of 1 n , for all t > 0 we have that
Revised: And since the V t time scale is on the order of 1 n , for all t > 0 we have that

Original: [Equation]
Revised: [Equation]

Original: therefore if b > 0 we have that V ααt = 0 or ∞ as desired in the ﬁrst case of (i).
Revised: therefore if b > 0 we have that V ααt = 0 or ∞ as desired in the ﬁrst case of (i).

Original: When b = 0 we observe that V ααt =
Revised: When b = 0 we observe that V ααt =

Original: V ααsince the time derivative is zero.
Revised: V ααsince the time derivative is zero.

Original: Furthermore if b < 0 we also have that
Revised: Furthermore if b < 0 we also have that

Original: V ααt = 1 in the second case of (i).
Revised: V ααt = 1 in the second case of (i).

Original: When b ≤ 0 , we can also write down the ODE for U αβt using a similar argument and keeping only the s − 2 terms.
Revised: When b ≤ 0 , we can also write down the ODE for U αβt using a similar argument and keeping only the s − 2 terms.

Original: More precisely, we can modify (D.18) to get
Revised: More precisely, we can modify (D.18) to get

Original: [Equation]
Revised: [Equation]

Original: which leads to the following ODE
Revised: which leads to the following ODE

Original: [Equation]
Revised: [Equation]

Original: Since U ααt , U ββt converge to constants as t → ∞ , | U αβt | ≤ q
Revised: Since U ααt , U ββt converge to constants as t → ∞ , | U αβt | ≤ q

Original: U ααt U ββt by deﬁnition and Cauchy–
Revised: U ααt U ββt by deﬁnition and Cauchy–

Original: Schwarz inequality, and that U αβt satisﬁes a ﬁrst order ODE (so it cannot have a periodic solution), we must also have that lim t →∞ U αβt = const.
Revised: Schwarz inequality, and that U αβt satisﬁes a ﬁrst order ODE (so it cannot have a periodic solution), we must also have that lim t →∞ U αβt = const.

Original: This completes the proof for case (i).
Revised: This completes the proof for case (i).

Original: Case (ii) follows directly from Theorem 3.9, therefore we can then consider case (iii) with the same Taylor expansion, however this time on the time scale of n − 1 instead.
Revised: Case (ii) follows directly from Theorem 3.9, therefore we can then consider case (iii) with the same Taylor expansion, however this time on the time scale of n − 1 instead.

Original: We will again follow
Revised: We will again follow

Original: Proposition A.6 to only track the mean of the order n − 1 term and the variance of the n − 1 / 2 term.
Revised: Proposition A.6 to only track the mean of the order n − 1 term and the variance of the n − 1 / 2 term.

Original: Since p > 12 , the only term that remains is the diffusion on the order of n − 1 /
Revised: Since p > 12 , the only term that remains is the diffusion on the order of n − 1 /

Original: [Equation]
Revised: [Equation]

Original: (D.
Revised: (D.

Original: [Equation]
Revised: [Equation]

Original: which gives us the desired SDE from calculating the covariance from Theorem 3.9.
Revised: which gives us the desired SDE from calculating the covariance from Theorem 3.9.

Original: D.3 Proof of Proposition 3.7 (Finite Time Explosion Criterion)
Revised: D.3 Proof of Proposition 3.7 (Finite Time Explosion Criterion)

Original: We will start by recalling several deﬁnitions from [42, Section 5.5].
Revised: We will start by recalling several deﬁnitions from [42, Section 5.5].

Original: Firstly, we consider the one dimensional Itô diffusion on I :=
Revised: Firstly, we consider the one dimensional Itô diffusion on I :=

Original: (0 , ∞ )
Revised: (0 , ∞ )

Original: [Equation]
Revised: [Equation]

Original: where the drift and diffusion coefﬁcients satisfy the following conditions
Revised: where the drift and diffusion coefﬁcients satisfy the following conditions

Original: [Equation]
Revised: [Equation]

Original: We will also deﬁne the following functions for some ﬁxed c ∈
Revised: We will also deﬁne the following functions for some ﬁxed c ∈

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: We will also deﬁne the following sequence of stopping times for M >
Revised: We will also deﬁne the following sequence of stopping times for M >

Original: [Equation]
Revised: [Equation]

Original: and let τ ∗ :
Revised: and let τ ∗ :

Original: = sup M> 0 τ M .
Revised: = sup M> 0 τ M .

Original: Now we will state the main results we need for ﬁnite time explosions.
Revised: Now we will state the main results we need for ﬁnite time explosions.

Original: Lemma D.4 ([42, Problem 5.5.27]) .
Revised: Lemma D.4 ([42, Problem 5.5.27]) .

Original: We have the following implications
Revised: We have the following implications

Original: [Equation]
Revised: [Equation]

Original: Theorem D.5 (Feller’s Test for Explosions [42, Theorem 5.5.29]) .
Revised: Theorem D.5 (Feller’s Test for Explosions [42, Theorem 5.5.29]) .

Original: Assume the conditions in (D.28) are satisﬁed.
Revised: Assume the conditions in (D.28) are satisﬁed.

Original: Then P [ τ ∗ = ∞ ] = 

Revised: Then P [ τ ∗ = ∞ ] =

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: We will begin our derivations for the SDE (D.27).
Revised: We will begin our derivations for the SDE (D.27).

Original: Lemma D.6 (Geometric Brownian Motion, the b = 0 Case) .
Revised: Lemma D.6 (Geometric Brownian Motion, the b = 0 Case) .

Original: Let X t be a solution to the following
Revised: Let X t be a solution to the following

Original: SDE
Revised: SDE

Original: [Equation]
Revised: [Equation]

Original: then we have that τ ∗ =
Revised: then we have that τ ∗ =

Original: ∞
Revised: ∞

Original: a.s.
Revised: a.s.

Original: Proof.
Revised: Proof.

Original: Here we observe that
Revised: Here we observe that

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Then we have that
Revised: Then we have that

Original: [Equation]
Revised: [Equation]

Original: which implies v
Revised: which implies v

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and therefore lim
Revised: and therefore lim

Original: [Equation]
Revised: [Equation]

Original: By Feller’s test for explosions Theorem D.5, we have the desired result.
Revised: By Feller’s test for explosions Theorem D.5, we have the desired result.

Original: Proposition D.7 (Calculate p ( x ) , m ( dx ) and the b ≤ − 1 Case) .
Revised: Proposition D.7 (Calculate p ( x ) , m ( dx ) and the b ≤ − 1 Case) .

Original: Suppose X t is a solution of the following equation
Revised: Suppose X t is a solution of the following equation

Original: [Equation]
Revised: [Equation]

Original: then for all b ̸ = 

Revised: then for all b ̸ =

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: In particular, when b ≤ − 1 , we have that lim x
Revised: In particular, when b ≤ − 1 , we have that lim x

Original: → 0 v ( x ) =
Revised: → 0 v ( x ) =

Original: lim x →∞ v
Revised: lim x →∞ v

Original: [Equation]
Revised: [Equation]

Original: x ) = ∞
Revised: x ) = ∞

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by writing
Revised: We start by writing

Original: [Equation]
Revised: [Equation]

Original: Then we can also calculate the integral via a substitution of y = bx to get the desired result.
Revised: Then we can also calculate the integral via a substitution of y = bx to get the desired result.

Original: At this time, we observe that when b >
Revised: At this time, we observe that when b >

Original: [Equation]
Revised: [Equation]

Original: where γ is the lower incomplete gamma function, and therefore ﬁnite for all values of x including the limits x → 0 , ∞ .
Revised: where γ is the lower incomplete gamma function, and therefore ﬁnite for all values of x including the limits x → 0 , ∞ .

Original: The b = 0 case follows from Lemma D.6.
Revised: The b = 0 case follows from Lemma D.6.

Original: Finally when b < 0 we can write
Revised: Finally when b < 0 we can write

Original: [Equation]
Revised: [Equation]

Original: which clearly diverges to ∞ as x → ∞ .
Revised: which clearly diverges to ∞ as x → ∞ .

Original: On the other hand, we can observe as that as x → 0 , we have that y ∈
Revised: On the other hand, we can observe as that as x → 0 , we have that y ∈

Original: [0 , 1] and therefore 1 ≤ e | b | y ≤ e | b | .
Revised: [0 , 1] and therefore 1 ≤ e | b | y ≤ e | b | .

Original: This implies we only need to consider the integral − R 1 x y −| b | dy , which diverges to −∞ if and only if | b | ≥ 1 .
Revised: This implies we only need to consider the integral − R 1 x y −| b | dy , which diverges to −∞ if and only if | b | ≥ 1 .

Original: In other words we have
Revised: In other words we have

Original: [Equation]
Revised: [Equation]

Original: The limits on v ( x ) follows from Lemma D.4.
Revised: The limits on v ( x ) follows from Lemma D.4.

Original: Proposition D.8 (The b > − 1 Case) .
Revised: Proposition D.8 (The b > − 1 Case) .

Original: Suppose X t is a solution of the following equation
Revised: Suppose X t is a solution of the following equation

Original: [Equation]
Revised: [Equation]

Original: D.45) then when b > − 1 , we have that
Revised: D.45) then when b > − 1 , we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will start by calculating the following integral using the exponential series expansion Z y
Revised: We will start by calculating the following integral using the exponential series expansion Z y

Original: [Equation]
Revised: [Equation]

Original: Now we can compute v ( x ) v ( x ) =
Revised: Now we can compute v ( x ) v ( x ) =

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst consider the case when x → 0 , in which case we have e −| b | ≤ e − by ≤ e | b | and therefore will not affect convergence or divergence, so we can safely ignore the factor e − by and write (for k > 0 , x → 0 ) Z
Revised: We ﬁrst consider the case when x → 0 , in which case we have e −| b | ≤ e − by ≤ e | b | and therefore will not affect convergence or divergence, so we can safely ignore the factor e − by and write (for k > 0 , x → 0 ) Z

Original: [Equation]
Revised: [Equation]

Original: Since the exponential series P k> 0 b k k ! =
Revised: Since the exponential series P k> 0 b k k ! =

Original: e b − 1 converges, and we have terms strictly smaller than the exponential series, we have convergence of these terms when k > 0 .
Revised: e b − 1 converges, and we have terms strictly smaller than the exponential series, we have convergence of these terms when k > 0 .

Original: We now return to handle a couple of edge case terms, ﬁrstly when k =
Revised: We now return to handle a couple of edge case terms, ﬁrstly when k =

Original: [Equation]
Revised: [Equation]

Original: which is a desired behaviour.
Revised: which is a desired behaviour.

Original: Secondly we consider when k = b + 1 Z
Revised: Secondly we consider when k = b + 1 Z

Original: [Equation]
Revised: [Equation]

Original: from which we can conclude lim x → 0 v ( x ) = ∞ .
Revised: from which we can conclude lim x → 0 v ( x ) = ∞ .

Original: Next we consider the case when x → ∞ .
Revised: Next we consider the case when x → ∞ .

Original: Firstly, since we already have that p ( x ) → ∞ when b ≤ 0 , therefore Lemma D.4 implies v ( x ) → ∞ .
Revised: Firstly, since we already have that p ( x ) → ∞ when b ≤ 0 , therefore Lemma D.4 implies v ( x ) → ∞ .

Original: Therefore we only need to consider when b > 0 .
Revised: Therefore we only need to consider when b > 0 .

Original: Since b > 0 we will have that e − bx will dominate, and therefore we can safely ignore all the edge case terms and consider the series
Revised: Since b > 0 we will have that e − bx will dominate, and therefore we can safely ignore all the edge case terms and consider the series

Original: [Equation]
Revised: [Equation]

Original: Observe that as x → ∞ we actually recover the gamma integral in the terms i.e. Z ∞
Revised: Observe that as x → ∞ we actually recover the gamma integral in the terms i.e. Z ∞

Original: [Equation]
Revised: [Equation]

Original: where we observe the second term is independent of k , and therefore the series converges due to comparison with the exponential Taylor series.
Revised: where we observe the second term is independent of k , and therefore the series converges due to comparison with the exponential Taylor series.

Original: This implies we only need to focus on the ﬁrst term, which is
Revised: This implies we only need to focus on the ﬁrst term, which is

Original: [Equation]
Revised: [Equation]

Original: where the series converges since it’s a sum of k − 2 type.
Revised: where the series converges since it’s a sum of k − 2 type.

Original: This allows us to conclude that lim x →∞ v ( x ) < ∞ as desired.
Revised: This allows us to conclude that lim x →∞ v ( x ) < ∞ as desired.

Original: We can now prove the desired result of Proposition 3.7, which we restate below.
Revised: We can now prove the desired result of Proposition 3.7, which we restate below.

Original: Proposition D.9 (Finite Time Explosion) .
Revised: Proposition D.9 (Finite Time Explosion) .

Original: Let X t ∈ R
Revised: Let X t ∈ R

Original: + be a solution to the following SDE
Revised: + be a solution to the following SDE

Original: [Equation]
Revised: [Equation]

Original: Let τ ∗ =
Revised: Let τ ∗ =

Original: sup M> 0 inf { t : X t ≥ M or X t ≤ M − 1 } be the explosion time, and we say X t has a ﬁnite time explosion if τ ∗ < ∞ .
Revised: sup M> 0 inf { t : X t ≥ M or X t ≤ M − 1 } be the explosion time, and we say X t has a ﬁnite time explosion if τ ∗ < ∞ .

Original: For this equation, P [ τ ∗ = ∞ ]
Revised: For this equation, P [ τ ∗ = ∞ ]

Original: = 1 if and only if b ≤ 0 .
Revised: = 1 if and only if b ≤ 0 .

Original: Proof.
Revised: Proof.

Original: Putting the results of Proposition D.7 and Proposition D.8 together, we have the following table
Revised: Putting the results of Proposition D.7 and Proposition D.8 together, we have the following table

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: ∞ ∞ ﬁnite ∞
Revised: ∞ ∞ ﬁnite ∞

Original: [Equation]
Revised: [Equation]

Original: ∞ ﬁnite ﬁnite ﬁnite
Revised: ∞ ﬁnite ﬁnite ﬁnite

Original: Therefore, invoking Feller’s test for explosions from Theorem D.5, we have that P [ τ ∗ = ∞ ] if and only if b ≤ 0 .
Revised: Therefore, invoking Feller’s test for explosions from Theorem D.5, we have that P [ τ ∗ = ∞ ] if and only if b ≤ 0 .

Original: E Lower Bound for the Recursion ρ ℓ +1 = cK 1 ( ρ ℓ )
Revised: E Lower Bound for the Recursion ρ ℓ +1 = cK 1 ( ρ ℓ )

Original: In this section, we consider a Taylor expansion of cK 1 ( ρ ) around ρ → 1 from the left hand side to get
Revised: In this section, we consider a Taylor expansion of cK 1 ( ρ ) around ρ → 1 from the left hand side to get

Original: [Equation]
Revised: [Equation]

Original: which we can rewrite using r ℓ = 1 − ρ ℓ as
Revised: which we can rewrite using r ℓ = 1 − ρ ℓ as

Original: [Equation]
Revised: [Equation]

Original: We will compute an upper bound on r ℓ inspired by the following result.
Revised: We will compute an upper bound on r ℓ inspired by the following result.

Original: Lemma E.1 (Lemma A.6, [52]) .
Revised: Lemma E.1 (Lemma A.6, [52]) .

Original: The logistic recursion
Revised: The logistic recursion

Original: [Equation]
Revised: [Equation]

Original: for x 0 , α ∈ [0 , 1] satisﬁes
Revised: for x 0 , α ∈ [0 , 1] satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We will extend the above Lemma to a slightly modiﬁed update as well.
Revised: We will extend the above Lemma to a slightly modiﬁed update as well.

Original: Lemma E.2.
Revised: Lemma E.2.

Original: Suppose the recursive map satisﬁes
Revised: Suppose the recursive map satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for x 0 ∈ [0 , 1] , then we also have that
Revised: for x 0 ∈ [0 , 1] , then we also have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will start the induction proof at n =
Revised: We will start the induction proof at n =

Original: [Equation]
Revised: [Equation]

Original: When x 0 ≤ 9 we have that
Revised: When x 0 ≤ 9 we have that

Original: [Equation]
Revised: [Equation]

Original: and hence
Revised: and hence

Original: [Equation]
Revised: [Equation]

Original: which proves the case for n = 1 .
Revised: which proves the case for n = 1 .

Original: Then we assume the inequality holds for x n , we will similarly write
Revised: Then we assume the inequality holds for x n , we will similarly write

Original: [Equation]
Revised: [Equation]

Original: and plugging in the inequality for x n we get
Revised: and plugging in the inequality for x n we get

Original: [Equation]
Revised: [Equation]

Original: To complete the proof it’s sufﬁcient to show
Revised: To complete the proof it’s sufﬁcient to show

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Since 2 n 3 + 1 ≥ 2( n +1) , we only need to compare the ﬁrst coefﬁcient, which is
Revised: Since 2 n 3 + 1 ≥ 2( n +1) , we only need to compare the ﬁrst coefﬁcient, which is

Original: [Equation]
Revised: [Equation]

Original: 14) and this is equivalent to n ≥ 1 , and therefore satisﬁed by the induction.
Revised: 14) and this is equivalent to n ≥ 1 , and therefore satisﬁed by the induction.

Original: This completes the proof.
Revised: This completes the proof.

Original: At the same time, we also conjecture the following bound.
Revised: At the same time, we also conjecture the following bound.

Original: Conjecture E.3.
Revised: Conjecture E.3.

Original: Suppose the recursive map satisﬁes
Revised: Suppose the recursive map satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for x 0 ∈ [0 , 1] , then x
Revised: for x 0 ∈ [0 , 1] , then x

Original: [Equation]
Revised: [Equation]

Original: Sketch of Conjecture.
Revised: Sketch of Conjecture.

Original: Suppose we want to establish the approximation of
Revised: Suppose we want to establish the approximation of

Original: [Equation]
Revised: [Equation]

Original: Then for the initial induction n = 1 step, we only need
Revised: Then for the initial induction n = 1 step, we only need

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Using WolframAlpha (probably through the quartic formula), we ﬁnd the desired solution for b ∈ (0 , 1 / 2) is
Revised: Using WolframAlpha (probably through the quartic formula), we ﬁnd the desired solution for b ∈ (0 , 1 / 2) is

Original: [Equation]
Revised: [Equation]

Original: This function b ( x 0 ) is a strictly decreasing function on [0 , 1] , and it satisﬁes b (0) =
Revised: This function b ( x 0 ) is a strictly decreasing function on [0 , 1] , and it satisﬁes b (0) =

Original: 12 , b (1) = √ 2 − 1 .
Revised: 12 , b (1) = √ 2 − 1 .

Original: This implies that whenever x 0 is small, we can choose b closer to 1 2 in the n = 1 step of the induction.
Revised: This implies that whenever x 0 is small, we can choose b closer to 1 2 in the n = 1 step of the induction.

Original: Similarly, for the induction step, it’s sufﬁcient to show (cid:16)
Revised: Similarly, for the induction step, it’s sufﬁcient to show (cid:16)

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Again, since we are always choosing b ≤ 1 / 2 , therefore we have 1 ≥ 2 b , and we will only need to focus on the ﬁrst coefﬁcient.
Revised: Again, since we are always choosing b ≤ 1 / 2 , therefore we have 1 ≥ 2 b , and we will only need to focus on the ﬁrst coefﬁcient.

Original: To this end we rewrite the ﬁrst term as
Revised: To this end we rewrite the ﬁrst term as

Original: [Equation]
Revised: [Equation]

Original: This implies we require n ≥ b 1 − 2 b , which increases as we choose b closer to 1 / 2 .
Revised: This implies we require n ≥ b 1 − 2 b , which increases as we choose b closer to 1 / 2 .

Original: However, if the induction starts the step ⌈ b 1 − 2 b ⌉ , then this is not a problem, which leads to our conjecture.
Revised: However, if the induction starts the step ⌈ b 1 − 2 b ⌉ , then this is not a problem, which leads to our conjecture.

Original: Using the above results, we can have a similar approximation for r ℓ given the inﬁnite-width update
Revised: Using the above results, we can have a similar approximation for r ℓ given the inﬁnite-width update

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which we can rewrite using ˆ r ℓ := (cid:16)
Revised: which we can rewrite using ˆ r ℓ := (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)r ℓ to get
Revised: (cid:17)r ℓ to get

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: This allows us to consider the upper bound
Revised: This allows us to consider the upper bound

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: Similarly, the conjecture leads to the following approximation
Revised: Similarly, the conjecture leads to the following approximation

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: F Additional Simulations and Discussions
Revised: F Additional Simulations and Discussions

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we have additional simulations plotting the densities of ρ αβd and V αβd for shaped
Revised: we have additional simulations plotting the densities of ρ αβd and V αβd for shaped

Original: ReLU-like, sigmoid, and softplus networks.
Revised: ReLU-like, sigmoid, and softplus networks.

Original: In particular, the density of V αβd for ReLU-like networks can be found in Figure 4, the densities for sigmoid in Figure 5, and the densities for softplus in Figure
Revised: In particular, the density of V αβd for ReLU-like networks can be found in Figure 4, the densities for sigmoid in Figure 5, and the densities for softplus in Figure

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F.1 Convergence in Kolmogorov–Smirnov Distance
Revised: F.1 Convergence in Kolmogorov–Smirnov Distance

Original: From Figure
Revised: From Figure

Original: [Figure]
Revised: [Figure]

Original: we can show that our results (Theorem 3.3) converges at a rate of n − 1 / 2 in terms of the KS-distance.
Revised: we can show that our results (Theorem 3.3) converges at a rate of n − 1 / 2 in terms of the KS-distance.

Original: F.2 Tuning Shape and Depth-to
Revised: F.2 Tuning Shape and Depth-to

Original: -Width Ratio
Revised: -Width Ratio

Original: Since the existing shaping methods [40, 41] estimates the output correlation based on the inﬁnitewidth limit, we can easily improve the shape tuning based on the covariance SDEs.
Revised: Since the existing shaping methods [38, 39] estimates the output correlation based on the inﬁnitewidth limit, we can easily improve the shape tuning based on the covariance SDEs.

Original: In particular, we consider the example of ReLU-like activations with correlation described by the SDE (3.4).
Revised: In particular, we consider the example of ReLU-like activations with correlation described by the SDE (3.4).

Original: By simulating both the SDE and the inﬁnite-width limit ODE, we arrive at the results in Figure 8.
Revised: By simulating both the SDE and the inﬁnite-width limit ODE, we arrive at the results in Figure 8.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: d
Revised: d

Original: D e n s i t y
Revised: D e n s i t y

Original: SDE (
Revised: SDE (

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We observe that simply by increasing c − towards zero does not automatically reduce effects on the correlation when time t (the depth-to-width ratio) is large.
Revised: We observe that simply by increasing c − towards zero does not automatically reduce effects on the correlation when time t (the depth-to-width ratio) is large.

Original: In other words, even a linear network will observe an increase in correlation when depth is large enough.
Revised: In other words, even a linear network will observe an increase in correlation when depth is large enough.

Original: Therefore shaping the activation alone is insufﬁcient , but we also need to account for the depth-to-width ratio.
Revised: Therefore shaping the activation alone is insufﬁcient , but we also need to account for the depth-to-width ratio.

Original: We also remark that Figure 8 only plotted the median for simplicity, but if we recall the density plots from Figure 1, correlation is heavily skewed and concentrated near 1 .
Revised: We also remark that Figure 8 only plotted the median for simplicity, but if we recall the density plots from Figure 1, correlation is heavily skewed and concentrated near 1 .

Original: More precisely, while the median correlation is approximately 0 .
Revised: More precisely, while the median correlation is approximately 0 .

Original: 55 , roughly 20% of the samples are larger than 0 .
Revised: 55 , roughly 20% of the samples are larger than 0 .

Original: In other words, one in ﬁve random initializations will lead to a correlation worse than 0 .
Revised: In other words, one in ﬁve random initializations will lead to a correlation worse than 0 .

Original: As a consequence, practitioners implementing the shaping methods of [40, 41] should consider simulating the correlation SDE to account for the heavy skew.
Revised: As a consequence, practitioners implementing the shaping methods of [38, 39] should consider simulating the correlation SDE to account for the heavy skew.