Original: E XPECTED G RADIENTS OF M AXOUT N ETWORKS AND C ONSEQUENCES TO P ARAMETER I NITIALIZATION
Revised: E XPECTED G RADIENTS OF M AXOUT N ETWORKS AND C ONSEQUENCES TO P ARAMETER I NITIALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution.
Revised: We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution.

Original: We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization.
Revised: We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization.

Original: Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks.
Revised: Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks.

Original: Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks.
Revised: Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks.

Original: In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.
Revised: In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We study the gradients of maxout networks and derive several implications for training stability, parameter initialization, and expressivity.
Revised: We study the gradients of maxout networks and derive several implications for training stability, parameter initialization, and expressivity.

Original: Concretely, we compute stochastic order bounds and bounds on the moments depending on the parameter distribution and the network architecture.
Revised: Concretely, we compute stochastic order bounds and bounds on the moments depending on the parameter distribution and the network architecture.

Original: The analysis is based on the input-output Jacobian of maxout networks.
Revised: The analysis is based on the input-output Jacobian of maxout networks.

Original: We discover that, in contrast to ReLU networks, when initialized with a zero-mean Gaussian distribution, the distribution of the input-output Jacobian of a maxout network depends on the network input, which may lead to unstable gradients and training difficulties.
Revised: We discover that, in contrast to ReLU networks, when initialized with a zero-mean Gaussian distribution, the distribution of the input-output Jacobian of a maxout network depends on the network input, which may lead to unstable gradients and training difficulties.

Original: Nonetheless, we can obtain a rigorous parameter initialization recommendation for wide networks.
Revised: Nonetheless, we can obtain a rigorous parameter initialization recommendation for wide networks.

Original: The analysis of gradients also allows us to refine previous bounds on the expected number of linear regions of maxout networks at initialization and derive new results on the length distortion and the NTK.
Revised: The analysis of gradients also allows us to refine previous bounds on the expected number of linear regions of maxout networks at initialization and derive new results on the length distortion and the NTK.

Original: Maxout networks A rank- K maxout unit, introduced by Goodfellow et al.
Revised: Maxout networks A rank- K maxout unit, introduced by Goodfellow et al.

Original: (2013), computes the maximum of K real-valued parametric affine functions.
Revised: (2013), computes the maximum of K real-valued parametric affine functions.

Revised: Concretely, a rank- K maxout unit with n inputs implements a function R n → R ; x (cid:55)→ max k ∈ [ K ] {⟨ W k , x ⟩ + b k } , where W k ∈ R n and b k ∈ R , k ∈ [ K ] := { 1 , . . .

Revised: , K } , are trainable weights and biases.

Revised: The K arguments of the maximum are called the pre-activation features of the maxout unit.

Original: This may be regarded as a multi-argument generalization of a ReLU, which computes the maximum of a real-valued affine function and zero.
Revised: This may be regarded as a multi-argument generalization of a ReLU, which computes the maximum of a real-valued affine function and zero.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2013) demonstrated that maxout networks could perform better than ReLU networks under similar circumstances.
Revised: (2013) demonstrated that maxout networks could perform better than ReLU networks under similar circumstances.

Original: Additionally, maxout networks have been shown to be useful for combating catastrophic forgetting in neural networks (Goodfellow et al., 2015).
Revised: Additionally, maxout networks have been shown to be useful for combating catastrophic forgetting in neural networks (Goodfellow et al., 2015).

Original: On the other hand, Castaneda et al.
Revised: On the other hand, Castaneda et al.

Original: (2019) evaluated the performance of maxout networks in a big data setting and observed that increasing the width of ReLU networks is more effective in improving performance than replacing ReLUs with maxout units and that ReLU networks converge faster than maxout networks.
Revised: (2019) evaluated the performance of maxout networks in a big data setting and observed that increasing the width of ReLU networks is more effective in improving performance than replacing ReLUs with maxout units and that ReLU networks converge faster than maxout networks.

Original: We observe that proper initialization strategies for maxout networks have not been studied in the same level of detail as for ReLU networks and that this might resolve some of the problems encountered in previous maxout network applications.
Revised: We observe that proper initialization strategies for maxout networks have not been studied in the same level of detail as for ReLU networks and that this might resolve some of the problems encountered in previous maxout network applications.

Original: Parameter initialization
Revised: Parameter initialization

Original: The vanishing and exploding gradient problem has been known since the work of Hochreiter (1991).
Revised: The vanishing and exploding gradient problem has been known since the work of Hochreiter (1991).

Original: It makes choosing an appropriate learning rate harder and slows training Sun (2019).
Revised: It makes choosing an appropriate learning rate harder and slows training Sun (2019).

Original: Common approaches to address this difficulty include the choice of specific architectures, e.g. LSTMs (Hochreiter, 1991) or ResNets
Revised: Common approaches to address this difficulty include the choice of specific architectures, e.g. LSTMs (Hochreiter, 1991) or ResNets

Original: (He et al., 2016), and normalization methods such as batch normalization (Ioffe & Szegedy, 2015) or explicit control of the gradient magnitude with gradient clipping (Pascanu et al., 2013).
Revised: (He et al., 2016), and normalization methods such as batch normalization (Ioffe & Szegedy, 2015) or explicit control of the gradient magnitude with gradient clipping (Pascanu et al., 2013).

Original: We will focus on approaches based on parameter initialization that control the activation length and parameter gradients (LeCun et al., 2012; Glorot & Bengio, 2010; He et al., 2015; Gurbuzbalaban & Hu,  2021).
Revised: We will focus on approaches based on parameter initialization that control the activation length and parameter gradients (LeCun et al., 2012; Glorot & Bengio, 2010; He et al., 2015; Gurbuzbalaban & Hu, 2021; Zhang et al., 2019; Bachlechner et al., 2021).

Original: He et al.
Revised: He et al.

Original: (2015) studied forward and backward passes to obtain initialization recommendations for ReLU.
Revised: (2015) studied forward and backward passes to obtain initialization recommendations for ReLU.

Original: A more rigorous analysis of the gradients was performed by Hanin & Rolnick (2018); Hanin (2018), who also considered higher-order moments and derived recommendations on the network architecture.
Revised: A more rigorous analysis of the gradients was performed by Hanin & Rolnick (2018); Hanin (2018), who also considered higher-order moments and derived recommendations on the network architecture.

Original: Sun et al.
Revised: Sun et al.

Original: (2018) derived a corresponding strategy for rank K = 2 maxout networks.
Revised: (2018) derived a corresponding strategy for rank K = 2 maxout networks.

Original: For higher maxout ranks, Tseran & Montúfar (2021) considered balancing the forward pass, assuming Gaussian or uniform distribution on the pre-activation features of each layer.
Revised: For higher maxout ranks, Tseran & Montúfar (2021) considered balancing the forward pass, assuming Gaussian or uniform distribution on the pre-activation features of each layer.

Original: However, this assumption is not fully justified.
Revised: However, this assumption is not fully justified.

Original: We will analyze maxout network gradients, including the higher order moments, and give a rigorous justification for the initialization suggested by Tseran & Montúfar (2021).
Revised: We will analyze maxout network gradients, including the higher order moments, and give a rigorous justification for the initialization suggested by Tseran & Montúfar (2021).

Original: Expected number of linear regions Neural networks with piecewise linear activation functions subdivide their input space into linear regions, i.e., regions over which the computed function is (affine) linear.
Revised: Expected number of linear regions Neural networks with piecewise linear activation functions subdivide their input space into linear regions, i.e., regions over which the computed function is (affine) linear.

Original: The number of linear regions serves as a complexity measure to differentiate network architectures (Pascanu et al., 2014; Montúfar et al., 2014; Telgarsky, 2015; 2016).
Revised: The number of linear regions serves as a complexity measure to differentiate network architectures (Pascanu et al., 2014; Montufar et al., 2014; Telgarsky, 2015; 2016).

Original: The first results on the expected number of linear regions were obtained by Hanin & Rolnick (2019a;b) for
Revised: The first results on the expected number of linear regions were obtained by Hanin & Rolnick (2019a;b) for

Original: ReLU networks, showing that it can be much smaller than the maximum possible number.
Revised: ReLU networks, showing that it can be much smaller than the maximum possible number.

Original: Tseran &
Revised: Tseran &

Original: Montúfar (2021) obtained corresponding results for maxout networks.
Revised: Montúfar (2021) obtained corresponding results for maxout networks.

Original: An important factor controlling the bounds in these works is a constant depending on the gradient of the neuron activations with respect to the network input.
Revised: An important factor controlling the bounds in these works is a constant depending on the gradient of the neuron activations with respect to the network input.

Original: By studying the input-output Jacobian of maxout networks, we obtain a refined bound for this constant and, consequently, the expected number of linear regions.
Revised: By studying the input-output Jacobian of maxout networks, we obtain a refined bound for this constant and, consequently, the expected number of linear regions.

Original: Expected curve distortion Another complexity measure is the distortion of the length of an input curve as it passes through a network.
Revised: Expected curve distortion Another complexity measure is the distortion of the length of an input curve as it passes through a network.

Original: Poole et al.
Revised: Poole et al.

Original: (2016) studied the propagation of Riemannian curvature through wide neural networks using a mean-field approach, and later, a related notion of “trajectory length” was considered by Raghu et al.
Revised: (2016) studied the propagation of Riemannian curvature through wide neural networks using a mean-field approach, and later, a related notion of “trajectory length” was considered by Raghu et al.

Original: It was demonstrated that these measures can grow exponentially with the network depth, which was linked to the ability of deep networks to “disentangle” complex representations.
Revised: It was demonstrated that these measures can grow exponentially with the network depth, which was linked to the ability of deep networks to “disentangle” complex representations.

Original: Based on these notions, Murray et al.
Revised: Based on these notions, Murray et al.

Original: (2022) studies how to avoid rapid convergence of pairwise input correlations, vanishing and exploding gradients.
Revised: (2022) studies how to avoid rapid convergence of pairwise input correlations, vanishing and exploding gradients.

Original: However,
Revised: However,

Original: Hanin et al.
Revised: Hanin et al.

Original: (2021) proved that for a ReLU network with He initialization the length of the curve does not grow with the depth and even shrinks slightly.
Revised: (2021) proved that for a ReLU network with He initialization the length of the curve does not grow with the depth and even shrinks slightly.

Original: We establish similar results for maxout networks.
Revised: We establish similar results for maxout networks.

Original: NTK It is known that the Neural Tangent Kernel (NTK) of a finite network can be approximated by its expectation (Jacot et al., 2018).
Revised: NTK It is known that the Neural Tangent Kernel (NTK) of a finite network can be approximated by its expectation (Jacot et al., 2018).

Original: However, for ReLU networks Hanin & Nica (2020a) showed that if both the depth and width tend to infinity, the NTK does not converge to a constant in probability.
Revised: However, for ReLU networks Hanin & Nica (2020a) showed that if both the depth and width tend to infinity, the NTK does not converge to a constant in probability.

Original: By studying the expectation of the gradients, we show that similarly to ReLU, the NTK of maxout networks does not converge to a constant when both width and depth are sent to infinity.
Revised: By studying the expectation of the gradients, we show that similarly to ReLU, the NTK of maxout networks does not converge to a constant when both width and depth are sent to infinity.

Original: Contributions Our contributions can be summarized as follows.
Revised: Contributions Our contributions can be summarized as follows

Original: • We derive stochastic order bounds for the directional derivative of the input-output map of a deep fully-connected maxout network (Theorem 1) as well as bounds for the moments (Corollary 2).
Revised: • For expected gradients , we derive stochastic order bounds for the directional derivative of the input-output map of a deep fully-connected maxout network (Theorem 1) as well as bounds for the moments (Corollary 2).

Original: In contrast to ReLU networks, in maxout networks, the distribution of the input-output Jacobian depends on the network input.

Original: Additionally, we derive an equality in distribution for the directional derivatives (Theorem 3), based on which we also discuss the moments (Remark 4) in wide networks.
Revised: Additionally, we derive an equality in distribution for the directional derivatives (Theorem 3), based on which we also discuss the moments (Remark 4) in wide networks.

Original: • We  derive the moments of the activation length of a fully-connected maxout network (Corollary 5).
Revised: We further derive the moments of the activation length of a fully-connected maxout network (Corollary 5).

Original: • Further, we identify potential instability sources of maxout networks at initialization: Since the expectation value of the gradients depends on the input, finding the appropriate parameter scaling to stabilize the gradients is difficult.

Original: We show that this stabilizes for wide networks.

Original: • We rigorously derive parameter initialization guidelines for wide maxout networks preventing vanishing and exploding gradients and formulate architecture recommendations.
Revised: • We rigorously derive parameter initialization guidelines for wide maxout networks preventing vanishing and exploding gradients and formulate architecture recommendations.

Original: We experimentally demonstrate that they make it possible to train standard-width deep fully-connected and convolutional maxout networks using simple procedures (such as SGD with momentum and Adam), yielding higher accuracy than other initializations or ReLU networks on image classification tasks.
Revised: We experimentally demonstrate that they make it possible to train standard-width deep fully-connected and convolutional maxout networks using simple procedures (such as SGD with momentum and Adam), yielding higher accuracy than other initializations or ReLU networks on image classification tasks.

Original: • We derive several implications refining previous bounds on the expected number of linear regions (Corollary 6), and new results on length distortion (Corollary 7) and the NTK (Corollary 9).
Revised: • We derive several implications refining previous bounds on the expected number of linear regions (Corollary 6), and new results on length distortion (Corollary 7) and the NTK (Corollary 9).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Architecture We consider feedforward fully-connected maxout neural networks with n 0 inputs, L hidden layers of widths n 1 , . .
Revised: Architecture We consider feedforward fully-connected maxout neural networks with n 0 inputs, L hidden layers of widths n 1 , . .

Original: . , n L  , and a linear output layer, which implement functions of the form N = ψ 
Revised: . , n L − 1 , and a linear output layer, which implement functions of the form N = ψ ◦ ϕ

Original: ◦ ϕ L ◦ · · · ◦ ϕ 1 .
Revised: L − 1 ◦· · ·◦ ϕ 1 .

Original: The l -th hidden layer is a function ϕ
Revised: The l -th hidden layer is a function ϕ

Original: l : R n l − 1 → R n l with components i ∈ [ n l ] := { 1 , . .
Revised: l : R n l − 1 → R n l with components i ∈ [ n l ] := { 1 , . .

Original: , n l } given by the maximum of K ≥ 2 trainable affine functions ϕ l,i : R n l − 1 → R ; x ( l − 1) (cid:55)→ max k ∈ [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , where W ( l ) i,k ∈ R n l − 1 , b i,k ∈ R .
Revised: , n l } given by the maximum of K ≥ 2 trainable affine functions ϕ l,i : R n l − 1 → R ; x ( l − 1) (cid:55)→ max k ∈ [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , where W ( l ) i,k ∈ R n l − 1 , b i,k ∈ R .

Original: Here x ( l − 1) ∈
Revised: Here x ( l − 1) ∈

Original: R n l −denotes the output of the ( l − 1) th layer and x (0) :
Revised: R n l −denotes the output of the ( l − 1) th layer and x (0) :

Original: = x .
Revised: = x .

Original: We will write x ( l ) i,k = W ( l ) i,k x ( l − 1)
Revised: We will write x ( l ) i,k = W ( l ) i,k x ( l − 1)

Original: + b ( l ) i,k to denote the k th pre-activation of the i th neuron in the l th layer.
Revised: + b ( l ) i,k to denote the k th pre-activation of the i th neuron in the l th layer.

Original: Finally ψ : R n L  → R n L+1 is a linear output layer.
Revised: Finally ψ : R n L − 1 → R n L is a linear output layer.

Original: We will write Θ = { W , b } for the parameters.
Revised: We will write Θ = { W , b } for the parameters.

Original: Unless stated otherwise, we assume that for each layer, the weights and biases are initialized as i.i.d.
Revised: Unless stated otherwise, we assume that for each layer, the weights and biases are initialized as i.i.d.

Original: samples from a Gaussian distribution with mean 0 and variance c/n l − 1 , where c is a positive constant.
Revised: samples from a Gaussian distribution with mean 0 and variance c/n l − 1 , where c is a positive constant.

Original: For the linear output layer, the variance is set as 1 /n L − 1 .
Revised: For the linear output layer, the variance is set as 1 /n L − 1 .

Original: We shall study appropriate choices of c .
Revised: We shall study appropriate choices of c .

Original: We will use ∥ · ∥ to denote the ℓ 2 vector norm.
Revised: We will use ∥ · ∥ to denote the ℓ 2 vector norm.

Original: We recall that a real valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .
Revised: We recall that a real valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .

Original: In Appendix A we review basic notions about  random variables that we will use to state our results.
Revised: In Appendix A we review basic notions about maxout networks and random variables that we will use in our results.

Original: Input-output Jacobian and activation length
Revised: Input-output Jacobian and activation length

Original: We are concerned with the gradients of the outputs with respect to the inputs, ∇N i ( x ) = ∇ x N i , and with respect to the parameters, ∇N i ( Θ ) =
Revised: We are concerned with the gradients of the outputs with respect to the inputs, ∇N i ( x ) = ∇ x N i , and with respect to the parameters, ∇N i ( Θ ) =

Original: ∇ Θ N i .
Revised: ∇ Θ N i .

Original: In our notation, the argument indicates the variables with respect to which we are taking the derivatives.
Revised: In our notation, the argument indicates the variables with respect to which we are taking the derivatives.

Original: To study these gradients, we consider the input-output Jacobian
Revised: To study these gradients, we consider the input-output Jacobian

Original: J N ( x ) =
Revised: J N ( x ) =

Original: [ ∇N 1 ( x ) , . .
Revised: [ ∇N 1 ( x ) , . .

Original: , ∇N n L ( x )] T .
Revised: , ∇N n L ( x )] T .

Original: To see the connection to the gradient with respect to the network parameters, consider any loss function L : R n L
Revised: To see the connection to the gradient with respect to the network parameters, consider any loss function L : R n L

Original: → R .
Revised: → R .

Original: A short calculation shows that, for a fixed input x ∈ R n 0 , the derivative of the loss with respect to one of the weights W ( l ) i,k ′ ,j of a maxout unit is (cid:10) ∇L ( N ( x )) , J N ( x ( l ) i ) (cid:11) x
Revised: A short calculation shows that, for a fixed input x ∈ R n 0 , the derivative of the loss with respect to one of the weights W ( l ) i,k ′ ,j of a maxout unit is (cid:10) ∇L ( N ( x )) , J N ( x ( l ) i ) (cid:11) x

Original: ( l
Revised: ( l

Original: [Equation]
Revised: [Equation]

Original: 1) j if k ′ =
Revised: 1) j if [Equation]

Original: argmaxk
Revised: argmax

Original: { x ( l ) i,k } and zero otherwise, i.e.
Revised: [Equation] zero otherwise [Equation] .e

Original: [Equation]
Revised: [Equation]

Original: where C ( x , W ) :
Revised: where C ( x , W ) :

Original: =
Revised: =

Original: ∥ J N ( x ( l ) i ) ∥ − 1 ⟨∇L ( N ( x )) , J N ( x ( l ) i ) ⟩ and u = e i ∈ R n l .
Revised: ∥ J N ( x ( l ) i ) ∥ − 1 ⟨∇L ( N ( x )) , J N ( x ( l ) i ) ⟩ and u = e i ∈ R n l .

Original: A similar decomposition of the derivative was used by Hanin (2018);
Revised: A similar decomposition of the derivative was used by Hanin (2018);

Original: Hanin & Rolnick (2018) for ReLU networks.
Revised: Hanin & Rolnick (2018) for ReLU networks.

Original: By (1) the fluctuation of the gradient norm around its mean is captured by the joint distribution of the squared norm of the directional derivative ∥ J N ( x ) u ∥ 2 and the normalized activation length
Revised: By (1) the fluctuation of the gradient norm around its mean is captured by the joint distribution of the squared norm of the directional derivative ∥ J N ( x ) u ∥ 2 and the normalized activation length

Original: A ( l ) =
Revised: A ( l ) =

Original: ∥ x ( l ) ∥ 2 /n l .
Revised: ∥ x ( l ) ∥ 2 /n l .

Original: We also observe that ∥ J N ( x ) u ∥ 2 is related to the singular values of the inputoutput Jacobian, which is of interest since a spectrum concentrated around one at initialization can speed up training (Saxe et al., 2014; Pennington et al., 2017; 2018):
Revised: We also observe that ∥ J N ( x ) u ∥ 2 is related to the singular values of the inputoutput Jacobian, which is of interest since a spectrum concentrated around one at initialization can speed up training (Saxe et al., 2014; Pennington et al., 2017; 2018):

Original: First, the sum of singular values is tr ( J N ( x ) T J N ( x ))
Revised: First, the sum of singular values is tr ( J N ( x ) T J N ( x ))

Original: = (cid:80) n L i =1 ⟨ J N ( x ) T J N ( x ) u
Revised: = (cid:80) n L i =1 ⟨ J N ( x ) T J N ( x ) u

Original: i , u i ⟩ = (cid:80) n L i =1 ∥ J N ( x ) u i ∥ 2 , where the vectors u i form an orthonormal basis.
Revised: i , u i ⟩ = (cid:80) n L i =1 ∥ J N ( x ) u i ∥ 2 , where the vectors u i form an orthonormal basis.

Original: Second, using the Stieltjes transform, one can show that singular values of the Jacobian depend on the even moments of the entries of J N (Hanin, 2018, Section 3.1).
Revised: Second, using the Stieltjes transform, one can show that singular values of the Jacobian depend on the even moments of the entries of J N (Hanin, 2018, Section 3.1).

Original:  R ESULTS 3. B OUNDS ON THE INPUT - OUTPUT J ACOBIAN
Revised:  R ESULTS 3. B OUNDS ON THE INPUT - OUTPUT J ACOBIAN

Original: Theorem 1 (Bounds on ∥ J N ( x ) u ∥ 2 ) .
Revised: Theorem 1 (Bounds on ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let u ∈ R n 0 be a fixed unit vector.
Revised: Let u ∈ R n 0 be a fixed unit vector.

Original: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 , the following stochastic order bounds hold:
Revised: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 , the following stochastic order bounds hold:

Original: [Equation]
Revised: [Equation]

Original: where ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) are respectively the smallest and largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, independent of each other and of the vectors u and x .
Revised: where ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) are respectively the smallest and largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, independent of each other and of the vectors u and x .

Original: The proof is in Appendix B. It is based on appropriate modifications to the ReLU discussion of Hanin & Nica (2020b);
Revised: The proof is in Appendix B. It is based on appropriate modifications to the ReLU discussion of Hanin & Nica (2020b);

Original: Hanin et al.
Revised: Hanin et al.

Original: (2021) and proceeds by writing the Jacobian norm as the product of the layer norms and bounding them with min k ∈
Revised: (2021) and proceeds by writing the Jacobian norm as the product of the layer norms and bounding them with min k ∈

Original: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } and max k ∈ [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } .
Revised: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } and max k ∈ [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } .

Original: Since the product of a Gaussian vector with a unit vector is always Gaussian, the lower and upper bounds are distributed as the smallest and largest order statistics in a sample of size K of chi-squared
Revised: Since the product of a Gaussian vector with a unit vector is always Gaussian, the lower and upper bounds are distributed as the smallest and largest order statistics in a sample of size K of chi-squared

Original: [Figure]
Revised: [Figure]

Original: random variables with 1 degree of freedom.
Revised: random variables with 1 degree of freedom.

Original: In contrast to ReLU networks, we found that for maxout networks, it is impossible to write equality in distribution involving only independent random variables because of the dependency of the distribution of ∥ J N ( x ) u ∥ 2 on the network input x and the direction vector u (see Figure 1).
Revised: In contrast to ReLU networks, we found that for maxout networks, it is impossible to write equality in distribution involving only independent random variables because of the dependency of the distribution of ∥ J N ( x ) u ∥ 2 on the network input x and the direction vector u (see Figure 1).

Original: We discuss this in more detail in Section 3.2.
Revised: We discuss this in more detail in Section 3.2.

Original: Corollary 2 (Bounds on the moments of ∥ J N ( x ) u ∥ 2 ) .
Revised: Corollary 2 (Bounds on the moments of ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 be any input into the network, Then
Revised: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 be any input into the network, Then

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ii)
Revised: ii)

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to the distribution of the network weights.
Revised: where the expectation is taken with respect to the distribution of the network weights.

Original: The constants S and L depend on K and denote the means of the smallest and the largest order statistic in a sample of
Revised: The constants S and L depend on K and denote the means of the smallest and the largest order statistic in a sample of

Original: K chi-squared random variables.
Revised: K chi-squared random variables.

Original: For K = 2 , . . . , 10 , S ∈ [0 . 02 , 0 . 4] and L ∈ [1 . 6 , 4] .
Revised: For K = 2 , . . . , 10 , S ∈ [0 . 02 , 0 . 4] and L ∈ [1 . 6 , 4] .

Original: See Tablein Appendix C for the exact values.
Revised: See Tablein Appendix C for the exact values.

Original: Notice that for t ≥ 2 , the t th moments of the input-output Jacobian depend on the architecture of the network, but the mean does not (Corollary 2), similarly to their behavior in ReLU networks Hanin (2018).
Revised: Notice that for t ≥ 2 , the t th moments of the input-output Jacobian depend on the architecture of the network, but the mean does not (Corollary 2), similarly to their behavior in ReLU networks Hanin (2018).

Original: We also observe that the upper bound on the t th moments can grow exponentially with the network depth depending on the maxout rank.
Revised: We also observe that the upper bound on the t th moments can grow exponentially with the network depth depending on the maxout rank.

Original: However, the upper bound on the moments can be tightened provided corresponding bounds for the largest order statistics of the chi-squared distribution.
Revised: However, the upper bound on the moments can be tightened provided corresponding bounds for the largest order statistics of the chi-squared distribution.

Original: 3. D ISTRIBUTION OF THE INPUT - OUTPUT J ACOBIAN
Revised: 3. D ISTRIBUTION OF THE INPUT - OUTPUT J ACOBIAN

Original: Here we present the equality in distribution for the input-output Jacobian.
Revised: Here we present the equality in distribution for the input-output Jacobian.

Original: It contains dependent variables for the individual layers and thus cannot be readily used to obtain bounds on the moments, but it is particularly helpful for studying the behavior of wide maxout networks.
Revised: It contains dependent variables for the individual layers and thus cannot be readily used to obtain bounds on the moments, but it is particularly helpful for studying the behavior of wide maxout networks.

Original: Theorem 3 (Equality in distribution for ∥ J N ( x ) u ∥ 2 ) .
Revised: Theorem 3 (Equality in distribution for ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 , x ̸ = 0 be any input into the network.
Revised: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 , x ̸ = 0 be any input into the network.

Original: Then, almost surely, with respect to the parameter initialization, ∥ J N ( x ) u ∥ 2 equals in distribution
Revised: Then, almost surely, with respect to the parameter initialization, ∥ J N ( x ) u ∥ 2 equals in distribution

Original: [Equation]
Revised: [Equation]

Original: L −
Revised: L −

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: l n l
Revised: l n l

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where v i and Ξ l,i ( N (0 , 1) , K ) are independent, v i ∼ N (0 , 1) , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables.
Revised: where v i and Ξ l,i ( N (0 , 1) , K ) are independent, v i ∼ N (0 , 1) , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables.

Original: Here γ x ( l ) , u ( l ) denotes the
Revised: Here γ x ( l ) , u ( l ) denotes the

Original: [Figure]
Revised: [Figure]

Original: Convolutional network
Revised: Convolutional network

Original: Width = Width =
Revised: Width = Width =

Original: Width =
Revised: Width =

Original: Width = angle between x ( l ) :
Revised: Width = angle between x ( l ) :

Original: = ( x ( l ) 1 , . . .
Revised: = ( x ( l ) 1 , . . .

Original: , x ( l ) n l , 1) and u ( l ) := ( u ( l ) 1 , . . .
Revised: , x ( l ) n l , 1) and u ( l ) := ( u ( l ) 1 , . . .

Original: , u ( l ) n l , 0) in R n l +1 , where u ( l ) =
Revised: , u ( l ) n l , 0) in R n l +1 , where u ( l ) =

Original: W ( l ) u ( l − 1) /∥ W ( l ) u ( −
Revised: W ( l ) u ( l − 1) /

Original: ∥  when W (  ) u (  −  = 0 and 0 otherwise, and u (
Revised: ∥ W ( l ) u ( l − 1) ∥ when W ( l ) u ( l − 1) ̸ = 0 and 0 otherwise, and u (0) =

Original: = u .
Revised: u .

Original: The matrices  rows 
Revised: The matrices W ( l ) consist of rows W ( l ) i =

Original:  ∈ R n  k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original:  K ] { W ,k
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } .

Original: This statement is proved in Appendix D.
Revised: This statement is proved in Appendix D.

Original: The main strategy is to construct an orthonormal basis
Revised: The main strategy is to construct an orthonormal basis

Original: B = ( b 1 , . . .
Revised: B = ( b 1 , . . .

Original: , b n l ) , where
Revised: , b n l ) , where

Original: b 1 :=
Revised: b 1 :=

Original: x ( l ) / ∥ x ( l ) ∥ , which allows us to express the layer gradient depending on the angle between x ( l ) and u ( l ) .
Revised: x ( l ) / ∥ x ( l ) ∥ , which allows us to express the layer gradient depending on the angle between x ( l ) and u ( l ) .

Original: Remark 4 (Wide networks) .
Revised: Remark 4 (Wide networks) .

Original: By Theorem 3, in a maxout network the distribution of ∥ J N ( x ) u ∥depends on the cos γ x ( l − 1) , u ( l − 1) , which changes as the network gets wider or deeper.
Revised: By Theorem 3, in a maxout network the distribution of ∥ J N ( x ) u ∥depends on the cos γ x ( l − 1) , u ( l − 1) , which changes as the network gets wider or deeper.

Original: Since independent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal, we expect that the cosine will be close to zero for the earlier layers of wide networks, and individual units will behave similarly to squared standard Gaussians.
Revised: Since independent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal, we expect that the cosine will be close to zero for the earlier layers of wide networks, and individual units will behave similarly to squared standard Gaussians.

Original: In wide and deep networks, if the network parameters are sampled from N (0 , c/n l − 1 ) , c = 1 / M , and K ≥ 3 , we expect that | cos γ x ( l ) , u ( l ) | ≈for the later layers of deep networks and individual units will behave more as the squared largest order statistics.
Revised: In wide and deep networks, if the network parameters are sampled from N (0 , c/n l − 1 ) , c = 1 / M , and K ≥ 3 , we expect that | cos γ x ( l ) , u ( l ) | ≈for the later layers of deep networks and individual units will behave more as the squared largest order statistics.

Original: Here M is the second moment of the largest order statistic in a sample of size K of standard Gaussian random variables.
Revised: Here M is the second moment of the largest order statistic in a sample of size K of standard Gaussian random variables.

Original: Based on this, for deep and wide networks, we can expect that
Revised: Based on this, for deep and wide networks, we can expect that

Original: [Equation]
Revised: [Equation]

Original: This intuition is discussed in more detail in Appendix D. According to (2), we expect that the expected gradient magnitude will be stable with depth when an appropriate initialization is used.
Revised: This intuition is discussed in more detail in Appendix D. According to (2), we expect that the expected gradient magnitude will be stable with depth when an appropriate initialization is used.

Original: See Figurefor a numerical evaluation of the effects of the width and depth on the gradients.
Revised: See Figurefor a numerical evaluation of the effects of the width and depth on the gradients.

Original: 3. A CTIVATION LENGTH
Revised: 3. A CTIVATION LENGTH

Original: To have a full picture of the derivatives in (1), we consider the activation length.
Revised: To have a full picture of the derivatives in (1), we consider the activation length.

Original: The full version and proof of Corollary 5 are in
Revised: The full version and proof of Corollary 5 are in

Original: Appendix E.
Revised: Appendix E.

Original: The proof is based on Theorem 3, replacing u with x / ∥ x ∥ .
Revised: The proof is based on Theorem 3, replacing u with x / ∥ x ∥ .

Original: Corollary 5 (Moments of the normalized activation length) .
Revised: Corollary 5 (Moments of the normalized activation length) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Let x ∈ R n 0 be any input into the network.
Revised: Let x ∈ R n 0 be any input into the network.

Original: Then, for the moments of the normalized activation length
Revised: Then, for the moments of the normalized activation length

Original: A ( l ′ ) of the l ′
Revised: A ( l ′ ) of the l ′

Original: th layer we have
Revised: th layer we have

Original: Mean:
Revised: Mean:

Original: [Equation]
Revised: [Equation]

Original: Moments of order t ≥ 2 :
Revised: Moments of order t ≥ 2 :

Original: [Equation]
Revised: [Equation]

Revised: 

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The expectation is taken with respect to the distribution of the network weights and biases, and M is a constant depending on K that can be computed approximately, see Table 4 for the values for
Revised: The expectation is taken with respect to the distribution of the network weights and biases, and M is a constant depending on K that can be computed approximately, see Table 4 for the values for

Original: K = 2 , . .
Revised: K = 2 , . .

Original: See Appendix E for the variance bounds and details on functions G 1 , G 2 .
Revised: See Appendix E for the variance bounds and details on functions G 1 , G 2 .

Original: We could obtain an exact expression for the mean activation length for a finitely wide maxout network since its distribution only depends on the norm of the input, while this is not the case for the inputoutput Jacobian (Sections 3.1 and 3.2).
Revised: We could obtain an exact expression for the mean activation length for a finitely wide maxout network since its distribution only depends on the norm of the input, while this is not the case for the inputoutput Jacobian (Sections 3.1 and 3.2).

Original: We observe that the variance and the t th moments, t ≥ 2 , have an exponential dependence on the network architecture, including the maxout rank, whereas the mean does not, similarly to the input-output Jacobian (Corollary 2).
Revised: We observe that the variance and the t th moments, t ≥ 2 , have an exponential dependence on the network architecture, including the maxout rank, whereas the mean does not, similarly to the input-output Jacobian (Corollary 2).

Original: Such behavior also occurs for
Revised: Such behavior also occurs for

Original: ReLU networks (Hanin & Rolnick, 2018).
Revised: ReLU networks (Hanin & Rolnick, 2018).

Original: See Figure 7 in Appendix E for an evaluation of the result.
Revised: See Figure 8 in Appendix E for an evaluation of the result.

Original:  I MPLICATIONS TO INITIALIZATION AND NETWORK ARCHITECTURE
Revised:  I MPLICATIONS TO INITIALIZATION AND NETWORK ARCHITECTURE

Original: We now aim to find initialization approaches and architectures that can avoid exploding and vanishing gradients.
Revised: We now aim to find initialization approaches and architectures that can avoid exploding and vanishing gradients.

Original: We take the annealed exploding and vanishing gradients definition from Hanin (2018) as a starting point for such investigation for maxout networks.
Revised: We take the annealed exploding and vanishing gradients definition from Hanin (2018) as a starting point for such investigation for maxout networks.

Original: Formally, we require
Revised: Formally, we require

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: , ∀ t ≥ 3 , where the expectation is with respect to the weights and biases.
Revised: , ∀ t ≥ 3 , where the expectation is with respect to the weights and biases.

Original: Based on (1) these conditions can be attained by ensuring that similar conditions hold for ∥ J N ( x ) u ∥ 2 and A ( l ) .
Revised: Based on (1) these conditions can be attained by ensuring that similar conditions hold for ∥ J N ( x ) u ∥ 2 and A ( l ) .

Original: Initialization recommendations Based on Corollary 2, the mean of ∥ J N ( x ) u ∥ 2 can be stabilized for some c ∈ [1 / L , 1 / S ] .
Revised: Initialization recommendations Based on Corollary 2, the mean of ∥ J N ( x ) u ∥ 2 can be stabilized for some c ∈ [1 / L , 1 / S ] .

Original: However, Theorem 3 shows that ∥ J N ( x ) u ∥ 2 depends on the input into the network.
Revised: However, Theorem 3 shows that ∥ J N ( x ) u ∥ 2 depends on the input into the network.

Original: Hence, we expect that there is no value of c stabilizing input-output Jacobian for every input simultaneously.
Revised: Hence, we expect that there is no value of c stabilizing input-output Jacobian for every input simultaneously.

Original: Nevertheless, based on Remark 4, for wide and deep maxout networks, E [ ∥ J N ( x ) u ∥ 2 ] ≈ n L /n 

Revised: Nevertheless, based on Remark 4, for wide and deep maxout networks, E [ ∥ J N ( x ) u ∥ 2 ] ≈ n L /n

Original: if c = 1 / M , and the mean becomes stable.
Revised: if c = 1 / M , and the mean becomes stable.

Original: While Remark 4 does not include maxout rank K = 2 , the same recommendation can be obtained for it using the approach from He et al.
Revised: While Remark 4 does not include maxout rank K = 2 , the same recommendation can be obtained for it using the approach from He et al.

Original: (2016), see Sun et al.
Revised: , see Sun et al.

Original: Moreover, according to Corollary 5, the mean of the normalized activation length remains stable for different network depths if c = 1 / M .
Revised: Moreover, according to Corollary 5, the mean of the normalized activation length remains stable for different network depths if c = 1 / M .

Original: Hence, we recommend c = 1 / M as an appropriate value for initialization.
Revised: Hence, we recommend c = 1 / M as an appropriate value for initialization.

Original: See Table 1 for the numerical value of c for K = 2 , . .
Revised: See Table 1 for the numerical value of c for K = 2 , . .

Original: We call this type of initialization, when the parameters are sampled from N (0 , c/ fan-in ) , c = 1 / M , “maxout initialization”.
Revised: We call this type of initialization, when the parameters are sampled from N (0 , c/ fan-in ) , c = 1 / M , “maxout initialization”.

Original: We note that this matches the previous recommendation from Tseran & Montúfar (2021), which we now derived rigorously.
Revised: We note that this matches the previous recommendation from Tseran & Montúfar (2021), which we now derived rigorously.

Original: Architecture recommendations In Corollaries 2 and 5 the upper bound on the moments t ≥ 2 of ∥ J N ( x ) u ∥ 2 and A ( l ) =
Revised: Architecture recommendations In Corollaries 2 and 5 the upper bound on the moments t ≥ 2 of ∥ J N ( x ) u ∥ 2 and A ( l ) =

Original: ∥ x ( l ) ∥ 2 /n l can grow exponentially with the depth depending on the values of ( cK ) L and (cid:80) L − 1 l =1 1 / ( n l K ) .
Revised: ∥ x ( l ) ∥ 2 /n l can grow exponentially with the depth depending on the values of ( cK ) L and (cid:80) L − 1 l =1 1 / ( n l K ) .

Original: Hence, we recommend choosing the widths such that (cid:80) L − 1 l =1 1 / ( n l K ) ≤ 1 , which holds, e.g., if n l ≥ L/K, ∀ l = 1 , . . .
Revised: Hence, we recommend choosing the widths such that (cid:80) L − 1 l =1 1 / ( n l K ) ≤ 1 , which holds, e.g., if n l ≥ L/K, ∀ l = 1 , . . .

Original: , L − 1 , and choosing a moderate value of the maxout rank K .
Revised: , L − 1 , and choosing a moderate value of the maxout rank K .

Original: However, the upper bound can still tend to infinity for the high-order moments.
Revised: However, the upper bound can still tend to infinity for the high-order moments.

Original: From
Revised: From

Original: Remark 4, it follows that for K ≥ 3 to have a stable initialization independent of the network input, a maxout network has to be deep and wide.
Revised: Remark 4, it follows that for K ≥ 3 to have a stable initialization independent of the network input, a maxout network has to be deep and wide.

Original: Experimentally, we observe that for 100 -neuron wide networks with K = 3 , the absolute value of the cosine that determines the initialization stability converges to 1 at around 60 layers, and for K = 4 , 5 , at around 30 layers.
Revised: Experimentally, we observe that for 100 -neuron wide networks with K = 3 , the absolute value of the cosine that determines the initialization stability converges to 1 at around 60 layers, and for K = 4 , 5 , at around 30 layers.

Original: See Figure 4 in Appendix D.
Revised: See Figure 5 in Appendix D.

Original: To sum up, we recommend working with deep and wide maxout networks with widths satisfying (cid:80) L −l =1 1 / ( n l K ) ≤ 1 , and choosing the maxout-rank not too small nor too large, e.g., K = 5 .
Revised: To sum up, we recommend working with deep and wide maxout networks with widths satisfying (cid:80) L −l =1 1 / ( n l K ) ≤ 1 , and choosing the maxout-rank not too small nor too large, e.g., K = 5 .

Original:  I MPLICATIONS TO E XPRESSIVITY AND NTK
Revised:  I MPLICATIONS TO E XPRESSIVITY AND NTK

Original: With Theorems 1 and 3 in place, we can now obtain maxout versions of the several types of results that previously have been derived only for ReLU networks.
Revised: With Theorems 1 and 3 in place, we can now obtain maxout versions of the several types of results that previously have been derived only for ReLU networks.

Original: 5. E XPECTED NUMBER OF LINEAR REGIONS OF MAXOUT NETWORKS
Revised: 5. E XPECTED NUMBER OF LINEAR REGIONS OF MAXOUT NETWORKS

Original: For a piece-wise linear function f : R n 0 → R , a linear region is defined as a maximal connected subset of R n 0 on which f has a constant gradient.
Revised: For a piece-wise linear function f : R n 0 → R , a linear region is defined as a maximal connected subset of R n 0 on which f has a constant gradient.

Original: Tseran & Montúfar (2021) and Hanin & Rolnick (2019b) established upper bounds on the expected number of linear regions of maxout and ReLU networks, respectively.
Revised: Tseran & Montúfar (2021) and Hanin & Rolnick (2019b) established upper bounds on the expected number of linear regions of maxout and ReLU networks, respectively.

Original: One of the key factors controlling these bounds is C grad , which is any upper bound on (sup x ∈ R n 0 E [ ∥∇ ζ z,k ( x ) ∥ t ]) 1 /t , for any t ∈ N and z = 1 , . . .
Revised: One of the key factors controlling these bounds is C grad , which is any upper bound on (sup x ∈ R n 0 E [ ∥∇ ζ z,k ( x ) ∥ t ]) 1 /t , for any t ∈ N and z = 1 , . . .

Original: , N .
Revised: , N .

Original: Here ζ z,k is the k th pre-activation feature of the z th unit in the network, N is the total number of units, and the gradient is with respect to the network input.
Revised: Here ζ z,k is the k th pre-activation feature of the z th unit in the network, N is the total number of units, and the gradient is with respect to the network input.

Original: Using Corollary 2, we obtain a value for C grad for maxout networks, which remained an open problem in the work of Tseran & Montúfar (2021).
Revised: Using Corollary 2, we obtain a value for C grad for maxout networks, which remained an open problem in the work of Tseran & Montúfar (2021).

Original: The proof of Corollaryand the resulting refined bound on the expected number of linear regions are in Appendix F.
Revised: The proof of Corollaryand the resulting refined bound on the expected number of linear regions are in Appendix F.

Original: Corollary 6 (Value for C grad ) .
Revised: Corollary 6 (Value for C grad ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Consider the pre-activation feature ζ z,k of a unit z = 1 , . .
Revised: Consider the pre-activation feature ζ z,k of a unit z = 1 , . .

Original: . , N .
Revised: . , N .

Original: Then, for any t ∈ N ,
Revised: Then, for any t ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: The value of C grad given in Corollary 6 grows as O (( cK )
Revised: The value of C grad given in Corollary 6 grows as O (( cK )

Original: [Equation]
Revised: [Equation]

Original: l =1 1 / ( n l K ) } ) .
Revised: l =1 1 / ( n l K ) } ) .

Original: The first factor grows exponentially with the network depth if cK > 1 .
Revised: The first factor grows exponentially with the network depth if cK > 1 .

Original: This is the case when the network is initialized as in Section 4.
Revised: This is the case when the network is initialized as in Section 4.

Original: However, since K is usually a small constant and c ≤ 1 , cK ≥ 1 is a small constant.
Revised: However, since K is usually a small constant and c ≤ 1 , cK ≥ 1 is a small constant.

Original: The second factor grows exponentially with the depth if (cid:80) L − 1 l =1 1 / ( n l K ) ≥ 1 .
Revised: The second factor grows exponentially with the depth if (cid:80) L − 1 l =1 1 / ( n l K ) ≥ 1 .

Original: Hence, the exponential growth can be avoided if n l ≥ ( L − 1) /K, ∀ l = 1 , . . .
Revised: Hence, the exponential growth can be avoided if n l ≥ ( L − 1) /K, ∀ l = 1 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: 5. E XPECTED CURVE LENGTH DISTORTION
Revised: 5. E XPECTED CURVE LENGTH DISTORTION

Original: Let M be a smooth 1 -dimensional curve in R n 0 of length len ( M ) and N ( M ) ⊆
Revised: Let M be a smooth 1 -dimensional curve in R n 0 of length len ( M ) and N ( M ) ⊆

Original: R n L the image of M under the map x (cid:55)→ N ( x ) .
Revised: R n L the image of M under the map x (cid:55)→ N ( x ) .

Original: We are interested in the length distortion of M , defined as len ( N ( M )) / len ( M ) .
Revised: We are interested in the length distortion of M , defined as len ( N ( M )) / len ( M ) .

Original: Using the results from Section 3.1, observing that the input-output Jacobian of maxout networks is well defined almost everywhere, and following Hanin et al.
Revised: Using the results from Section 3.1, observing that the input-output Jacobian of maxout networks is well defined almost everywhere, and following Hanin et al.

Original: (2021), we obtain the following corollary.
Revised: (2021), we obtain the following corollary.

Original: The proof is in Appendix G.
Revised: The proof is in Appendix G.

Original: Corollary 7 (Expected curve length distortion) .
Revised: Corollary 7 (Expected curve length distortion) .

Original: Consider a maxout network with the settings of
Revised: Consider a maxout network with the settings of

Original: Section 2.
Revised: Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let M be a smooth 1 -dimensional curve of unit length in R n 0 .
Revised: Let M be a smooth 1 -dimensional curve of unit length in R n 0 .

Original: Then, the following upper bounds on the moments of len ( N ( M )) hold:
Revised: Then, the following upper bounds on the moments of len ( N ( M )) hold:

Original: [Equation]
Revised: [Equation]

Original: n L (cid:33)(cid:41) , where L is a constant depending on K , see Table 4 in Appendix C for values for K = 2 , . .
Revised: n L (cid:33)(cid:41) , where L is a constant depending on K , see Table 4 in Appendix C for values for K = 2 , . .

Original: Remark 8 (Expected curve length distortion in wide maxout networks) .
Revised: Remark 8 (Expected curve length distortion in wide maxout networks) .

Original: If the network is initialized according to Section 4, using Remark 4 and repeating the steps of the proof of Corollary 7, we get E [ len ( N ( M ))] ≲ ( n L /n 0 ) 1 / 2 and Var [ len ( N ( M ))]
Revised: If the network is initialized according to Section 4, using Remark 4 and repeating the steps of the proof of Corollary 7, we get E [ len ( N ( M ))] ≲ ( n L /n 0 ) 1 / 2 and Var [ len ( N ( M ))]

Original: ≈ n L /n 0 .
Revised: ≈ n L /n 0 .

Original: Hence, similarly to ReLU networks, wide maxout networks, if initialized to keep the gradients stable, have low expected curve length distortion at initialization.
Revised: Hence, similarly to ReLU networks, wide maxout networks, if initialized to keep the gradients stable, have low expected curve length distortion at initialization.

Original: However, we cannot conclude whether the curve length shrinks.
Revised: However, we cannot conclude whether the curve length shrinks.

Original: For narrow networks, the upper bound does not exclude the possibility that the expected distortion grows exponentially with the network depth, depending on the initialization.
Revised: For narrow networks, the upper bound does not exclude the possibility that the expected distortion grows exponentially with the network depth, depending on the initialization.

Original: 5. O N - DIAGONAL NTK
Revised: 5. O N - DIAGONAL NTK

Original: We denote the on-diagonal NTK with K N ( x , x ) =
Revised: We denote the on-diagonal NTK with K N ( x , x ) =

Original: (cid:80) i ( ∂ N ( x ) /∂θ i ) 2 .
Revised: (cid:80) i ( ∂ N ( x ) /∂θ i ) 2 .

Original: In Appendix H we show:
Revised: In Appendix H we show:

Original: Corollary 9 (On-diagonal NTK) .
Revised: Corollary 9 (On-diagonal NTK) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that n L = 1 and that the biases are initialized to zero and are not trained.
Revised: Assume that n L = 1 and that the biases are initialized to zero and are not trained.

Original: Assume that S ≤ c ≤ L ,
Revised: Assume that S ≤ c ≤ L ,

Original: [Table]
Revised: [Table]

Original: where the constants S
Revised: where the constants S

Original: [Table]
Revised: [Table]

Original: L are as specified in Table 4.
Revised: L are as specified in Table 4.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: P,
Revised: P,

Original: E [ K N ( x , x ) 2 ] ≤ 2 PPW ( cK )
Revised: E [ [Equation] [Equation] PP

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 4 nexp   L − 1 (cid:88) j =  n j K +  , where P = (cid:80) L − 1 l =0 n l , P W = (cid:80) Ll [-=
-] n l n l − 1 , and M is as specified in Table 4.
Revised:   , where P = (cid:80) L − 1 l =0 n l , P W = (cid:80) Ll [-=
-]=0 n l n l − 1 , and M is as specified in Table 4.

Original: By Corollary 9, E [ K N ( x , x ) 2 ] / ( E [ K N ( x , x )]) 2 is in O (( P W /P )
Revised: By Corollary 9, E [ K N ( x , x ) 2 ] / ( E [ K N ( x , x )]) 2 is in O (( P W /P )

Original: C L exp { (cid:80) Ll
Revised: C L exp { (cid:80) Ll

Original: =1 1 / ( n l K ) } ) , where
Revised: =1 1 / ( n l K ) } ) , where

Original: C depends on L , M and K .
Revised: C depends on L , M and K .

Original: Hence, if widths n 1 , . . . , n L − 1 and depth L tend to infinity, this upper bound does not converge to a constant, suggesting that the NTK might not converge to a constant in probability.
Revised: Hence, if widths n 1 , . . . , n L − 1 and depth L tend to infinity, this upper bound does not converge to a constant, suggesting that the NTK might not converge to a constant in probability.

Original: This is in line with previous results for ReLU networks by Hanin & Nica (2020a).
Revised: This is in line with previous results for ReLU networks by Hanin & Nica (2020a).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We check how the initialization proposed in Section 4 affects the network training.
Revised: We check how the initialization proposed in Section 4 affects the network training.

Original: This initialization was first proposed heuristically by Tseran & Montúfar (2021), where it was tested for 10 -layer fullyconnected networks with an MNIST experiment.
Revised: This initialization was first proposed heuristically by Tseran & Montúfar (2021), where it was tested for 10 -layer fullyconnected networks with an MNIST experiment.

Original: We consider both fully-connected and convolutional neural networks and run experiments for MNIST (LeCun & Cortes, 2010), Iris (Fisher, 1936), Fashion
Revised: We consider both fully-connected and convolutional neural networks and run experiments for MNIST (LeCun & Cortes, 2010), Iris (Fisher, 1936), Fashion

Original: MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009).
Revised: MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009).

Original: Fully connected networks have 21 layers and CNNs have a VGG-19-like architecture (Simonyan & Zisserman, 2015) with 20 or 16 layers depending on the input size, all with maxout rank 5 .
Revised: Fully connected networks have 21 layers and CNNs have a VGG-19-like architecture (Simonyan & Zisserman, 2015) with 20 or 16 layers depending on the input size, all with maxout rank 5 .

Original: Weights are sampled from N (0 , c/ fan-in ) in fully-connected networks and N (0 , c/ ( k 2 · fan-in )) in CNNs of kernel size k .
Revised: Weights are sampled from N (0 , c/ fan-in ) in fully-connected networks and N (0 , c/ ( k 2 · fan-in )) in CNNs of kernel size k .

Original: The biases are initialized to zero.
Revised: The biases are initialized to zero.

Original: We report the mean and std of 4 runs.
Revised: We report the mean and std of 4 runs.

Original: We use plain deep networks without any kind of modifications or pre-training.
Revised: We use plain deep networks without any kind of modifications or pre-training.

Original: We do not use normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015), since this would obscure the effects of the initialization.
Revised: We do not use normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015), since this would obscure the effects of the initialization.

Original: Because of this, our results are not necessarily state-of-the-art.
Revised: Because of this, our results are not necessarily state-of-the-art.

Original: More details on the experiments are given in Appendix I, and the implementation is made available at https://anonymous.4open.science/r/maxout_expected_gradient-68BD .
Revised: More details on the experiments are given in Appendix I, and the implementation is made available at https://anonymous.4open.science/r/maxout_expected_gradient-68BD .

Original: Max-pooling initialization To account for the maximum in max-pooling layers, a maxout layer appearing after a max-pooling layer is initialized as if its maxout rank was K × m 2 , where m 2 is the max-pooling window size.
Revised: Max-pooling initialization To account for the maximum in max-pooling layers, a maxout layer appearing after a max-pooling layer is initialized as if its maxout rank was K × m 2 , where m 2 is the max-pooling window size.

Original: For example, we used K = 5 and m 2 = 4 , resulting in c = 0 .for
Revised: For example, we used K = 5 and m 2 = 4 , resulting in c = 0 .for

Original: such maxout layers.
Revised: such maxout layers.

Original: All other layers are initialized according to Section 4.
Revised: All other layers are initialized according to Section 4.

Original: We observe that max-pooling initialization often leads to slightly higher accuracy.
Revised: We observe that max-pooling initialization often leads to slightly higher accuracy.

Original: Results for SGD with momentum Table 2 reports test accuracy for networks trained using SGD with Nesterov momentum.
Revised: Results for SGD with momentum Table 2 reports test accuracy for networks trained using SGD with Nesterov momentum.

Original: We compare ReLU and maxout networks with different initializations: maxout, max-pooling, small value c = 0 .
Revised: We compare ReLU and maxout networks with different initializations: maxout, max-pooling, small value c = 0 .

Original: 1 , and He c
Revised: 1 , and He c

Original: = 2 .
Revised: = 2 .

Original: We observe that maxout and max-
Revised: We observe that maxout and maxpooling initializations allow training deep maxout networks and obtaining better accuracy than ReLU

Original: [Table]
Revised: [Table]

Original: pooling initializations allow training deep maxout networks and obtaining better accuracy than ReLU networks, whereas  training does not progress for maxout networks with other initializations.
Revised: networks, whereas performance is significantly worse or training does not progress for maxout networks with other initializations.

Original: Results for Adam Table 3 reports test accuracy for networks trained using Adam (Kingma &
Revised: Results for Adam Table 3 reports test accuracy for networks trained using Adam (Kingma &

Original: Ba, 2015).
Revised: Ba, 2015).

Original: We compare ReLU and maxout networks with the following initializations: maxout, max-pooling, and He c
Revised: We compare ReLU and maxout networks with the following initializations: maxout, max-pooling, and He c

Original: = 2 .
Revised: = 2 .

Original: We observe that, compared to He initialization, maxout and max-pooling initializations lead to faster convergence and better test accuracy.
Revised: We observe that, compared to He initialization, maxout and max-pooling initializations lead to faster convergence and better test accuracy.

Original: Compared to ReLU networks, maxout networks have better or comparable accuracy if maxout or max-pooling initialization is used.
Revised: Compared to ReLU networks, maxout networks have better or comparable accuracy if maxout or max-pooling initialization is used.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We study the gradients of maxout networks with respect to the parameters and the inputs by analyzing a directional derivative of the input-output map.
Revised: We study the gradients of maxout networks with respect to the parameters and the inputs by analyzing a directional derivative of the input-output map.

Original: We observe that the distribution of the input-output
Revised: We observe that the distribution of the input-output

Original: Jacobian of maxout networks depends on the network input (in contrast to ReLU networks), which can complicate the stable initialization of maxout networks.
Revised: Jacobian of maxout networks depends on the network input (in contrast to ReLU networks), which can complicate the stable initialization of maxout networks.

Original: Based on bounds on the moments, we derive an initialization that provably avoids vanishing and exploding gradients in wide networks.
Revised: Based on bounds on the moments, we derive an initialization that provably avoids vanishing and exploding gradients in wide networks.

Original: Experimentally, we show that, compared to other initializations, the suggested approach leads to better performance for fully connected and convolutional deep networks of standard width trained with SGD or Adam and better or similar performance compared to ReLU networks.
Revised: Experimentally, we show that, compared to other initializations, the suggested approach leads to better performance for fully connected and convolutional deep networks of standard width trained with SGD or Adam and better or similar performance compared to ReLU networks.

Original: Additionally, we refine previous upper bounds on the expected number of linear regions.
Revised: Additionally, we refine previous upper bounds on the expected number of linear regions.

Original: We also derive results for the expected curve length distortion, observing that it does not grow exponentially with the depth in wide networks.
Revised: We also derive results for the expected curve length distortion, observing that it does not grow exponentially with the depth in wide networks.

Original: Furthermore, we obtain bounds on the maxout NTK, suggesting that it might not converge to a constant when both the width and depth are large.
Revised: Furthermore, we obtain bounds on the maxout NTK, suggesting that it might not converge to a constant when both the width and depth are large.

Original: These contributions enhance the applicability of maxout networks and add to the theoretical exploration of activation functions beyond ReLU.
Revised: These contributions enhance the applicability of maxout networks and add to the theoretical exploration of activation functions beyond ReLU.

Original: Limitations Even though our proposed initialization is in a sense optimal, our results are applicable only when the weights are sampled from N (0 , c/ fan-in ) for some c .
Revised: Limitations Even though our proposed initialization is in a sense optimal, our results are applicable only when the weights are sampled from N (0 , c/ fan-in ) for some c .

Original: Further, we derived theoretical results only for fully-connected networks.
Revised: Further, we derived theoretical results only for fully-connected networks.

Original: Our experiments indicate that they also hold for CNNs:
Revised: Our experiments indicate that they also hold for CNNs:

Original: Figure 2 demonstrates that gradients behave according to the theory for fully connected and convolutional networks, and Tables 2 and 3 show improvement in CNNs performance under the initialization suggested in Section 4.
Revised: Figure 2 demonstrates that gradients behave according to the theory for fully connected and convolutional networks, and Tables 2 and 3 show improvement in CNNs performance under the initialization suggested in Section 4.

Original: However, we have yet to conduct the theoretical analysis of CNNs.
Revised: However, we have yet to conduct the theoretical analysis of CNNs.

Original: Future work In future work, we would like to obtain more general results in settings involving multi-argument functions, such as aggregation functions in graph neural networks, and investigate the effects that initialization strategies stabilizing the initial gradients have at later stages of training.
Revised: Future work In future work, we would like to obtain more general results in settings involving multi-argument functions, such as aggregation functions in graph neural networks, and investigate the effects that initialization strategies stabilizing the initial gradients have at later stages of training.

Original: Reproducibility Statement To ensure reproducibility, we make the code public on GitHub at https: //anonymous.4open.science/r/maxout_expected_gradient-68BD , and Section 6 and Appendix I provide a detailed description of the experimental settings.
Revised: Reproducibility Statement To ensure reproducibility, we make the code public on GitHub at https: //anonymous.4open.science/r/maxout_expected_gradient-68BD , and Section 6 and Appendix I provide a detailed description of the experimental settings.

Original: Proofs are in Appendices B–H.
Revised: Proofs are in Appendices B–H.

Original: A PPENDIX
Revised: A PPENDIX

Original: The appendix is organized as follows.
Revised: The appendix is organized as follows.

Original: • Appendix A Basic notions of probability • Appendix B Bounds for the input-output Jacobian norm ∥ J N ( x ) u ∥• Appendix C Moments of the input-output Jacobian norm ∥ J N ( x ) u ∥• Appendix D Equality in distribution for the input-output Jacobian norm and wide network results • Appendix E Activation length • Appendix F Expected number of linear regions • Appendix G Expected curve length distortion • 
Revised: • Appendix A Basics • Appendix B Bounds for the input-output Jacobian norm ∥ J N ( x ) u ∥• Appendix C Moments of the input-output Jacobian norm ∥ J N ( x ) u ∥• Appendix D Equality in distribution for the input-output Jacobian norm and wide network results • Appendix E Activation length • Appendix F Expected number of linear regions • Appendix G Expected curve length distortion • Appendix H NTK • Appendix I Experiments A B ASICS A. B ASICS ON MAXOUT NETWORKS As mentioned in the introduction, a rank- K maxout unit computes the maximum of K real-valued affine functions. Concretely, a rank- K maxout unit with n inputs implements a function

Revised: [Equation]

Original: Appendix H NTK • Appendix I Experiments
Revised: where W k ∈ R n and b k ∈ R , k ∈ [ K ] := { 1 , . . . , K } , are trainable weights and biases. The K arguments of the maximum are called the pre-activation features of the maxout unit. A rank- K maxout unit can be regarded as a composition of an affine map with K outputs and a maximum gate.

Revised: A layer corresponds to parallel computation of several such units.

Revised: For instance a layer with n inputs and m maxout units computes functions of the form

Revised: [Equation]

Revised: where now W (1) i,k and b (1) i,k are the weights and biases of the k th pre-activation feature of the i th maxout unit in the first layer.

Revised: The situation is illustrated in Figure 3 for the case of a network with two inputs, one layer with two maxout units of rank three, and one output layer with a single output unit.

Revised: [Equation]

Original: A B ASIC NOTIONS OF PROBABILITY
Revised: A. B ASIC NOTIONS OF PROBABILITY

Original: We ought to remind several probability theory notions that we use to state our results.
Revised: We ought to remind several probability theory notions that we use to state our results.

Original: Firstly, recall that if v 1 , . .
Revised: Firstly, recall that if v 1 , . .

Original: . , v k are independent, univariate standard normal random variables, then the sum of their squares, (cid:80) ki
Revised: . , v k are independent, univariate standard normal random variables, then the sum of their squares, (cid:80) ki

Original: =1 v 2 i , is distributed according to the chi-squared distribution with k degrees of freedom.
Revised: =1 v 2 i , is distributed according to the chi-squared distribution with k degrees of freedom.

Original: We will denote such a random variable with χ 2 k .
Revised: We will denote such a random variable with χ 2 k .

Original: Secondly, the largest order statistic is a random variable defined as the maximum of a random sample, and the smallest order statistic is the minimum of a sample.
Revised: Secondly, the largest order statistic is a random variable defined as the maximum of a random sample, and the smallest order statistic is the minimum of a sample.

Original: And finally, a real valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if
Revised: And finally, a real valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if

Original: Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .
Revised: Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .

Original: We will also denote with d = equality in distribution (meaning the cdfs are the same).
Revised: We will also denote with d = equality in distribution (meaning the cdfs are the same).

Original: With this, we start with the results for the squared norm of the input-output Jacobian ∥ J N ( x ) u ∥ 2 .
Revised: With this, we start with the results for the squared norm of the input-output Jacobian ∥ J N ( x ) u ∥ 2 .

Revised: A. D ETAILS ON THE EQUATION (1)

Revised: In (1) we are investigating magnitude of ∂ L ( x ) ∂W ( l ) i,k ′ ,j .

Revised: The reason we focus on the Jacobian norm rather than on C is as follows.

Revised: We have

Original: B B OUNDS FOR THE INPUT - OUTPUT J ACOBIAN NORM ∥ J N ( x  )  u ∥B. P RELIMINARIES
Revised: [Equation] Note that C ( x , W ) = ⟨∇ N L ( N ( x )) , v ⟩ with v = J N (cid:0) x ( l ) (cid:1) u / ∥ J N (cid:0) x (

Revised: [Equation]

Revised: ) (cid:1) u ∥ , ∥ v ∥ = 1 .

Revised: Hence

Revised: C ( x , W ) ≤ ∥∇ N L ( N ( x ))

Revised: ∥∥ v ∥ =

Revised: ∥∇ N L ( N ( x ))

Revised: ∥ .

Revised: The latter term does not directly depend on the specific parametrization nor the specific architecture of the network but only on the loss function and the prediction.

Revised: In view of the description of ∂ L ( x ) ∂W ( l ) i,k ′ ,j , the variance depends on the square of x ( l − 1) j .

Revised: Similarly, the variance of the gradient ∇ W ( l ) L ( x ) =

Revised: ( ∂ L ( x ) ∂W ( l ) i,k ′ ,j ) j depends on x ( l − 1) =

Revised: ( x ( l − 1) j ) j and thus depends on ∥ x ( l − 1)

Revised: ∥ 2 .

Revised: This is how activation length appears in (1).

Revised: B B OUNDS FOR THE INPUT - OUTPUT J ACOBIAN NORM ∥ J N ( x ) u ∥B. P RELIMINARIES

Original: We start by presenting several well-known results we will need for further discussion.
Revised: We start by presenting several well-known results we will need for further discussion.

Original: Product of a Gaussian matrix and a unit vector
Revised: Product of a Gaussian matrix and a unit vector

Original: Lemma 10.
Revised: Lemma 10.

Original: Suppose W is an n × n ′ matrix with i.i.d.
Revised: Suppose W is an n × n ′ matrix with i.i.d.

Original: Gaussian entries and u is a random unit vector in R n ′ that is independent of W but otherwise has any distribution.
Revised: Gaussian entries and u is a random unit vector in R n ′ that is independent of W but otherwise has any distribution.

Original: Then1.
Revised: Then

Original: W u is independent of u and is equal in distribution to W v where v is any fixed unit vector in R n ′ .
Revised: W u is independent of u and is equal in distribution to W v where v is any fixed unit vector in R n ′ .

Original: If the entries of W are sampled i.i
Revised: If the entries of W are sampled i.i.d.

Original: d. from N ( µ, σ 2 ) , then for all i = 1 , . . . , n , W i u ∼ N ( µ, σ 2 ) and independent of u . 3.
Revised: from N ( µ, σ 2 ) , then for all i = 1 , . . . , n , W i u ∼ N ( µ, σ 2 ) and independent of u . 3.

Original: If the entries of W are sampled i.i
Revised: If the entries of W are sampled i.i.d.

Original: d. from N (0 , σ 2 ) , then the squared ℓ 2 norm ∥ W u ∥ 2 d = σ 2 χ 2 n , where χ 2 n is a chi-squared random variable with n degrees of freedom that is independent of u .
Revised: from N (0 , σ 2 ) , then the squared ℓ 2 norm ∥ W u ∥ 2 d = σ 2 χ 2 n , where χ 2 n is a chi-squared random variable with n degrees of freedom that is independent of u .

Original: Proof.
Revised: Proof.

Original: Statement 1 was proved in, e.g., Hanin et al
Revised: Statement 1 was proved in, e.g., Hanin et al.

Original:  Lemma C.3)
Revised: (2021, Lemma C.3)

Original: by considering directly the joint distribution of W u and u .
Revised: by considering directly the joint distribution of W u and u .

Original: Statement 2 follows from Statement 1 if we pick v = e 1 .
Revised: Statement 2 follows from Statement 1 if we pick v = e 1 .

Original: To prove Statement 3, recall that by definition of the ℓ 2 norm, ∥ W u ∥ 2 = (cid:80) ni
Revised: To prove Statement 3, recall that by definition of the ℓ 2 norm, ∥ W u ∥ 2 = (cid:80) ni

Original: =

Revised: =

Original: ( W i u ) 2 .
Revised: ( W i u ) 2 .

Original: By
Revised: By

Original: Statement 2, for all i = 1 , . . . , n , W i u are Gaussian random variables independent of u with mean zero and variance σ 2 .
Revised: Statement 2, for all i = 1 , . . . , n , W i u are Gaussian random variables independent of u with mean zero and variance σ 2 .

Original: Since any Gaussian random variable sampled from N ( µ, σ 2 ) can be written as µ + σv , where v ∼ N (0 , 1) , we can write (cid:80) ni =

Revised: Since any Gaussian random variable sampled from N ( µ, σ 2 ) can be written as µ + σv , where v ∼ N (0 , 1) , we can write (cid:80) ni =

Original: ( W i u ) 2 = σ 2 (cid:80) ni
Revised: ( W i u ) 2 = σ 2 (cid:80) ni

Original: =1 v 2 i .
Revised: =1 v 2 i .

Original: By definition of the chisquared distribution, (cid:80) ni
Revised: By definition of the chisquared distribution, (cid:80) ni

Original: =1 v 2 i is a chi-squared random variable with n degrees of freedom denoted with χ 2 n , which leads to the desired result.
Revised: =1 v 2 i is a chi-squared random variable with n degrees of freedom denoted with χ 2 n , which leads to the desired result.

Original: Stochastic order We recall the definition of a stochastic order.
Revised: Stochastic order We recall the definition of a stochastic order.

Original: A real-valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .
Revised: A real-valued random variable X is said to be smaller than Y in the stochastic order, denoted by X ≤ st Y , if Pr( X > x ) ≤ Pr( Y > x ) for all x ∈ R .

Original: Remark 11 (Stochastic ordering for functions) .
Revised: Remark 11 (Stochastic ordering for functions) .

Original: Consider two functions f : X → R and g :
Revised: Consider two functions f : X → R and g :

Original: X → R that satisfy f ( x ) ≤ g ( x ) for all x ∈ X .
Revised: X → R that satisfy f ( x ) ≤ g ( x ) for all x ∈ X .

Original: Then, for a random variable X , f ( X ) ≤ st g ( X ) .
Revised: Then, for a random variable X , f ( X ) ≤ st g ( X ) .

Original: To see this, observe that for any y ∈ R , Pr( f ( X ) > y )
Revised: To see this, observe that for any y ∈ R , Pr( f ( X ) > y )

Original: =
Revised: =

Original: Pr( X ∈ { x : f ( x ) > y } ) and
Revised: Pr( X ∈ { x : f ( x ) > y } ) and

Original: Pr( g ( X ) > y ) =
Revised: Pr( g ( X ) > y ) =

Original: Pr( X ∈ { x : g ( x ) > y } ) .
Revised: Pr( X ∈ { x : g ( x ) > y } ) .

Original: Since f ( x ) ≤ g ( x ) for all x ∈ X , { x : f ( x ) > y } ⊆ { x : g ( x ) > y } .
Revised: Since f ( x ) ≤ g ( x ) for all x ∈ X , { x : f ( x ) > y } ⊆ { x : g ( x ) > y } .

Original: Hence, Pr( f ( X ) > y ) ≤ Pr( g ( X ) > y ) , and f ( X ) ≤ st g ( X ) .
Revised: Hence, Pr( f ( X ) > y ) ≤ Pr( g ( X ) > y ) , and f ( X ) ≤ st g ( X ) .

Original: Remark 12 (Stochastic order and equality in distribution) .
Revised: Remark 12 (Stochastic order and equality in distribution) .

Original: Consider real-valued random variables X ,
Revised: Consider real-valued random variables X ,

Original: Y and ˆ Y .
Revised: Y and ˆ Y .

Original: If X ≤ st Y and Y d = ˆ Y , then X ≤ st ˆ Y .
Revised: If X ≤ st Y and Y d = ˆ Y , then X ≤ st ˆ Y .

Original: Since Y and ˆ Y have the same cdfs by definition of equality in distribution, for any y ∈ R , Pr( X > y )
Revised: Since Y and ˆ Y have the same cdfs by definition of equality in distribution, for any y ∈ R , Pr( X > y )

Original: ≤
Revised: ≤

Original: Pr( Y > y )
Revised: Pr( Y > y )

Original: =
Revised: =

Original: Pr( ˆ Y > y ) .
Revised: Pr( ˆ Y > y ) .

Original: B. 
Revised: B. E XPRESSION FOR ∥ J N ( x ) u ∥

Original: E XPRESSION FOR ∥ J N ( x ) u ∥

Original: Before proceeding to the proof of the main statement, given in Theorem 1, we present Proposition 13.
Revised: Before proceeding to the proof of the main statement, given in Theorem 1, we present Proposition 13.

Original: Firstly, in Proposition 13 below, we prove an equality that holds almost surely for an input-output
Revised: Firstly, in Proposition 13 below, we prove an equality that holds almost surely for an input-output

Original: Jacobian under our assumptions.
Revised: Jacobian under our assumptions.

Original: In this particular statement the reasoning closely follows Hanin et al.
Revised: In this particular statement the reasoning closely follows Hanin et al.

Original: (2021, Proposition C.2).
Revised: (2021, Proposition C.2).

Original: The modifications are due to the fact that a maxout network Jacobian is a product of matrices consisting of the rows of weights that are selected based on which pre-activation feature attains maximum, while in a ReLU network, the rows in these matrices are either the neuron weights or zeros.
Revised: The modifications are due to the fact that a maxout network Jacobian is a product of matrices consisting of the rows of weights that are selected based on which pre-activation feature attains maximum, while in a ReLU network, the rows in these matrices are either the neuron weights or zeros.

Original: Proposition 13 (Equality for ∥ J N ( x ) u ∥ 2 ) .
Revised: Proposition 13 (Equality for ∥ J N ( x ) u ∥ 2 ) .

Original: Let N be a fully-connected feed-forward neural network with maxout units of rank K and a linear last layer.
Revised: Let N be a fully-connected feed-forward neural network with maxout units of rank K and a linear last layer.

Original: Let the network have L layers of widths n 1 , . . .
Revised: Let the network have L layers of widths n 1 , . . .

Original: , n L and n 0 inputs.
Revised: , n L and n 0 inputs.

Original: Assume that the weights are continuous random variables (that have a density) and that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the weights are continuous random variables (that have a density) and that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let u ∈ R n 0 be a fixed unit vector.
Revised: Let u ∈ R n 0 be a fixed unit vector.

Original: Then, for any input into the network, x ∈ R n 0 , almost surely with respect to the parameter initialization the Jacobian with respect to the input satisfies
Revised: Then, for any input into the network, x ∈ R n 0 , almost surely with respect to the parameter initialization the Jacobian with respect to the input satisfies

Original: [Equation]
Revised: [Equation]

Original: where vectors u ( l ) , l = 1 , . .
Revised: where vectors u ( l ) , l = 1 , . .

Original: , L − 1 are defined recursively as u ( l ) =
Revised: , L − 1 are defined recursively as u ( l ) =

Original: W ( l ) u ( l − 1) /
Revised: W ( l ) u ( l − 1) /

Original: ∥ W ( l ) u ( l − 1) ∥ when W ( l ) u ( l − 1) ̸ = 0 and 0 otherwise, and u (0) =
Revised: ∥ W ( l ) u ( l − 1) ∥ when W ( l ) u ( l − 1) ̸ = 0 and 0 otherwise, and u (0) =

Original: u .
Revised: u .

Original: The matrices W ( l ) consist of rows W ( l ) i =
Revised: The matrices W ( l ) consist of rows W ( l ) i =

Original: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , x ( l ) is the output of the l th layer, and x (0) =
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , x ( l ) is the output of the l th layer, and x (0) =

Original: x .
Revised: x .

Original: Proof.
Revised: Proof.

Original: The Jacobian J N ( x ) of a network N ( x ): R n 0 → R n L can be written as a product of matrices
Revised: The Jacobian J N ( x ) of a network N ( x ): R n 0 → R n L can be written as a product of matrices

Original: W ( l ) , l = 1 , . . .
Revised: W ( l ) , l = 1 , . . .

Original: , L , depending on the activation region of the input x .
Revised: , L , depending on the activation region of the input x .

Original: The matrix W ( l ) consists of rows W ( l ) i =
Revised: The matrix W ( l ) consists of rows W ( l ) i =

Original: W ( l ) i,k ′ ∈ R n l − 1 , where k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } for i = 1 , . .
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } for i = 1 , . .

Original: . , n l , and x ( l − 1) is the l th layer’s input.
Revised: . , n l , and x ( l − 1) is the l th layer’s input.

Original: For the last layer, which is linear, we have W ( L ) =
Revised: For the last layer, which is linear, we have W ( L ) =

Original: W ( L ) .
Revised: W ( L ) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Further we denote u with u (0) and assume ∥ W (1) u (0) ∥
Revised: Further we denote u with u (0) and assume ∥ W (1) u (0) ∥

Original: ̸ = 0 .
Revised: ̸ = 0 .

Original: To see that this holds almost surely, note that for a fixed unit vector u (0) , the probability of W (1) being such that ∥ W (1) u (0) ∥
Revised: To see that this holds almost surely, note that for a fixed unit vector u (0) , the probability of W (1) being such that ∥ W (1) u (0) ∥

Original: = 0 is 0 .
Revised: = 0 is 0 .

Original: This is indeed the case since to satisfy ∥ W (1) u (0) ∥
Revised: This is indeed the case since to satisfy ∥ W (1) u (0) ∥

Original: = 0 , the weights must be a solution to a system of nlinear equations and this system is regular when u ̸ = 0 , so the solution set has positive co-dimension and hence zero measure.
Revised: = 0 , the weights must be a solution to a system of nlinear equations and this system is regular when u ̸ = 0 , so the solution set has positive co-dimension and hence zero measure.

Original: Multiplying and dividing (4) by ∥ W (1) u (0) ∥ 2 ,
Revised: Multiplying and dividing (4) by ∥ W (1) u (0) ∥ 2 ,

Original: [Equation]
Revised: [Equation]

Original: where u (1) =
Revised: where u (1) =

Original: W (1) u (0) /
Revised: W (1) u (0) /

Original: ∥ W (1) u (0) ∥ .
Revised: ∥ W (1) u (0) ∥ .

Original: Repeating this procedure layer-by-layer, we get∥
Revised: Repeating this procedure layer-by-layer, we get

Original: [Equation]
Revised: [Equation]

Original: By definition of the ℓ 2 norm, for any layer l , ∥ W ( l ) u ( l − 1) ∥ 2 = (cid:80) n l i =1 ⟨ W ( l ) i [Equation] u [Equation]
Revised: By definition of the ℓ 2 norm, for any layer l , ∥ W ( l ) u ( l − 1) ∥ 2 = (cid:80) n l i =1 ⟨ W ( l ) i , u ( l − 1) ⟩ 2 .

Original: Substituting this into (5) we get the desired statement.
Revised: Substituting this into (5) we get the desired statement.

Original: B. S TOCHASTIC ORDERING FOR ∥ J N ( x ) u ∥
Revised: B. S TOCHASTIC ORDERING FOR ∥ J N ( x ) u ∥

Original: Now we prove the result for the stochastic ordering of the input-output Jacobian in a finite-width maxout network.
Revised: Now we prove the result for the stochastic ordering of the input-output Jacobian in a finite-width maxout network.

Original: Theorem 1 (Bounds on ∥ J N ( x ) u ∥ 2 ) .
Revised: Theorem 1 (Bounds on ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let u ∈ R n 0 be a fixed unit vector.
Revised: Let u ∈ R n 0 be a fixed unit vector.

Original: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 , the following stochastic order bounds hold:
Revised: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 , the following stochastic order bounds hold:

Original: [Equation]
Revised: [Equation]

Original: where ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) are respectively the smallest and largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, independent of each other and of the vectors u and x .
Revised: where ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) are respectively the smallest and largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, independent of each other and of the vectors u and x .

Original: Proof.
Revised: Proof.

Original: From Proposition 13, we have the following equality
Revised: From Proposition 13, we have the following equality

Original: [Equation]
Revised: [Equation]

Original: where vectors u ( l ) , l = 0 , . . .
Revised: where vectors u ( l ) , l = 0 , . . .

Original: , L − 1 are defined recursively as u ( l ) =
Revised: , L − 1 are defined recursively as u ( l ) =

Original: W ( l ) u ( l −1) /
Revised: W ( l ) u ( l −

Revised: [Equation]

Original:  ∥ W ( l ) u ( l − 1) ∥ and u (0) =
Revised: / ∥ W ( l ) u ( l − 1) ∥ and u (0) =

Original: u .
Revised: u .

Original: Matrices W ( l ) consist of rows W ( l ) i =
Revised: Matrices W ( l ) consist of rows W ( l ) i =

Original: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , and x ( l − 1) is the l th layer’s input, x (0) =
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , and x ( l − 1) is the l th layer’s input, x (0) =

Original: x .
Revised: x .

Original: We assumed that weights in the last layer are sampled from a Gaussian distribution with mean zero and variance 1 /n
Revised: We assumed that weights in the last layer are sampled from a Gaussian distribution with mean zero and variance 1 /n

Original: L − 1 .
Revised: L − 1 .

Original: Then, by Lemma 10 item 3, ∥ W ( L ) u ( L − 1) ∥ 2 d = (1 /n L − 1 ) χ 2 n L and is independent of u ( L − 1) .
Revised: Then, by Lemma 10 item 3, ∥ W ( L ) u ( L − 1) ∥ 2 d = (1 /n L − 1 ) χ 2 n L and is independent of u ( L − 1) .

Original: In equation (6), using this observation and then multiplying and dividing the summands by c/n l − 1 and rearranging we obtain
Revised: In equation (6), using this observation and then multiplying and dividing the summands by c/n l − 1 and rearranging we obtain

Original: [Equation]
Revised: [Equation]

Original: Now we focus on (cid:112) n l − 1 /c ⟨ W ( l ) i , u ( l − 1) ⟩ .
Revised: Now we focus on (cid:112) n l − 1 /c ⟨ W ( l ) i , u ( l − 1) ⟩ .

Original: Since we have assumed that the weights are sampled from a Gaussian distribution with zero mean and variance c/n l − 1 , any weight W ( l ) i,k,j , j = 1 , . .
Revised: Since we have assumed that the weights are sampled from a Gaussian distribution with zero mean and variance c/n l − 1 , any weight W ( l ) i,k,j , j = 1 , . .

Original: . , n l − 1 , can be written as (cid:112) c/n l − 1 v ( l ) i,k,j , where v ( l ) i,k,j is a standard Gaussian random variable.
Revised: . , n l − 1 , can be written as (cid:112) c/n l − 1 v ( l ) i,k,j , where v ( l ) i,k,j is a standard Gaussian random variable.

Original: We also write
Revised: We also write

Original: W ( l ) i,k = (cid:112) c/n l − 1 V ( l ) i,k , where V ( l ) i,k is an n l − 1 -dimensional standard Gaussian random vector.
Revised: W ( l ) i,k = (cid:112) c/n l − 1 V ( l ) i,k , where V ( l ) i,k is an n l − 1 -dimensional standard Gaussian random vector.

Original: Observe that for any k ′ ∈
Revised: Observe that for any k ′ ∈

Original: [ K ] , ⟨ W ( l ) i,k ′ , u ( l − 1) ⟩ 2 ≤ max k ∈
Revised: [ K ] , ⟨ W ( l ) i,k ′ , u ( l − 1) ⟩ 2 ≤ max k ∈

Original: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } and ⟨ W ( l ) i,k ′ , u ( l − 1) ⟩ 2 ≥ min k ∈
Revised: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } and ⟨ W ( l ) i,k ′ , u ( l − 1) ⟩ 2 ≥ min k ∈

Original: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } .
Revised: [ K ] {⟨ W ( l ) i,k , u ( l − 1) ⟩ 2 } .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Notice that vectors u ( l − 1) are unit vectors by their definition.
Revised: Notice that vectors u ( l − 1) are unit vectors by their definition.

Original: By Lemma 10, the inner product of a standard Gaussian vector and a unit vector is a standard Gaussian random variable independent of the given unit vector.
Revised: By Lemma 10, the inner product of a standard Gaussian vector and a unit vector is a standard Gaussian random variable independent of the given unit vector.

Original: By definition, a squared standard Gaussian random variable is distributed as χ 21 , a chi-squared random variable with 1 degree of freedom.
Revised: By definition, a squared standard Gaussian random variable is distributed as χ 21 , a chi-squared random variable with 1 degree of freedom.

Original: Hence, max k ∈ [ K ] {⟨ V ( l ) i,k , u ( l − 1) ⟩ 2 } is distributed as the largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom.
Revised: Hence, max k ∈ [ K ] {⟨ V ( l ) i,k , u ( l − 1) ⟩ 2 } is distributed as the largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom.

Original: We will denote such a random variable with Ξ l,i ( χ 21 , K ) .
Revised: We will denote such a random variable with Ξ l,i ( χ 21 , K ) .

Original: Likewise, min k ∈ [ K ] {⟨ V ( l ) i,k , u ( l − 1) ⟩ 2 } is distributed as the smallest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, denoted with ξ l,i ( χ 2 1 , K ) .
Revised: Likewise, min k ∈ [ K ] {⟨ V ( l ) i,k , u ( l − 1) ⟩ 2 } is distributed as the smallest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, denoted with ξ l,i ( χ 21 , K ) .

Original: Combining results for each layer, we obtain the following bounds
Revised: Combining results for each layer, we obtain the following bounds

Original: [Equation]

Original: Then

Original: [Equation]
Revised: [Equation]

Original:  by Remarks 11 and [Equation] the following stochastic ordering holds
Revised: Then, by Remarks 11 and 12, the following stochastic ordering holds

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: C M OMENTS OF THE INPUT - OUTPUT J ACOBIAN NORM ∥ J N ( x ) u ∥
Revised: C M OMENTS OF THE INPUT - OUTPUT J ACOBIAN NORM ∥ J N ( x ) u ∥

Original: In the proof on the bounds of the moments we use an approach similar to Hanin et al. (2021) for upper bounding the moments of the chi-squared distribution.
Revised: In the proof on the bounds of the moments we use an approach similar to Hanin et al. (2021) for upper bounding the moments of the chi-squared distribution.

Original: Corollary 2 (Bounds on the moments of ∥ J N ( x ) u ∥ 2 ) .
Revised: Corollary 2 (Bounds on the moments of ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 be any input into the network, Then
Revised: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 be any input into the network, Then

Original: (i) n L n 0 ( c S )
Revised: (i) n L n 0 ( c S )

Original: L − 
Revised: L − 1 ≤ E [ ∥

Original: [Equation]
Revised: [Equation]

Original:  ∥
Revised: N ( x ) u ∥

Original: [Equation]
Revised: [Equation]

Original: ( x )
Revised: n L n

Original: [Equation]
Revised: [Equation]

Revised: ( c L ) L

Original: [Equation]
Revised: [Equation]

Original: (ii) Var (cid:2)
Revised: (ii) Var (cid:2)

Original: ∥ J N ( x ) u∥
Revised: ∥ J N ( x ) u

Revised: ∥ 2 (cid:3) ≤ (cid:18) n

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (iii) E (cid:2)

Original: [Equation]
Revised: ∥ J N ( x ) u

Original:  (cid:18)[Equation] L [Equation]
Revised: ∥ 2 t (cid:3) ≤ (cid:18)

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to the distribution of the network weights.
Revised: where the expectation is taken with respect to the distribution of the network weights.

Original: The constants S and L depend on K and denote the means of the smallest and the largest order statistic in a sample of
Revised: The constants S and L depend on K and denote the means of the smallest and the largest order statistic in a sample of

Original: K chi-squared random variables.
Revised: K chi-squared random variables.

Original: For K = 2 , . . . , 10 , S ∈ [0 . 02 , 0 . 4] and L ∈ [1 . 6 , 4] .
Revised: For K = 2 , . . . , 10 , S ∈ [0 . 02 , 0 . 4] and L ∈ [1 . 6 , 4] .

Original: See Tablein Appendix C for the exact values.
Revised: See Tablein Appendix C for the exact values.

Original: Proof.
Revised: Proof.

Original: We first prove results for the mean, then for the moments of order t > 1 , and finish with the proof for the variance.
Revised: We first prove results for the mean, then for the moments of order t > 1 , and finish with the proof for the variance.

Original: Mean Using mutual independence of the variables in the bounds in Theorem 1, and that if two non-negative univariate random variables X and Y are such that X ≤ st Y then E
Revised: Mean Using mutual independence of the variables in the bounds in Theorem 1, and that if two non-negative univariate random variables X and Y are such that X ≤ st Y then E

Original: [ X n ] ≤ E [ Y n ] for all n ≥ 1 (Müller & Stoyan, 2002, Theorem 1.2.12),
Revised: [ X n ] ≤ E [ Y n ] for all n ≥ 1 (Müller & Stoyan, 2002, Theorem 1.2.12),

Original: [Equation]
Revised: [Equation]

Original: where we used ξ l,i and Ξ l,i as shorthands for ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) .
Revised: where we used ξ l,i and Ξ l,i as shorthands for ξ l,i ( χ 21 , K ) and Ξ l,i ( χ 21 , K ) .

Original: Using the formulas for the largest and the smallest order statistic pdfs from Remark 14, the largest order statistic mean equals
Revised: Using the formulas for the largest and the smallest order statistic pdfs from Remark 14, the largest order statistic mean equals

Original: [Equation]
Revised: [Equation]

Original: and the smallest order statistic mean equals
Revised: and the smallest order statistic mean equals

Original: [Equation]
Revised: [Equation]

Original: Here we denoted the right hand-sides with L and S , which are constants depending on K, and can be computed exactly for K = 2 and K = 3 , and approximately for higher K -s, see Table 4.
Revised: Here we denoted the right hand-sides with L and S , which are constants depending on K, and can be computed exactly for K = 2 and K = 3 , and approximately for higher K -s, see Table 4.

Original: It is known that E (cid:2) χ 2 n L (cid:3) =
Revised: It is known that E (cid:2) χ 2 n L (cid:3) =

Original: n L .
Revised: n L .

Original: Combining, we get
Revised: Combining, we get

Original: [Equation]
Revised: [Equation]

Original: Moments of order t > 1 As above, using mutual independence of the variables in the bounds in
Revised: Moments of order t > 1 As above, using mutual independence of the variables in the bounds in

Original: Theorem 1, and that if two non-negative univariate random variables X and Y are such that X ≤ st Y then E [ X n ] ≤ E [ Y n ] for all n ≥ 1 (Müller & Stoyan, 2002, Theorem 1.2.12),
Revised: Theorem 1, and that if two non-negative univariate random variables X and Y are such that X ≤ st Y then E [ X n ] ≤ E [ Y n ] for all n ≥ 1 (Müller & Stoyan, 2002, Theorem 1.2.12),

Original: [Equation]
Revised: [Equation]

Original: Upper-bounding the maximum of chi-squared variables with a sum,
Revised: Upper-bounding the maximum of chi-squared variables with a sum,

Original: [Equation]
Revised: [Equation]

Original: where we used that a sum of n l K chi-squared variables with one degree of freedom is a chi-squared variable with n l K degrees of freedom.
Revised: where we used that a sum of n l K chi-squared variables with one degree of freedom is a chi-squared variable with n l K degrees of freedom.

Original: Using the formula for noncentral moments of the chi-squared distribution and the inequality 1 + x ≤ e x ,
Revised: Using the formula for noncentral moments of the chi-squared distribution and the inequality 1 + x ≤ e x ,

Original: [Equation]
Revised: [Equation]

Original: where we used the formula for calculating the sum of consecutive numbers (cid:80) t − 1 i
Revised: where we used the formula for calculating the sum of consecutive numbers (cid:80) t − 1 i

Original: =

Revised: =

Original: i = t ( t − 1) / 2 .
Revised: i = t ( t − 1) / 2 .

Original: Similarly, (cid:18)
Revised: Similarly, (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Combining, we upper bound (7) with
Revised: Combining, we upper bound (7) with

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Variance Combining the upper bound on the second moment and the lower bound on the mean, we get the following upper bound on the variance
Revised: Variance Combining the upper bound on the second moment and the lower bound on the mean, we get the following upper bound on the variance

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: Remark 14 (Computing the constants) .
Revised: Remark 14 (Computing the constants) .

Original: Here we provide the derivations necessary to compute the constants equal to the moments of the largest and the smallest order statistics appearing in the results.
Revised: Here we provide the derivations necessary to compute the constants equal to the moments of the largest and the smallest order statistics appearing in the results.

Original: Firstly, the cdf of the largest order statistic of independent univariate random variables y 1 , . . . , y K with cdf F ( x ) and pdf f ( x ) is
Revised: Firstly, the cdf of the largest order statistic of independent univariate random variables y 1 , . . . , y K with cdf F ( x ) and pdf f ( x ) is

Original: [Equation]
Revised: [Equation]

Original: Hence, the pdf is K ( F ( x ))
Revised: Hence, the pdf is K ( F ( x ))

Original: K − 1 f ( x ) .
Revised: K − 1 f ( x ) .

Original: For the smallest order statistic, the cdf is
Revised: For the smallest order statistic, the cdf is

Original: [Equation]
Revised: [Equation]

Original: Thus, the pdf is K (1 − F ( x ))
Revised: Thus, the pdf is K (1 − F ( x ))

Original: K − 1 f ( x ) .
Revised: K − 1 f ( x ) .

Original: Now we obtain pdfs for the distributions that are used in the results.
Revised: Now we obtain pdfs for the distributions that are used in the results.

Original: Chi-squared distribution
Revised: Chi-squared distribution

Original: The cdf of a chi-squared random variable χ 2 k with k = 1 degree of freedom is F ( x ) =
Revised: The cdf of a chi-squared random variable χ 2 k with k = 1 degree of freedom is F ( x ) =

Original: (Γ( k/ 2)) − 1 γ ( k/ 2 , x/ 2)
Revised: (Γ( k/ 2)) − 1 γ ( k/ 2 , x/ 2)

Original: = erf( (cid:112) x/ 2) , and the pdf is f ( x ) =
Revised: = erf( (cid:112) x/ 2) , and the pdf is f ( x ) =

Original: (2 k/ 2 Γ( k/ 2)) − 1 x k/ 2 − 1 e − x/ 2 = (2 π ) − 1 / 2 x − 1 / 2 e − x/ 2 .
Revised: (2 k/ 2 Γ( k/ 2)) − 1 x k/ 2 − 1 e − x/ 2 = (2 π ) − 1 / 2 x − 1 / 2 e − x/ 2 .

Original: Here we used that Γ(1 / 2) = √ π and γ (1 / 2 , x/ 2) = √ π erf ( (cid:112) x/ 2) .
Revised: Here we used that Γ(1 / 2) = √ π and γ (1 / 2 , x/ 2) = √ π erf ( (cid:112) x/ 2) .

Original: Therefore, the pdf of the largest order statistic in a sample of
Revised: Therefore, the pdf of the largest order statistic in a sample of

Original: K chi-squared random variables with 1 degree of freedom Ξ l,i ( χ 21 , K ) is
Revised: K chi-squared random variables with 1 degree of freedom Ξ l,i ( χ 21 , K ) is

Original: [Equation]
Revised: [Equation]

Original: The pdf of the smallest order statistic in a sample of K chi-squared random variables with 1 degree of freedom ξ l,i ( χ 21 , K ) is
Revised: The pdf of the smallest order statistic in a sample of K chi-squared random variables with 1 degree of freedom ξ l,i ( χ 21 , K ) is

Original: [Equation]
Revised: [Equation]

Original: Standard Gaussian distribution Recall that the cdf of a standard Gaussian random variable is
Revised: Standard Gaussian distribution Recall that the cdf of a standard Gaussian random variable is

Original: F ( x ) = 1 / 2(1 + erf( x/ √ 2)) , and the pdf is f ( x ) = 1 / √ 2 π exp {− x 2 / 2 } .
Revised: F ( x ) = 1 / 2(1 + erf( x/ √ 2)) , and the pdf is f ( x ) = 1 / √ 2 π exp {− x 2 / 2 } .

Original: Then, for the pdf of the largest order statistic in a sample of K standard Gaussian random variables Ξ l,i ( N (0 , 1) , K ) we get
Revised: Then, for the pdf of the largest order statistic in a sample of K standard Gaussian random variables Ξ l,i ( N (0 , 1) , K ) we get

Original: [Equation]
Revised: [Equation]

Original: Constants Now we obtain formulas for the constants.
Revised: Constants Now we obtain formulas for the constants.

Original: For the mean of the smallest order statistic in a sample of K chi-squared random variables with 1 degree of freedom ξ l,i ( χ 21 , K ) , we get
Revised: For the mean of the smallest order statistic in a sample of K chi-squared random variables with 1 degree of freedom ξ l,i ( χ 21 , K ) , we get

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The mean of the largest order statistic in a sample of K chi-squared random variables with 1 degree of freedom Ξ l,i ( χ 21 , K ) is
Revised: The mean of the largest order statistic in a sample of K chi-squared random variables with 1 degree of freedom Ξ l,i ( χ 21 , K ) is

Original: [Equation]
Revised: [Equation]

Original: The second moment of the largest order statistic in a sample of K standard Gaussian random variables Ξ l,i ( N (0 , 1) , K ) equals
Revised: The second moment of the largest order statistic in a sample of K standard Gaussian random variables Ξ l,i ( N (0 , 1) , K ) equals

Original: [Equation]
Revised: [Equation]

Original: These constants can be evaluated using numerical computation software.
Revised: These constants can be evaluated using numerical computation software.

Original: The values estimated for
Revised: The values estimated for

Original: K = 2 , . .
Revised: K = 2 , . .

Original: , 10 using Mathematica (Wolfram Research, Inc, 2022) are in Table 4.
Revised: , 10 using Mathematica (Wolfram Research, Inc, 2022) are in Table 4.

Original: [Table]
Revised: [Table]

Original: D E QUALITY IN DISTRIBUTION FOR THE INPUT - OUTPUT J ACOBIAN NORM AND WIDE NETWORK RESULTS
Revised: D E QUALITY IN DISTRIBUTION FOR THE INPUT - OUTPUT J ACOBIAN NORM AND WIDE NETWORK RESULTS

Original: Here we prove results from Section 3.2.
Revised: Here we prove results from Section 3.2.

Original: We will use the following theorem from Anderson (2003).
Revised: We will use the following theorem from Anderson (2003).

Original: We reference it here without proof, but remark that it is based on the well-known result that uncorrelated jointly Gaussian random variables are independent.
Revised: We reference it here without proof, but remark that it is based on the well-known result that uncorrelated jointly Gaussian random variables are independent.

Original: Theorem 15 (Anderson 2003, Theorem 3.3.1) .
Revised: Theorem 15 (Anderson 2003, Theorem 3.3.1) .

Original: Suppose X 1 , . . .
Revised: Suppose X 1 , . . .

Original: , X N are independent, where X α is distributed according to N ( µ α , Σ ) .
Revised: , X N are independent, where X α is distributed according to N ( µ α , Σ ) .

Original: Let C
Revised: Let C

Original: =
Revised: =

Original: ( c αβ ) be an N × N orthogonal matrix.
Revised: ( c αβ ) be an N × N orthogonal matrix.

Original: Then Y α
Revised: Then Y α

Original: = (cid:80)
Revised: = (cid:80)

Original: Nβ =1 c αβ X β is distributed according to N ( ν α , Σ ) ,
Revised: Nβ =1 c αβ X β is distributed according to N ( ν α , Σ ) ,

Original: where ν α = (cid:80) Nβ =1 c αβ µ β , α = 1 , . .
Revised: where ν α = (cid:80) Nβ =1 c αβ µ β , α = 1 , . .

Original: . , N , and
Revised: . , N , and

Original: Y 1 , . . .
Revised: Y 1 , . . .

Original: , Y N are independent.
Revised: , Y N are independent.

Original: Remark 16.
Revised: Remark 16.

Original: We will use Theorem 15 in the following way.
Revised: We will use Theorem 15 in the following way.

Original: Notice that it is possible to consider a vector v with entries sampled i.i.d.
Revised: Notice that it is possible to consider a vector v with entries sampled i.i.d.

Original: from N (0 , σ 2 ) in Theorem 15 and treat entries of v as a set of 1 -dimensional vectors X 1 , . . .
Revised: from N (0 , σ 2 ) in Theorem 15 and treat entries of v as a set of 1 -dimensional vectors X 1 , . . .

Original: , X N .
Revised: , X N .

Original: Then we can obtain that products of the columns of the orthogonal matrix C and the vector v , Y β
Revised: Then we can obtain that products of the columns of the orthogonal matrix C and the vector v , Y β

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Nα =1 c αβ v α , are distributed according to N (0 , σ 2 ) and are mutually independent.
Revised: Nα =1 c αβ v α , are distributed according to N (0 , σ 2 ) and are mutually independent.

Original: Theorem 3 (Equality in distribution for ∥ J N ( x ) u ∥ 2 ) .
Revised: Theorem 3 (Equality in distribution for ∥ J N ( x ) u ∥ 2 ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 , x ̸ = 0 be any input into the network.
Revised: Let u ∈ R n 0 be a fixed unit vector and x ∈ R n 0 , x ̸ = 0 be any input into the network.

Original: Then, almost surely, with respect to the parameter initialization, ∥ J N ( x ) u ∥ 2 equals in distribution
Revised: Then, almost surely, with respect to the parameter initialization, ∥ J N ( x ) u ∥ 2 equals in distribution

Original: [Equation]
Revised: [Equation]

Original: , where v i and Ξ l,i ( N (0 , 1) , K ) are independent, v i ∼ N (0 , 1) , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables.
Revised: , where v i and Ξ l,i ( N (0 , 1) , K ) are independent, v i ∼ N (0 , 1) , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables.

Original: Here γ x ( l ) , u ( l ) denotes the angle between x ( l ) :
Revised: Here γ x ( l ) , u ( l ) denotes the angle between x ( l ) :

Original: = ( x ( l ) 1 , . . .
Revised: = ( x ( l ) 1 , . . .

Original: , x ( l ) n l , 1) and u ( l ) := ( u ( l ) 1 , . . .
Revised: , x ( l ) n l , 1) and u ( l ) := ( u ( l ) 1 , . . .

Original: , u ( l ) n l , 0) in R n l +1 , where u ( l ) =
Revised: , u ( l ) n l , 0) in R n l +1 , where u ( l ) =

Original: W ( l ) u ( l − 1) /
Revised: W ( l ) u ( l − 1) /

Original: ∥ W ( l ) u ( l − 1) ∥ when W ( l ) u ( l − 1) ̸ = 0 and 0 otherwise, and u (0) =
Revised: ∥ W ( l ) u ( l − 1) ∥ when W ( l ) u ( l − 1) ̸ = 0 and 0 otherwise, and u (0) =

Original: u .
Revised: u .

Original: The matrices W ( l ) consist of rows W ( l ) i =
Revised: The matrices W ( l ) consist of rows W ( l ) i =

Original: W ( l ) i,k ′ ∈ R n l − 1 , where k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } .
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } .

Original: Proof.
Revised: Proof.

Original: By Proposition 13, almost surely with respect to the parameter initialization,
Revised: By Proposition 13, almost surely with respect to the parameter initialization,

Original: [Equation]
Revised: [Equation]

Original: where vectors u ( l ) , l = 0 , . . .
Revised: where vectors u ( l ) , l = 0 , . . .

Original: , L − 1 are defined recursively as u ( l ) =
Revised: , L − 1 are defined recursively as u ( l ) =

Original: W ( l ) u ( l − 1) /
Revised: W ( l ) u ( l − 1) /

Original: ∥ W ( l ) u ( l − 1) ∥ and u (0) =
Revised: ∥ W ( l ) u ( l − 1) ∥ and u (0) =

Original: u .
Revised: u .

Original: Matrices W ( l ) consist of rows W ( l ) i =
Revised: Matrices W ( l ) consist of rows W ( l ) i =

Original: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =
Revised: W ( l ) i,k ′ ∈ R n l − 1 , i = 1 , . . . , n l , where k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , and x ( l − 1) is the l th layer’s input, x (0) =
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , and x ( l − 1) is the l th layer’s input, x (0) =

Original: x .
Revised: x .

Original: We assumed that weights in the last layer are sampled from a Gaussian distribution with mean zero and variance 1 /n
Revised: We assumed that weights in the last layer are sampled from a Gaussian distribution with mean zero and variance 1 /n

Original: L − 1 .
Revised: L − 1 .

Original: Then, by Lemma 10, ∥ W ( L ) u ( L − 1) ∥ 2 d = (1 /n L − 1 ) χ 2 n L and is independent of u ( L − 1) .
Revised: Then, by Lemma 10, ∥ W ( L ) u ( L − 1) ∥ 2 d = (1 /n L − 1 ) χ 2 n L and is independent of u ( L − 1) .

Original: We use this observation in the equation (6), multiply and divide the summands in the expression by c/n l − 1 and rearrange to obtain that
Revised: We use this observation in the equation (6), multiply and divide the summands in the expression by c/n l − 1 and rearrange to obtain that

Original: [Equation]
Revised: [Equation]

Original: We define x ( l − 1) :
Revised: We define x ( l − 1) :

Original: = ( x ( l − 1), . . .
Revised: = ( x ( l − 1), . . .

Original: , x ( l − 1) n l − 1 , 1) ∈ R n l − 1 +1 and u ( l − 1) :
Revised: , x ( l − 1) n l − 1 , 1) ∈ R n l − 1 +1 and u ( l − 1) :

Original: = ( u ( l − 1), . . .
Revised: = ( u ( l − 1), . . .

Original: , u ( l − 1) n l − 1 , 0) ∈
Revised: , u ( l − 1) n l − 1 , 0) ∈

Original: R n l − 1 +1 , ∥ u ∥ = 1 .
Revised: R n l − 1 +1 , ∥ u ∥ = 1 .

Original: We append the vectors of biases to the weight matrices and denote obtained matrices with W ( l ) ∈ R n l × ( n l − 1 +1) .
Revised: We append the vectors of biases to the weight matrices and denote obtained matrices with W ( l ) ∈ R n l × ( n l − 1 +1) .

Original: Then (9) equals
Revised: Then (9) equals

Original: [Equation]
Revised: [Equation]

Original: Now we focus on (cid:112) n l − 1 /c ⟨ W ( l ) i , u ( l − 1) ⟩ .
Revised: Now we focus on (cid:112) n l − 1 /c ⟨ W ( l ) i , u ( l − 1) ⟩ .

Original: Since we have assumed that the weights and biases are sampled from the Gaussian distribution with zero mean and variance c/n l − 1 , any weight W ( l ) i,k,j , j = 1 , . .
Revised: Since we have assumed that the weights and biases are sampled from the Gaussian distribution with zero mean and variance c/n l − 1 , any weight W ( l ) i,k,j , j = 1 , . .

Original: . , n l − 1 (or bias), can be written as (cid:112) c/n l − 1 v ( l ) i,k,j , where v ( l ) i,k,j is standard Gaussian.
Revised: . , n l − 1 (or bias), can be written as (cid:112) c/n l − 1 v ( l ) i,k,j , where v ( l ) i,k,j is standard Gaussian.

Original: Therefore, n l −
Revised: Therefore, n l −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where V ( l ) i
Revised: where V ( l ) i

Original: =
Revised: =

Original: V ( l ) i,k ′ ∈ R n l − 1 +1 , k ′ =
Revised: V ( l ) i,k ′ ∈ R n l − 1 +1 , k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] {⟨ V ( l ) i,k , x ( l − 1) ⟩} , V ( l ) i,k are ( n l − 1 + 1) - dimensional standard Gaussian random vectors.
Revised: [ K ] {⟨ V ( l ) i,k , x ( l − 1) ⟩} , V ( l ) i,k are ( n l − 1 + 1) - dimensional standard Gaussian random vectors.

Original: We construct an orthonormal basis B = ( b 1 , . . .
Revised: We construct an orthonormal basis B = ( b 1 , . . .

Original: , b n l − 1 +1 ) of R n l − 1 +1 , where we set b 1 = x ( l − 1) /
Revised: , b n l − 1 +1 ) of R n l − 1 +1 , where we set b 1 = x ( l − 1) /

Original: ∥ x ( l − 1) ∥ and choose the other vectors to be unit vectors orthogonal to b 1 .
Revised: ∥ x ( l − 1) ∥ and choose the other vectors to be unit vectors orthogonal to b 1 .

Original: The change of basis matrix from the standard basis I to the basis B is given by B T ; see, e.g., Anton & Rorres (2013, Theorem 6.6.4).
Revised: The change of basis matrix from the standard basis I to the basis B is given by B T ; see, e.g., Anton & Rorres (2013, Theorem 6.6.4).

Original: Then, any row V ( l ) i,k an be expressed as
Revised: Then, any row V ( l ) i,k an be expressed as

Original: [Equation]
Revised: [Equation]

Original: where c k,j = ⟨ V ( l ) i,k , b j ⟩ , j = 1 , . .
Revised: where c k,j = ⟨ V ( l ) i,k , b j ⟩ , j = 1 , . .

Original: n l − 1 + 1 .
Revised: n l − 1 + 1 .

Original: The coordinate vector of x ( l − 1) relative to B is ( ∥ x ( l − 1) ∥ , 0 , . . . , 0) .
Revised: The coordinate vector of x ( l − 1) relative to B is ( ∥ x ( l − 1) ∥ , 0 , . . . , 0) .

Original: Vector u ( l − 1) in B has the coordinate vector ( ⟨ u ( l − 1) , b 1 ⟩ , . . .
Revised: Vector u ( l − 1) in B has the coordinate vector ( ⟨ u ( l − 1) , b 1 ⟩ , . . .

Original: , ⟨ u ( l − 1) , b n l − 1 +1 ⟩ ) .
Revised: , ⟨ u ( l − 1) , b n l − 1 +1 ⟩ ) .

Original: This coordinate vector has norm 1 since the change of basis between two orthonormal bases does not change the ℓ 2 norm; see, e.g., Anton &
Revised: This coordinate vector has norm 1 since the change of basis between two orthonormal bases does not change the ℓ 2 norm; see, e.g., Anton &

Original: Rorres (2013, Theorem 6.3.2).
Revised: Rorres (2013, Theorem 6.3.2).

Original: For the maximum, using the representation of the vectors in the basis B , we get
Revised: For the maximum, using the representation of the vectors in the basis B , we get

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Therefore, in the basis B , V ( l ) i has components (max k ∈ [ K ] { c k, 1 } , c k ′ , 2 , . . .
Revised: Therefore, in the basis B , V ( l ) i has components (max k ∈ [ K ] { c k, 1 } , c k ′ , 2 , . . .

Original: , c k ′ ,n l − 1 +1 ) .
Revised: , c k ′ ,n l − 1 +1 ) .

Original: By Theorem 15, for all k = 1 , . . .
Revised: By Theorem 15, for all k = 1 , . . .

Original: , K , j = 1 , . .
Revised: , K , j = 1 , . .

Original: . , n l − 1 + 1 , the coefficients c k,j are mutually independent standard Gaussian random variables that are also independent of vectors b j , j = 1 , . .
Revised: . , n l − 1 + 1 , the coefficients c k,j are mutually independent standard Gaussian random variables that are also independent of vectors b j , j = 1 , . .

Original: . , n l − 1 , by
Revised: . , n l − 1 , by

Original: Lemma 10 and of u
Revised: Lemma 10 and of u

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: where Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and v j ∼ N (0 , 1) .
Revised: where Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and v j ∼ N (0 , 1) .

Original: Since we have simply written equality in distribution for max k ∈
Revised: Since we have simply written equality in distribution for max k ∈

Original: [ K ] { c k, 1 } and c k ′ ,j , the variables Ξ l,i ( N (0 , 1) , K ) and v j , j = 2 , . .
Revised: [ K ] { c k, 1 } and c k ′ ,j , the variables Ξ l,i ( N (0 , 1) , K ) and v j , j = 2 , . .

Original: . , n l − 1 are also mutually independent, and independent of vectors b j , j = 1 , . .
Revised: . , n l − 1 are also mutually independent, and independent of vectors b j , j = 1 , . .

Original: . , n l − 1 , and of u ( l − 1) .
Revised: . , n l − 1 , and of u ( l − 1) .

Original: In the following we will use Ξ l,i as a shorthand for Ξ l,i ( N (0 , 1) , K ) .
Revised: In the following we will use Ξ l,i as a shorthand for Ξ l,i ( N (0 , 1) , K ) .

Original: A linear combination (cid:80) ni
Revised: A linear combination (cid:80) ni

Original: =1 a i , v i of Gaussian random variables v 1 , . .
Revised: =1 a i , v i of Gaussian random variables v 1 , . .

Original: , v n , v j ∼ N ( µ j , σ 2 j ) , j = 1 , . . .
Revised: , v n , v j ∼ N ( µ j , σ 2 j ) , j = 1 , . . .

Original: , n with coefficients a 1 , . . .
Revised: , n with coefficients a 1 , . . .

Original: , a n is distributed according to N ( (cid:80) ni
Revised: , a n is distributed according to N ( (cid:80) ni

Original: =1 a i µ i , (cid:80) ni
Revised: =1 a i µ i , (cid:80) ni

Original: =1 a 2 i σ 2 i ) .
Revised: =1 a 2 i σ 2 i ) .

Original: Hence, (cid:80) n l − 1 +1 j = v j ⟨ u ( l − 1) , b j ⟩
Revised: Hence, (cid:80) n l − 1 +1 j = v j ⟨ u ( l − 1) , b j ⟩

Original: [Equation]
Revised: [Equation]

Original: N (0 , (cid:80) n l − 1 +1 j = ⟨ u ( l − 1) , b j ⟩ 2 ) .
Revised: N (0 , (cid:80) n l − 1 +1 j = ⟨ u ( l − 1) , b j ⟩ 2 ) .

Original: Since (cid:80) n l − 1 +1 j = ⟨ u ( l − 1) , b j ⟩ 2 = 1 − ⟨ u ( l − 1) , b 1 ⟩ 2 = 1 − cos 2 γ x ( l − 1) , u ( l − 1) , we get n l − 1 +1 (cid:88)
Revised: Since (cid:80) n l − 1 +1 j = ⟨ u ( l − 1) , b j ⟩ 2 = 1 − ⟨ u ( l − 1) , b 1 ⟩ 2 = 1 − cos 2 γ x ( l − 1) , u ( l − 1) , we get n l − 1 +1 (cid:88)

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: (13) where v i ∼ N (0 , 1) .
Revised: (13) where v i ∼ N (0 , 1) .

Original: Notice that v i (cid:112) 1 − cos 2 γ x ( l − 1) , u ( l − 1) and Ξ l,i cos γ u ( l − 1) , x ( l − 1) are stochastically independent because v i and Ξ l,i are independent and multiplying random variables by constants does not affect stochastic independence.
Revised: Notice that v i (cid:112) 1 − cos 2 γ x ( l − 1) , u ( l − 1) and Ξ l,i cos γ u ( l − 1) , x ( l − 1) are stochastically independent because v i and Ξ l,i are independent and multiplying random variables by constants does not affect stochastic independence.

Original: Remark 17.
Revised: Remark 17.

Original: The result in Theorem 3 also holds when the biases are initialized to zero.
Revised: The result in Theorem 3 also holds when the biases are initialized to zero.

Original: The proof is simplified in this case.
Revised: The proof is simplified in this case.

Original: There is no need to define additional vectors x ( l − 1) and u ( l − 1) , and when constructing the basis, the first vector is defined as b 1 := x ( l − 1) /
Revised: There is no need to define additional vectors x ( l − 1) and u ( l − 1) , and when constructing the basis, the first vector is defined as b 1 := x ( l − 1) /

Original: ∥ x ( l − 1) ∥ .
Revised: ∥ x ( l − 1) ∥ .

Original: The rest of the proof remains the same.
Revised: The rest of the proof remains the same.

Original: Remark 18 (Effects of the width and depth on a maxout network) .
Revised: Remark 18 (Effects of the width and depth on a maxout network) .

Original: According to Theorem 3, the behavior of ∥ J N ( x ) u ∥ 2 in a maxout network depends on the cos γ x ( l − 1) , u ( l − 1) , which changes as the network gets wider or deeper.
Revised: According to Theorem 3, the behavior of ∥ J N ( x ) u ∥ 2 in a maxout network depends on the cos γ x ( l − 1) , u ( l − 1) , which changes as the network gets wider or deeper.

Original: Figure 6 demonstrates how the width and depth affect ∥ J N ( x ) u ∥ 2 .
Revised: Figure 7 demonstrates how the width and depth affect ∥ J N ( x ) u ∥ 2 .

Original: Wide shallow networks Since independent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal (Vershynin, 2018, Remark 2.3.5), cos γ x (0) , u (0) will be close
Revised: Wide shallow networks Since independent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal (Vershynin, 2018, Remark 2.3.5), cos γ x (0) , u (0) will be close

Original: [Equation]
Revised: [Equation]

Original: to 0 with high probability for wide networks if the entries of the vectors x and u are i.i.d.
Revised: to 0 with high probability for wide networks if the entries of the vectors x and u are i.i.d.

Original: standard
Revised: standard

Original: Gaussian (or i.i.d.
Revised: Gaussian (or i.i.d.

Original: from an isotropic distribution).
Revised: from an isotropic distribution).

Original: Hence, we expect that the cosine will be around zero for the earlier layers of wide networks and individual units will behave more as the squared standard Gaussians.
Revised: Hence, we expect that the cosine will be around zero for the earlier layers of wide networks and individual units will behave more as the squared standard Gaussians.

Original: Wide deep networks Consider wide and deep networks, where the layers l = 0 , . .
Revised: Wide deep networks Consider wide and deep networks, where the layers l = 0 , . .

Original: , L − 1 are approximately of the same width n l 1 ≈ n l 2 , l 1 , l 2 = 0 , . . .
Revised: , L − 1 are approximately of the same width n l 1 ≈ n l 2 , l 1 , l 2 = 0 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: Assume that c = 1 / M = 1 / E [(Ξ( N (0 , 1) , K )) 2 ] .
Revised: Assume that c = 1 / M = 1 / E [(Ξ( N (0 , 1) , K )) 2 ] .

Original: We will demonstrate that under these conditions.
Revised: We will demonstrate that under these conditions.

Original: | cos γ x ( l ) , u ( l ) | ≈ 1 for the later layers for 2 < K < 100 .
Revised: | cos γ x ( l ) , u ( l ) | ≈ 1 for the later layers for 2 < K < 100 .

Original: Thus, individual units behave as the squared largest order statistics.
Revised: Thus, individual units behave as the squared largest order statistics.

Original: To see this, we need to estimate
Revised: To see this, we need to estimate

Original: cos γ x ( l ) , u ( l ) from Theorem 3, which is defined as cos
Revised: cos γ x ( l ) , u ( l ) from Theorem 3, which is defined as cos

Original: [Equation]
Revised: [Equation]

Original: where we denoted cos γ x ( l ) , u ( l ) with ρ ( l ) xu , and with u ( l ) , u ( l ) before the normalization.
Revised: where we denoted cos γ x ( l ) , u ( l ) with ρ ( l ) xu , and with u ( l ) , u ( l ) before the normalization.

Original: Firstly, for x ( l ) we get
Revised: Firstly, for x ( l ) we get

Original: [Equation]
Revised: [Equation]

Original: l −
Revised: l −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where in the second line we used that W ( l ) i,k = (cid:112) c/n l − 1 V ( l ) i,k , V ( l ) i,k,j ∼ N (0 , 1) , j = 1 , . .
Revised: where in the second line we used that W ( l ) i,k = (cid:112) c/n l − 1 V ( l ) i,k , V ( l ) i,k,j ∼ N (0 , 1) , j = 1 , . .

Original: . , n l − 1 .
Revised: . , n l − 1 .

Original: In the third line, Ξ l,i d = Ξ( N (0 , 1) , K ) is the largest order statistic in a sample of K standardWidth [Table]
Revised: In the third line, Ξ l,i d = Ξ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard

Original: Width
Revised: Width

Original: [Table]
Revised: [Table]

Original: Width
Revised: Width

Original: [Table]
Revised: [Table]

Original: Width
Revised: Width

Original: [Table]
Revised: [Table]

Original: Width1 l a y e r 2 l a y e r s 5 l a y e r s 10 l a y e r s 30 l a y e r s
Revised: Width1 l a y e r 2 l a y e r s 5 l a y e r s 10 l a y e r s 30 l a y e r s

Original: Gaussians, since by Lemma 10, V ( l ) i,k x ( l − 1) /
Revised: Gaussians, since by Lemma 10, V ( l ) i,k x ( l − 1) /

Original: ∥ x ( l − 1) ∥ are mutually independent standard Gaussian random variables.
Revised: ∥ x ( l − 1) ∥ are mutually independent standard Gaussian random variables.

Original: When the network width is large, 1 /n l (cid:80) n l i =1 Ξ 2 l,i approximates the second moment of the largest order statistic
Revised: When the network width is large, 1 /n l (cid:80) n l i =1 Ξ 2 l,i approximates the second moment of the largest order statistic

Original: [Equation]
Revised: [Equation]

Original: and n l − 1 /n l ≈ 1 when the layer widths are approximately the same.
Revised: and n l − 1 /n l ≈ 1 when the layer widths are approximately the same.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Now we will show that 1 / ∥ x ( l − 1) ∥ 2 ≈ 0 .
Revised: Now we will show that 1 / ∥ x ( l − 1) ∥ 2 ≈ 0 .

Original: Firstly, by the same reasoning as above,
Revised: Firstly, by the same reasoning as above,

Original: [Equation]
Revised: [Equation]

Original: Since we assumed thatthat the layer widths are large and approximately the same, ∥
Revised: Since we assumed that the layer widths are large and approximately the same, ∥

Original: [Equation]
Revised: [Equation]

Original: Using the assumption that c = 1 / E
Revised: Using the assumption that c = 1 / E

Original: [Ξ 2 ] , we obtain that ∥ x ( l − 1) ∥ 2 ≈ ∥ x (0) ∥ 2 + ( l − 1) and goes to infinity with the network depth.
Revised: [Ξ 2 ] , we obtain that ∥ x ( l − 1) ∥ 2 ≈ ∥ x (0) ∥ 2 + ( l − 1) and goes to infinity with the network depth.

Original: Hence, 1 / ∥ x ( l − 1) ∥ 2 ≈ 0 and n l −
Revised: Hence, 1 / ∥ x ( l − 1) ∥ 2 ≈ 0 and n l −

Original: [Equation]
Revised: [Equation]

Original: Now consider u ( l ) .
Revised: Now consider u ( l ) .

Original: Using the reasoning from Theorem 3, see equations (12) and (13), u ( l ) i
Revised: Using the reasoning from Theorem 3, see equations (12) and (13), u ( l ) i

Original: d = c/n l − 1 (Ξ l,i ρ ( l − 1) xu 
Revised: d = c/n l − 1 (Ξ l,i ρ ( l − 1) xu +

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  = [Equation]
Revised: ) , i = 1 , . .

Original: . , n l , v i ∼ N [Equation] [Equation]
Revised: . , n l , v i ∼ N (0 , 1) .

Original: Then in a wide network
Revised: Then in a wide network

Original: [Equation]
Revised: [Equation]

Original: Note that the random variable Ξ in equations (14) and (15) is the same based on the derivations in
Revised: Note that the random variable Ξ in equations (14) and (15) is the same based on the derivations in

Original: Theorem 3, to see this, compare equations (11) and (12).
Revised: Theorem 3, to see this, compare equations (11) and (12).

Original: Similarly, for the dot product ⟨ x ( l ) , u ( l ) ⟩ in a wide network we obtain that
Revised: Similarly, for the dot product ⟨ x ( l ) , u ( l ) ⟩ in a wide network we obtain that

Original: [Equation]
Revised: [Equation]

Original: Hence, we have the following recursive map for ρ ( l ) xu
Revised: Hence, we have the following recursive map for ρ ( l ) xu

Original: [Equation]
Revised: [Equation]

Original: where we used independence of v and Ξ , see Theorem 3, and that E [ v ] = 0 and E [ v 2 ] = 1 .
Revised: where we used independence of v and Ξ , see Theorem 3, and that E [ v ] = 0 and E [ v 2 ] = 1 .

Original: This map has fixed points ρ ∗ = ± 1 , which can be confirmed by direct calculation.
Revised: This map has fixed points ρ ∗ = ± 1 , which can be confirmed by direct calculation.

Original: To check if these fixed points are stable, we need to consider the values of the derivative ∂ρ ( l )
Revised: To check if these fixed points are stable, we need to consider the values of the derivative ∂ρ ( l )

Original: xu /∂ρ
Revised: xu /∂ρ

Original: ( l − 1) xu at them.
Revised: ( l − 1) xu at them.

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: When ρ ( l − 1) xu = ± 

Revised: When ρ ( l − 1) xu = ±

Original: this partial derivative equals 1 / E
Revised: this partial derivative equals 1 / E

Original: [Ξ 2 ] < 1 for K > 2 , since E [Ξ 2 ] > 1 , see Table 4 for K = 2 , . .
Revised: [Ξ 2 ] < 1 for K > 2 , since E [Ξ 2 ] > 1 , see Table 4 for K = 2 , . .

Original: , 10 and Figure 5 for K = 2 , . .
Revised: , 10 and Figure 6 for K = 2 , . .

Original: Hence, the fixed points are stable (Strogatz, 2018, Chapter 10.1).
Revised: Hence, the fixed points are stable (Strogatz, 2018, Chapter 10.1).

Original: Note that for K = 2 , 1 / E
Revised: Note that for K = 2 , 1 / E

Original: [Ξ 2 ] = 1 , and this analysis is inconclusive.
Revised: [Ξ 2 ] = 1 , and this analysis is inconclusive.

Original: Therefore, if the network parameters are sampled from N (0 , c/n l − 1 ) , c = 1 / M = 1 / E [Ξ( N (0 , 1) , K ) 2 ] , we expect that | cos γ x ( l ) , u ( l ) | ≈ 1 for the later layers of deep networks and individual units will behave more as the squared largest order statistics.
Revised: Therefore, if the network parameters are sampled from N (0 , c/n l − 1 ) , c = 1 / M = 1 / E [Ξ( N (0 , 1) , K ) 2 ] , we expect that | cos γ x ( l ) , u ( l ) | ≈ 1 for the later layers of deep networks and individual units will behave more as the squared largest order statistics.

Original: Figure 3 demonstrates convergence of | cos γ x ( l ) , u ( l ) | towith the depth for wide networks, and Figure 4 shows that there is no convergence for c < 1 / E
Revised: Figure 4 demonstrates convergence of | cos γ x ( l ) , u ( l ) | towith the depth for wide networks, and Figure 5 shows that there is no convergence for c < 1 / E

Original: [Ξ 2 ] and that the cosine still converges for c > 1 / E
Revised: [Ξ 2 ] and that the cosine still converges for c > 1 / E

Original: [Ξ 2 ] .
Revised: [Ξ 2 ] .

Original: Remark 19 (Expectation of ∥ J N ( x ) u ∥ 2 in a wide and deep network) .
Revised: Remark 19 (Expectation of ∥ J N ( x ) u ∥ 2 in a wide and deep network) .

Original: According to Remark 18, for deep and wide networks, we can expect that | cos γ x ( l − 1) , u ( l − 1) |
Revised: According to Remark 18, for deep and wide networks, we can expect that | cos γ x ( l − 1) , u ( l − 1) |

Original: ≈ 

Revised: ≈

Original: if c = 1 / M , which allows obtaining an approximate equality for the expectation of ∥ J N ( x ) u ∥ 2 .
Revised: if c = 1 / M , which allows obtaining an approximate equality for the expectation of ∥ J N ( x ) u ∥ 2 .

Original: Hence, using Theorem 3,
Revised: Hence, using Theorem 3,

Original: [Equation]
Revised: [Equation]

Original: Then, using mutual independence of the variables in equation (16),
Revised: Then, using mutual independence of the variables in equation (16),

Original: [Equation]
Revised: [Equation]

Original: Since M = E [(Ξ l,i ( N (0 , 1) , K )) 2 ] , see Table 4, and c = 1 / M , we get
Revised: Since M = E [(Ξ l,i ( N (0 , 1) , K )) 2 ] , see Table 4, and c = 1 / M , we get

Original: [Equation]
Revised: [Equation]

Revised: Remark 20 (Lower bound on the moments in a wide and deep network) .

Revised: Using (16) and taking into account the mutual independence of the variables,

Revised: [Equation]

Revised: [Equation]

Revised: where in the last inequality, we used linearity of expectation and Jensen’s inequality since taking the t th power for t ≥ 1 is a convex function for non-negative arguments.

Revised: Using the formula for noncentral moments of the chi-squared distribution and the inequality ln x ≥ 1 − 1 /x, ∀ x > 0 , meaning that x = exp { ln x } ≥ exp { 1 − 1 /x } , we get

Revised: [Equation]

Revised: where in the last inequality, we used that 2 i/ ( n L + 2 i ) ≥ 2 / ( n L + 2) ≥ 1 / (2 n L ) for all i, n L ≥ 1 .

Revised: Using that E [(Ξ l,i ( N (0 , 1) , K )) 2 ]

Revised: = M , see Table 4, and combing this with (17) and (18),

Revised: [Equation]

Revised: The bound in (19) can be tightened if a tighter lower bound on the moments of the sum of the squared largest order statistics in a sample of K standard Gaussians is known.

Revised: To derive a lower bound on the moments t ≥ 2 for the general case in Corollary 2, it is necessary to obtain a non-trivial lower bound on the moments of the sum of the smallest order statistics in a sample of K chi-squared random variables with 1 degree of freedom.

Original: E A CTIVATION LENGTH
Revised: E A CTIVATION LENGTH

Original: Here we prove the results from Subsection 3.3.
Revised: Here we prove the results from Subsection 3.3.

Original: Figure 7 demonstrates a close match between the estimated normalized activation length and the behavior predicted in Corollary 20 and 5.
Revised: Figure 8 demonstrates a close match between the estimated normalized activation length and the behavior predicted in Corollary 21 and 5.

Original: Corollary 20 (Distribution of the normalized activation length) .
Revised: Corollary 21 (Distribution of the normalized activation length) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 and l ′ = 1 , . . .
Revised: Then, almost surely with respect to the parameter initialization, for any input into the network x ∈ R n 0 and l ′ = 1 , . . .

Original: , L − 1 , the normalized activation length
Revised: , L − 1 , the normalized activation length

Original: A ( l ′ ) is equal in distribution to
Revised: A ( l ′ ) is equal in distribution to

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: where x (0) :
Revised: where x (0) :

Original: = ( x 1 , . . .
Revised: = ( x 1 , . . .

Original: , x n 0 , 1) ∈ R n 0 +1 , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and Ξ l,i ( N (0 , 1) , K ) are stochastically independent.
Revised: , x n 0 , 1) ∈ R n 0 +1 , Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and Ξ l,i ( N (0 , 1) , K ) are stochastically independent.

Original: Notice that variables Ξ l,i ( N (0 , 1) , K ) with the same indices are the same random variables.
Revised: Notice that variables Ξ l,i ( N (0 , 1) , K ) with the same indices are the same random variables.

Original: Proof.
Revised: Proof.

Original: Define x ( l ) =
Revised: Define x ( l ) =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n l , 1) ∈ R n l +1 .
Revised: , x n l , 1) ∈ R n l +1 .

Original: Append the bias columns to the weight matrices and denote obtained matrices with W ( l ) ∈ R n l × ( n l − 1 +1) .
Revised: Append the bias columns to the weight matrices and denote obtained matrices with W ( l ) ∈ R n l × ( n l − 1 +1) .

Original: Denote W ( l ) i,k ′ ∈ R n l − 1 +1 , k ′ =
Revised: Denote W ( l ) i,k ′ ∈ R n l − 1 +1 , k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] {⟨ W ( l ) i,k , x ( l − 1) ⟩} , with W ( l ) i .
Revised: [ K ] {⟨ W ( l ) i,k , x ( l − 1) ⟩} , with W ( l ) i .

Original: Under this notation, ∥ x ( l ) ∥ 2 = ( ∥ x ( l ) ∥ 2 + 1) , ∥ x ( l ) ∥ =
Revised: Under this notation, ∥ x ( l ) ∥ 2 = ( ∥ x ( l ) ∥ 2 + 1) , ∥ x ( l ) ∥ =

Original: ∥ W ( l ) x
Revised: ∥ W ( l ) x

Original: ( l − 1) ∥ .
Revised: ( l − 1) ∥ .

Original: Then ∥ x ( l ′ )
Revised: Then ∥ x ( l ′ )

Original: ∥ 2 equals ∥ x ( l ′ )
Revised: ∥ 2 equals ∥ x ( l ′ )

Original: ∥ 2 = ∥
Revised: ∥ 2 = [Equation]

Original: W
Revised: W

Revised: [Equation]

Original: l ′)
Revised: l ′

Original: x ( l ′ − 1) ∥ 2 = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) W ( l ′ ) x ( l ′ − 1) ∥ x ( l ′ − 1) ∥ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∥ x ( l ′ − 1) ∥= (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) W ( l ′ )
Revised: [Equation]

Original: x ( l ′ − 1) ∥ x ( l ′ − 1) ∥ (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) W ( l ′ − 1) x ( l ′ − 2) ∥ x ( l ′ − 2) ∥ (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: [Equation]

Original: ∥ x ( l′ − 2) ∥ 2 +
Revised: ∥ x ( l

Original:  =
Revised: [Equation]

Original: · · · = ∥ x (0) ∥l ′ (cid:89) l = (cid:13)(cid:13)(cid:13)(cid:13) W ( l ) x ( l − 1)

Original: ∥ x ( l − 1) ∥ (cid:13)(cid:13)(cid:13)(cid:13)
Revised: ) ∥ (cid:13)(cid:13)(cid:13)(cid:13)

Original: + l ′ (cid:88) j =   l ′ (cid:89) l =
Revised: [Equation]

Original: j (cid:13)(cid:13)(cid:13)(cid:13) W ( l ) x ( l − 1) ∥ x ( l − 1) ∥ (cid:13)(cid:13)(cid:13)(cid:13) 2   , where we multiplied and divided ∥ W ( l ) x ( l − 1) ∥ 2 by ∥ x ( l − 1) ∥ 2 at each step.
Revised: where we multiplied and divided ∥ W ( l ) x ( l − 1) ∥ 2 by ∥ x ( l − 1) ∥ 2 at each step.

Original: Using the approach from
Revised: Using the approach from

Original: Theorem 3, more specifically equations (10), (12) and (13), with u ( l ) =
Revised: Theorem 3, more specifically equations (10), (12) and (13), with u ( l ) =

Original: x ( l ) / ∥ x ( l ) ∥ , implying that cos γ x ( l ) , u ( l ) = 1 , 
Revised: x ( l ) / ∥ x ( l ) ∥ , implying that cos γ x ( l ) , u ( l ) = 1 , [Equation] 

Original: [Equation]
Revised: [Equation]

Original:  , where Ξ l,i =
Revised: where Ξ l,i =

Original: Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and stochastic independence of variables Ξ l,i follows from Theorem 3.
Revised: Ξ l,i ( N (0 , 1) , K ) is the largest order statistic in a sample of K standard Gaussian random variables, and stochastic independence of variables Ξ l,i follows from Theorem 3.

Original: Corollary 5 (Moments of the activation length) 
Revised: Corollary 5 (Moments of the activation length) .

Original: [Equation]

Original: Consider a maxout network with the settings of
Revised: Consider a maxout network with the settings of

Original: Section 2.
Revised: Section 2.

Original: Let x ∈ R n 0 be any input into the network.
Revised: Let x ∈ R n 0 be any input into the network.

Original: Then, for the moments of the normalized activation length, the following results hold.
Revised: Then, for the moments of the normalized activation length, the following results hold.

Original: Mean:
Revised: Mean:

Original: [Equation]

Original: Variance:

Original: Var[ A ( l

Original: [Equation]

Original: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: A ( l ′ ) ] ≥ ∥ x

Original: [Equation]
Revised: [Equation]

Original: Moments of the order t ≥ 2 :
Revised: Moments of the order t ≥ 2 :

Original: [Equation]
Revised: [Equation]

Original: where expectation is taken with respect to the distribution of the network weights and biases, and M is a constant depending on K that can be computed approximately, see Table 4 for the values for
Revised: where expectation is taken with respect to the distribution of the network weights and biases [Equation] and M is a constant depending on K that can be computed approximately, see Table 4 for the values for

Original: K = 2 , . .
Revised: K = 2 , . .

Original: Proof.
Revised: Proof.

Original: Mean Taking expectation in Corollary 20 and using independence of Ξ l,i ( N (0 , 1) , K ) ,
Revised: Mean Taking expectation in Corollary 21 and using independence of Ξ l,i ( N (0 , 1) , K ) ,

Original: [Equation]
Revised: [Equation]

Original: where M is the second moment of Ξ l,i ( N (0 , 1) , K ) , see Table 4 for its values for K = 2 , . .
Revised: where M is the second moment of Ξ l,i ( N (0 , 1) , K ) , see Table 4 for its values for K = 2 , . .

Original: Moments of the order t ≥ 2 Using Corollary 20, we get
Revised: Moments of the order t ≥ 2 Using Corollary 21, we get

Original: [Equation]
Revised: [Equation]

Original: Upper bound First, we derive an upper bound on (18).
Revised: Upper bound First, we derive an upper bound on (21).

Original: Notice that all arguments in (18) are positive except for a zero measure set of Ξ l,
Revised: Notice that all arguments in (21) are positive except for a zero measure set of Ξ l,

Original: i ∈ R (cid:80) l ′ l =1 n l .
Revised: i ∈ R (cid:80) l ′ l =1 n l .

Original: According to the power mean inequality, for any x 1 , . . .
Revised: According to the power mean inequality, for any x 1 , . . .

Original: , x n ∈ R , x 1 , . . .
Revised: , x n ∈ R , x 1 , . . .

Original: , x n > 0 and any t ∈ R , t > 1 , ( x 1 + · · · + x n ) t ≤ n t − 1 ( x t 1 + · · · + x tn ) .
Revised: , x n > 0 and any t ∈ R , t > 1 , ( x 1 + · · · + x n ) t ≤ n t − 1 ( x t 1 + · · · + x tn ) .

Original: Using the power mean inequality first on the whole expression and then on the second summand,
Revised: Using the power mean inequality first on the whole expression and then on the second summand,

Original: [Equation]
Revised: [Equation]

Original: Using independence of Ξ l,i , (19) equals
Revised: Using independence of Ξ l,i , (22) equals

Original: [Equation]
Revised: [Equation]

Original: Upper-bounding the largest order statistic with the sum of squared standard Gaussian random variables, we get that (cid:80) n l i =1 Ξ 2 l,i ≤ χ 2 n l K .
Revised: Upper-bounding the largest order statistic with the sum of squared standard Gaussian random variables, we get that (cid:80) n l i =1 Ξ 2 l,i ≤ χ 2 n l K .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Using the formula for noncentral moments of the chi-squared distribution and 1 + x ≤ e x , ∀ x ∈ R , c t
Revised: Using the formula for noncentral moments of the chi-squared distribution and 1 + x ≤ e x , ∀ x ∈ R , c t

Original: [Equation]
Revised: [Equation]

Original: where we used the formula for calculating the sum of consecutive numbers (cid:80) t − 1 i
Revised: where we used the formula for calculating the sum of consecutive numbers (cid:80) t − 1 i

Original: =

Revised: =

Original: i = t ( t − 1) / 2 .
Revised: i = t ( t − 1) / 2 .

Original: Using this result in (20), we get the final upper bound
Revised: Using this result in (23), we get the final upper bound

Original: E

Original: [Equation]
Revised: [Equation]

Original: Lower bound Using the Jensen’s inequality twice, we lower bound (18) with ∥
Revised: 

Original: [Equation]
Revised: [Equation]

Original: Lower bounding the largest order statistic with one squared Gaussian random variable
Revised:  

Original: [Equation]
Revised: [Equation]

Original: we get (cid:80) n l i =1 Ξ 2 l,i ≥ χ 2 n l . Hence, [Equation]
Revised: Lower bound Using that arguments in (21) are non-negative and t ≥ 1 , we can lower bound the power of the sum with the sum of the powers and get,

Original: [Equation]
Revised: [Equation]

Original: Using the formula for noncentral moments of the chi-squared distribution and the logarithm inequality ln x ≥ 1 − 1 /x, ∀ x > 0 , meaning that x = exp { ln x } ≥ exp { 1 − 1 /x } , we get c t 
Revised: l = j   c t n

Original: [Equation]
Revised: [Equation]

Original: Combining this with (21), we obtain the lower bound on the moments
Revised: l

Original: [Equation]
Revised: [Equation]

Original: Variance We can use an upper bound on the second moment as an upper bound on the variance.
Revised: n [Equation] = [Equation]

Revised:   

Original: To obtain a lower bound, we start with an upper bound on the square of the mean. Taking the square of (17) and using the AM-QM inequality twice,
Revised:  , where we used the linearity of expectation in both expressions and Jensen’s inequality in the last line. Using that E [(Ξ l,i ( N (0 , 1) , K )) 2 ] = M , see Table 4, we get

Original: [Equation]
Revised: [Equation]

Original: Combining this with the lower bound on the moments for t = 2 from (22), we get [Equation]
Revised: Variance We can use an upper bound on the second moment as an upper bound on the variance.

Original: Remark 21 (Zero bias) .
Revised: Remark 22 (Zero bias) .

Original: Similar results can be obtained for the zero bias case and would result in the same bounds without the second summand.
Revised: Similar results can be obtained for the zero bias case and would result in the same bounds without the second summand.

Original: For the proof one would work directly with the vectors x ( l ) , without defining the vectors x ( l ) , and to obtain the equality in distribution one would use
Revised: For the proof one would work directly with the vectors x ( l ) , without defining the vectors x ( l ) , and to obtain the equality in distribution one would use

Original: Remark 17.
Revised: Remark 17.

Original: F E XPECTED NUMBER OF LINEAR REGIONS
Revised: F E XPECTED NUMBER OF LINEAR REGIONS

Original: Here we prove the result from Section 5.1.
Revised: Here we prove the result from Section 5.1.

Original: Corollary 6 (Value for C grad ) .
Revised: Corollary 6 (Value for C grad ) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Consider the pre-activation feature ζ z,k of a unit z = 1 , . .
Revised: Consider the pre-activation feature ζ z,k of a unit z = 1 , . .

Original: . , N .
Revised: . , N .

Original: Then, for any t ∈ N , (cid:18)
Revised: Then, for any t ∈ N , (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Distribution of ∇ ζ z,k is the same as the distribution of the gradient with respect to the network input ∇ (cid:101) N 1 ( x ) in a maxout network that has a single linear output unit and ˜ L = l ( z ) layers, where l ( z ) is the depth of a unit z .
Revised: Distribution of ∇ ζ z,k is the same as the distribution of the gradient with respect to the network input ∇ (cid:101) N 1 ( x ) in a maxout network that has a single linear output unit and ˜ L = l ( z ) layers, where l ( z ) is the depth of a unit z .

Original: Therefore, we will consider (sup x ∈ R n 0 E [ ∥∇ (cid:101) N 1 ( x ) ∥ 2 t ]) 1 / 2 t .
Revised: Therefore, we will consider (sup x ∈ R n 0 E [ ∥∇ (cid:101) N 1 ( x ) ∥ 2 t ]) 1 / 2 t .

Original: Notice that since n ˜ L = 1 , ∇ (cid:101) N 1 ( x ) =
Revised: Notice that since n ˜ L = 1 , ∇ (cid:101) N 1 ( x ) =

Original: J (cid:101) N ( x ) T =
Revised: J (cid:101) N ( x ) T =

Original: J (cid:101) N ( x ) T u for a 1 -dimensional vector u
Revised: J (cid:101) N ( x ) T u for a 1 -dimensional vector u

Original: = (1) .
Revised: = (1) .

Original: Hence,
Revised: Hence,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the matrix norm is the spectral norm.
Revised: where the matrix norm is the spectral norm.

Original: Using that a matrix and its transpose have the same spectral norm, (23) equals
Revised: Using that a matrix and its transpose have the same spectral norm, (25) equals

Original: [Equation]
Revised: [Equation]

Original: Therefore, we need to upper bound 
Revised: Therefore, we need to upper bound 

Original: [Equation]
Revised: [Equation]

Original: where we used Jensen’s inequality.
Revised: where we used Jensen’s inequality.

Original: Now we can use an upper bound on E [ ∥ J (cid:101) N ( x ) u ∥ 2 t ] from Corollary 2, which holds for any x , u ∈ R n, ∥ u ∥ = 1 , and thus holds for the suprema.
Revised: Now we can use an upper bound on E [ ∥ J (cid:101) N ( x ) u ∥ 2 t ] from Corollary 2, which holds for any x , u ∈ R n, ∥ u ∥ = 1 , and thus holds for the suprema.

Original: Recalling that n ˜ L = 1 , we get 
Revised: Recalling that n ˜ L = 1 , we get [Equation]    

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: Taking the maximum over ˜ L ∈ { 1 , . .
Revised: Taking the maximum over ˜ L ∈ { 1 , . .

Original: , L } , the final upper bound is
Revised: , L } , the final upper bound is

Original: [Equation]
Revised: [Equation]

Original: Now we provide an updated upper bound on the number of r -partial activation regions from Tseran & Montúfar (2021, Theorem 9).
Revised: Now we provide an updated upper bound on the number of r -partial activation regions from Tseran & Montúfar (2021, Theorem 9).

Original: In this bound, the case r = 0 corresponds to the number of linear regions.
Revised: In this bound, the case r = 0 corresponds to the number of linear regions.

Original: For a detailed discussion of the activation regions of maxout networks and their differences from linear regions, see Tseran & Montúfar (2021).
Revised: For a detailed discussion of the activation regions of maxout networks and their differences from linear regions, see Tseran & Montúfar (2021).

Original: Since the proof of Tseran & Montúfar (2021,
Revised: Since the proof of Tseran & Montúfar (2021,

Original: Theorem 9) only uses C grad for t ≤ n 0 , we obtain the following statement.
Revised: Theorem 9) only uses C grad for t ≤ n 0 , we obtain the following statement.

Original: Theorem 22 (Upper bound on the expected number of partial activation regions) .
Revised: Theorem 23 (Upper bound on the expected number of partial activation regions) .

Original: Consider a maxout network with the settings of Section 2 with N maxout units.
Revised: Consider a maxout network with the settings of Section 2 with N maxout units.

Original: Assume that the biases are independent of the weights and initialized so that:
Revised: Assume that the biases are independent of the weights and initialized so that:

Original: Every collection of biases has a conditional density with respect to Lebesgue measure given the values of all other weights and biases.
Revised: Every collection of biases has a conditional density with respect to Lebesgue measure given the values of all other weights and biases.

Original: There exists C bias > 

Revised: There exists C bias >

Original: so that for any pre-activation features ζ 1 , . .
Revised: so that for any pre-activation features ζ 1 , . .

Original: . , ζ t from any neurons, the conditional density of their biases ρ b 1 ,...,b t given all the other weights and biases satisfies
Revised: . , ζ t from any neurons, the conditional density of their biases ρ b 1 ,...,b t given all the other weights and biases satisfies

Original: [Equation]
Revised: [Equation]

Original: Fix r ∈ { 0 , . . . , n 0 } .
Revised: Fix r ∈ { 0 , . . . , n 0 } .

Original: Let C grad = n − 1 / 2max { 1 , ( cK ){+
 [Equation]+}
Revised: Let C grad = n − 1 / {+
 [Equation]+}

Original: ( L − 1) / 2 } exp { n 0 / 2( (cid:80) L − 1 l =1 1 / ( n l K )+ 1) } and T = 2 5 C grad C bias .
Revised: [Equation] ) } and T = 2 5 C grad C bias .

Original: Then, there exists δ 0 ≤ 1 / (2 C grad C bias ) such that for all cubes C ⊆ R n 0 with side length δ > δ 

Revised: Then, there exists δ 0 ≤ 1 / (2 C grad C bias ) such that for all cubes C ⊆ R n 0 with side length δ > δ

Original: we have
Revised: we have

Original: E [# r -partial activation regions of N in C ]
Revised: E [# r -partial activation regions of N in C ]

Original: [Equation]
Revised: [Equation]

Revised: ) r n 0 ! ,

Original:  Here the expectation is taken with respect to the distribution of weights and biases in N .
Revised: N ≥ n. Here the expectation is taken with respect to the distribution of weights and biases in N .

Original: Of particular interest is the case r = 0 , which corresponds to the number of linear regions.
Revised: Of particular interest is the case r = 0 , which corresponds to the number of linear regions.

Original: G E XPECTED CURVE LENGTH DISTORTION
Revised: G E XPECTED CURVE LENGTH DISTORTION

Original: In this section we prove the result from Section 5.2.
Revised: In this section we prove the result from Section 5.2.

Original: Let M be a smooth 1 -dimensional curve in R n 0 .
Revised: Let M be a smooth 1 -dimensional curve in R n 0 .

Original: Fix a smooth unit speed parameterization of
Revised: Fix a smooth unit speed parameterization of

Original: M = γ ([0 , 1]) with γ : R → R n 0 , γ ( τ ) =
Revised: M = γ ([0 , 1]) with γ : R → R n 0 , γ ( τ ) =

Original: ( γ 1 ( τ ) , . . .
Revised: ( γ 1 ( τ ) , . . .

Original: , γ n 0 ( τ )) .
Revised: , γ n 0 ( τ )) .

Original: Then, parametrization of the curve
Revised: Then, parametrization of the curve

Original: N ( M ) is given by a mapping Γ :=
Revised: N ( M ) is given by a mapping Γ :=

Original: N ◦ γ , Γ : R → R n L .
Revised: N ◦ γ , Γ : R → R n L .

Original: Thus, the length of N ( M ) islen
Revised: Thus, the length of N ( M ) is

Original: [Equation]
Revised: [Equation]

Original: Notice that the input-output Jacobian of maxout networks is well defined almost everywhere because for any neuron, using that the biases are independent from weights, and the weights are initialized from a continuous distribution, P ( k ′ =
Revised: Notice that the input-output Jacobian of maxout networks is well defined almost everywhere because for any neuron, using that the biases are independent from weights, and the weights are initialized from a continuous distribution, P ( k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , k ′′ =
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } , k ′′ =

Original: argmax k ∈ [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } )
Revised: argmax k ∈ [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } )

Original: = 0 , i = 1 , . .
Revised: = 0 , i = 1 , . .

Original: . , n L − 1 .
Revised: . , n L − 1 .

Original: Hence, Γ ′ ( τ ) =
Revised: Hence, Γ ′ ( τ ) =

Original: J N ( γ ( τ )) γ ′ ( τ ) , where we used the chain rule, and we can employ the following lemma from Hanin et al.
Revised: J N ( γ ( τ )) γ ′ ( τ ) , where we used the chain rule, and we can employ the following lemma from Hanin et al.

Original: We state it here without proof which uses Tonelli’s theorem, power mean inequality and chain rule.
Revised: We state it here without proof which uses Tonelli’s theorem, power mean inequality and chain rule.

Original: Lemma 23 (Connection between the length of the curve and ∥ J N ( x ) u ∥ , Hanin et al. 2021,
Revised: Lemma 24 (Connection between the length of the curve and ∥ J N ( x ) u ∥ , Hanin et al. 2021,

Original: Lemma C.1) .
Revised: Lemma C.1) .

Original: For any integer t ≥ 0 ,
Revised: For any integer t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where u ∈ R n 0 is a unit vector.
Revised: where u ∈ R n 0 is a unit vector.

Original: Now we are ready to proof Corollary 7.
Revised: Now we are ready to proof Corollary 7.

Original: Corollary 7 (Expected curve length distortion) .
Revised: Corollary 7 (Expected curve length distortion) .

Original: Consider a maxout network with the settings of
Revised: Consider a maxout network with the settings of

Original: Section 2.
Revised: Section 2.

Original: Assume that the biases are independent of the weights but otherwise initialized using any approach.
Revised: Assume that the biases are independent of the weights but otherwise initialized using any approach.

Original: Let M be a smooth 1 -dimensional curve of unit length in R n 0 .
Revised: Let M be a smooth 1 -dimensional curve of unit length in R n 0 .

Original: Then, the following upper bounds on the moments of len ( N ( M )) hold:
Revised: Then, the following upper bounds on the moments of len ( N ( M )) hold:

Original: [Equation]
Revised: [Equation]

Original: where L is a constant depending on K , see Table 4 in Appendix C for values for K = 2 , . .
Revised: where L is a constant depending on K , see Table 4 in Appendix C for values for K = 2 , . .

Original: Proof.
Revised: Proof.

Original: By Lemma 23,
Revised: By Lemma 24,

Original: [Equation]
Revised: [Equation]

Original: 12 , where we used Jensen’s inequality to obtain the last upper bound.
Revised: where we used Jensen’s inequality to obtain the last upper bound.

Original: Hence, using Corollary 2, we get the following upper bounds on the moments on the length of the curve. 
Revised: Hence, using Corollary 2, we get the following upper bounds on the moments on the length of the curve. Mean [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] Moments of the order

Revised: t ≥

Original: [Equation]
Revised: [Equation]

Revised: H NTK

Original: Here we prove the results from Section 5.3.
Revised: Here we prove the results from Section [Equation] [Equation]

Original: Corollary 9 (On-diagonal NTK) .
Revised: Corollary 9 (On-diagonal NTK) .

Original: Consider a maxout network with the settings of Section 2.
Revised: Consider a maxout network with the settings of Section 2.

Original: Assume that n L = 1 and that the biases are initialized to zero and are not trained.
Revised: Assume that n L = 1 and that the biases are initialized to zero and are not trained.

Original: Assume that S ≤ c ≤ L , where the constants S , L are as specified in Table 4.
Revised: Assume that S ≤ c ≤ L , where the constants S , L are as specified in Table 4.

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where P = (cid:80) L − 1 l =0 n l , P W = (cid:80) Ll =0 n l n l − 1 , and M is as specified in Table 4.
Revised: where P = (cid:80) L − 1 l =0 n l , P W = (cid:80) Ll =0 n l n l − 1 , and M is as specified in Table 4.

Original: Proof.
Revised: Proof.

Original: Under the assumption that biases are not trained, on-diagonal NTK of a maxout network is
Revised: Under the assumption that biases are not trained, on-diagonal NTK of a maxout network is

Original: [Equation]
Revised: [Equation]

Original: Since in maxout network for all k -s except k = k ′ =
Revised: Since in maxout network for all k -s except k = k ′ =

Original: argmax k ∈
Revised: argmax k ∈

Original: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } the derivatives with respect to the weights and biases are zero, on-diagonal NTK equals
Revised: [ K ] { W ( l ) i,k x ( l − 1) + b ( l ) i,k } the derivatives with respect to the weights and biases are zero, on-diagonal NTK equals

Original: [Equation]
Revised: [Equation]

Original: Notice that since we assumed a continuous distribution over the network weights and the biases are zero, the partial derivatives are defined everywhere except for the set of measure zero.
Revised: Notice that since we assumed a continuous distribution over the network weights and the biases are zero, the partial derivatives are defined everywhere except for the set of measure zero.

Original: Part I. Kernel mean E [ K N (x , x)]
Revised: Part I. Kernel mean E [ K N (x , x)]

Original: Firstly, using the chain rule, a partial derivative with respect to the network weight is
Revised: Firstly, using the chain rule, a partial derivative with respect to the network weight is

Original: [Equation]
Revised: [Equation]

Original: Recall that we assumed n L = 1 .
Revised: Recall that we assumed n L = 1 .

Original: Therefore, we need to consider ( ∂ N ( x ) ∂W ( l ) i,k ′ ,j ) 2 =
Revised: Therefore, we need to consider ( ∂ N ( x ) ∂W ( l ) i,k ′ ,j ) 2 =

Original: ∥
Revised: ∥

Original: J N ( x ( l ) ) u
Revised: J N ( x ( l ) ) u

Original: ∥ 2 ( x ( l − 1) j ) 2 , where u = e i .
Revised: ∥ 2 ( x ( l − 1) j ) 2 , where u = e i .

Original: Combining Theorem 1 and Corollary 20 in combination with Remark 21 for the zero-bias case and using the independence of the random variables in the expressions,
Revised: Combining Theorem 1 and Corollary 21 in combination with Remark 22 for the zero-bias case and using the independence of the random variables in the expressions,

Original: [Equation]
Revised: [Equation]

Original: where we treat the ( l − 1) th layer as if it has one unit when we use the normalized activation length result.
Revised: where we treat the ( l − 1) th layer as if it has one unit when we use the normalized activation length result.

Original: Then, using Corollaries 2 and 5,
Revised: Then, using Corollaries 2 and 5,

Original: [Equation]
Revised: [Equation]

Original: Taking the sum
Revised: Taking the sum, we get

Revised: E [ K N ( x , x )] =

Revised: E [ K W ] =

Revised: E

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: where P = (cid:80) L − 1 l =0 n l denotes the number of neurons in the network up to the last layer, but including the input neurons.
Revised: where P = (cid:80) L − 1 l =0 n l denotes the number of neurons in the network up to the last layer, but including the input neurons.

Original: Here we used that for K ≥ 2 , both L , M ≥ 1 , see Table 4.
Revised: Here we used that for K ≥ 2 , both L , M ≥ 1 , see Table 4.

Original: Similarly,
Revised: Similarly,

Revised: E [ K W ] ≥

Original: [Equation]
Revised: [Equation]

Original: Here we used that for K ≥ 2 , S ≤ 1 and M ≥ 1 , see Table 4.
Revised: Here we used that for K ≥ 2 , S ≤ 1 and M ≥ 1 , see Table 4.

Original: Part II.
Revised: Part II.

Original: Second moment E [ K N (x , x) 2 ] Using equation (24) with Corollaries 2 and 5, 
Revised: Second moment E [ K N (x , x) 2 ] Using equation (26) with Corollaries 2 and 5, E  [Equation] ≤ so E

Revised: [Equation]

Revised: j

Original: [Equation]
Revised: [Equation]

Original: Notice that all summands are non-negative.
Revised: Notice that all summands are non-negative.

Original: Then, using AM-QM inequality,
Revised: Then, using AM-QM inequality,

Revised: E [ K N ( x , x ) 2 ] =

Revised: E    

Revised: L (cid:88) l = n l (cid:88) i = n l − 1 (cid:88)

Original: [Equation]
Revised: j = (cid:32) [Table] N [Table]

Revised: ( l ) i,k

Revised: [Table]

Revised: [Table]

Revised: x

Original: [Table]
Revised: [Table]

Revised: ≤ P W

Revised: [Table]

Revised: (cid:88)

Revised: [Table]

Revised: ≤

Revised: [Table]

Revised: PP W

Revised: [Table]

Original:  where P W = (cid:80) Ll =0 n l n l − 1 denotes the number of all weights in the network.
Revised: cK ) [Table]   , where P W = (cid:80) L l =0 n l n l − 1 denotes the number of all weights in the network.

Original: I E XPERIMENTS 
Revised: I E XPERIMENTS I. E XPERIMENTS WITH SGD AND A DAM FROM S ECTION

Original: In this section we provide more details on the experiments presented in Section 6.
Revised: In this subsection we provide more details on the experiments presented in Section 6.

Original: The implementation of the key routines is available at https://anonymous.4open.science/r/maxout_expected_ gradient-68BD .
Revised: The implementation of the key routines is available at https://anonymous.4open.science/r/maxout_expected_ gradient-68BD .

Original: Experiments were implemented in Python using TensorFlow (Martín Abadi et al., 2015), numpy (Harris et al., 2020) and mpi4py (Dalcin et al., 2011).
Revised: Experiments were implemented in Python using TensorFlow (Martín Abadi et al., 2015), numpy (Harris et al., 2020) and mpi4py (Dalcin et al., 2011).

Original: The plots were created using matplotlib (Hunter, 2007).
Revised: The plots were created using matplotlib (Hunter, 2007).

Original: We conducted all training experiments  on a GPU cluster with nodes having 4 Nvidia A100 GPUs with 40 GB of memory.
Revised: We conducted all training experiments from Section 6 on a GPU cluster with nodes having 4 Nvidia A100 GPUs with 40 GB of memory.

Original: The most extensive experiments were running for one day on one GPU.
Revised: The most extensive experiments were running for one day on one GPU.

Original: Experiment in Figure 2 was run on a CPU cluster that uses Intel Xeon IceLakeSP processors (Platinum 8360Y) with 72 cores per node and 256 GB RAM.
Revised: Experiment in Figure 2 was run on a CPU cluster that uses Intel Xeon IceLakeSP processors (Platinum 8360Y) with 72 cores per node and 256 GB RAM.

Original: All other experiments were executed on the laptop ThinkPad T470 with Intel Core i5-7200U CPU with 16 GB RAM.
Revised: All other experiments were executed on the laptop ThinkPad T470 with Intel Core i5-7200U CPU with 16 GB RAM.

Original: Training experiments
Revised: Training experiments

Original: Now we discuss the training experiments.
Revised: Now we discuss the training experiments.

Original: We use MNIST (LeCun & Cortes, 2010), Iris (Fisher, 1936), Fashion MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFARand CIFAR-100 datasets (Krizhevsky et al., 2009).
Revised: We use MNIST (LeCun & Cortes, 2010), Iris (Fisher, 1936), Fashion MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFARand CIFAR-100 datasets (Krizhevsky et al., 2009).

Original: All maxout networks have the maxout rank K = 5 .
Revised: All maxout networks have the maxout rank K = 5 .

Original: Weights are sampled from N (0 , c/ fan-in ) in fully-connected networks and N (0 , c/ ( k 2 · fan-in )) , where k is the kernel size, in CNNs.
Revised: Weights are sampled from N (0 , c/ fan-in ) in fully-connected networks and N (0 , c/ ( k 2 · fan-in )) , where k is the kernel size, in CNNs.

Original: The biases are initialized to zero.
Revised: The biases are initialized to zero.

Original: ReLU networks are initialized using He approach (He et al., 2015), meaning that c = 2 .
Revised: ReLU networks are initialized using He approach (He et al., 2015), meaning that c = 2 .

Original: All results are averaged over 4 runs.
Revised: All results are averaged over 4 runs.

Original: We do not use any weight normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015).
Revised: We do not use any weight normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015).

Original: We performed the dataset split into training, validation and test dataset and report the accuracy on the test set, while the validation set was used only for picking the hyper-parameters and was not used in training.
Revised: We performed the dataset split into training, validation and test dataset and report the accuracy on the test set, while the validation set was used only for picking the hyper-parameters and was not used in training.

Original: The mini-batch size in all experiments is 32 .
Revised: The mini-batch size in all experiments is 32 .

Original: The number of training epochs was picked by observing the training set loss and choosing the number of epochs for which the loss has converged.
Revised: The number of training epochs was picked by observing the training set loss and choosing the number of epochs for which the loss has converged.

Original: The exception is the SVHN dataset, for which we observe the double descent phenomenon and stop training after 150 epochs.
Revised: The exception is the SVHN dataset, for which we observe the double descent phenomenon and stop training after 150 epochs.

Original: Network architecture Fully connected networks have 21 layers.
Revised: Network architecture Fully connected networks have 21 layers.

Original: Specifically, their architecture is
Revised: Specifically, their architecture is

Original: [Equation]
Revised: [Equation]

Original:  where “ 5 × fc 64 ” means that there are 5 fully-connected layers with 64 neurons, and “out” stands for the output layer that has the number of neurons equal to the number of classes in a dataset.
Revised: , 5 × fc 32 , 5 × fc 16 , 5 × fc 8 , out ] , where “ 5 × fc 64 ” means that there are 5 fully-connected layers with 64 neurons, and “out” stands for the output layer that has the number of neurons equal to the number of classes in a dataset.

Original: CNNs have a VGG-19-like architecture (Simonyan & Zisserman, 2015) with 20 or 16 layers, depending the input size.
Revised: CNNs have a VGG-19-like architecture (Simonyan & Zisserman, 2015) with 20 or 16 layers, depending the input size.

Original: The 20 -layer architecture is [Equation] where “conv 64 ” stands for a convolutional layer with 64 neurons and “mp” for a max-pooling layer.
Revised: The 20 -layer architecture is [2 × conv 64 , mp , 2 × conv 128 , mp , 4 × conv 256 , mp , 4 × conv 512 , mp , 4 × conv 512 , mp , 2 × fc 4096 , fc 1000 , out ] , where “conv 64 ” stands for a convolutional layer with 64 neurons and “mp” for a max-pooling layer.

Original: The kernel size in all convolutional layers is 3 × 3 .
Revised: The kernel size in all convolutional layers is 3 × 3 .

Original: Max-pooling uses 2 × 2 pooling windows with stride 2 .
Revised: Max-pooling uses 2 × 2 pooling windows with stride 2 .

Original: Such architecture is used for datasets with the images that have the side length greater or equal to 32 : CIFAR-10, CIFAR-100 and SVHN.
Revised: Such architecture is used for datasets with the images that have the side length greater or equal to 32 : CIFAR-10, CIFAR-100 and SVHN.

Original: The 16 -layer architecture is used for images with the smaller image size: MNIST and Fashion MNIST.
Revised: The 16 -layer architecture is used for images with the smaller image size: MNIST and Fashion MNIST.

Original: This architecture does not have the last convolutional block of the 20 -layer version.
Revised: This architecture does not have the last convolutional block of the 20 -layer version.

Original: Concretely, it has the following layers: 
Revised: Concretely, it has the following layers: [2 × conv 64 , mp , 2 × conv 128 , mp , 4 × conv 256 , mp , 4 × conv 512 , mp , 2 × fc 4096 , fc 1000 , out ] , Max-pooling initialization To account for the maximum in max-pooling layers, a maxout layer appearing after a max-pooling layer is initialized as if its maxout rank was K × m 2 , where m 2 is the max-pooling window size. The reason for this is that the outputs of a computational block consisting of a max-pooling window and a maxout layer are taking maxima over K × m 2 linear functions, max { W 1 max { x 1 , . . . , x m 2 } + b 1 , . .

Revised: , W K max { x 1 , . . .

Revised: , x m 2 } + b K } = max { f 1 , . . .

Original: [Equation]
Revised: , f Km 2 } , where the f i are Km 2 affine functions. Therefore, we initialize the layers that follow max-pooling layers using the criterion for maxout rank m 2 × K instead of K . In our experiments, K = 5 , m = 2 , and m 2 × K = 20 . Hence, for such layers, we use the constant c = 1 /M = 0 .

Revised: 26573 , where M is computed for K = 20 using the formula from Remark 14 in Appendix C. All other layers that do not follow max-pooling layers are initialized as suggested in Section 4.

Revised: We observe that max-pooling initialization often leads to slightly higher accuracy.

Original: Data augmentation There is no data augmentation for fully connected networks.
Revised: Data augmentation There is no data augmentation for fully connected networks.

Original: For convolutional networks, for MNIST, Fashion MNIST and SVHN datasets we perform random translation, rotation and zoom of the input images.
Revised: For convolutional networks, for MNIST, Fashion MNIST and SVHN datasets we perform random translation, rotation and zoom of the input images.

Original: For CIFAR-10 and CIFAR-100, we additionally apply a random horizontal flip.
Revised: For CIFAR-10 and CIFAR-100, we additionally apply a random horizontal flip.

Original: Learning rate decay In all experiments, we use the learning rate decay.
Revised: Learning rate decay In all experiments, we use the learning rate decay and choose the optimal initial learning rate for all network and initialization types based on their accuracy on the validation dataset.

Original: The learning rate was halved every n th epoch.
Revised: The learning rate was halved every n th epoch.

Original: For a specific dataset, n was decided based on the dataset complexity, network type, optimization method, andperformance of maxout networks with maxout initialization on the validation dataset. Obtained n was then used for all networks in the same dataset experiment during the optimal learning rate search.
Revised: For SVHN, n = 10 , and for all other datasets, n = 100 .

Original: SGD with momentum We use SGD with Nesterov momentum, with the momentum value of 0 . 9 .
Revised: SGD with momentum We use SGD with Nesterov momentum, with the momentum value of 0 . 9 .

Original: We perform the initial learning rate search for maxout networks with the maxout initialization and

Original: ReLU networks.

Original: Other maxout networks use the same learning rate as the maxout network with the maxout initialization because we observed that changing the learning rate for c = 0 .

Original: 1 and c = 2 does not affect the network performance much, and, for the max-pooling initialization, in our experiments, the optimal learning rate was usually the same as for the maxout initialization.

Original: Specific dataset settings are the following.
Revised: Specific dataset settings are the following.

Original: • MNIST (fully-connected networks).
Revised: • MNIST (fully-connected networks).

Original: Networks are trained for 600 epochs.
Revised: Networks are trained for 600 epochs.

Original: Learning rate is halved every 100 epochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks: 0 . 002 ,  ReLU networks: 0 . 0008 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . maxout networks with c = 0 . 1 : 0 . 002 , maxout networks with c = 2 : 2 × 10 − 7 , ReLU networks: 0 . 002 .

Original: • Iris.
Revised: • Iris.

Original: Networks are trained for 500 epochs.
Revised: Networks are trained for 500 epochs.

Original: Learning rate is halved every 50 epochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks: 0 . 009 , ReLU networks: 0 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . maxout networks with c = 0 . 01 , maxout networks with c = 2 : 4 × 10 − 8 , ReLU networks: 0 .

Original: • MNIST (convolutional networks).
Revised: • MNIST (convolutional networks).

Original: Networks are trained for 800 epochs.
Revised: Networks are trained for 800 epochs.

Original: Learning rate is halved every 140 epochs.
Revised: The learning rate is halved every 100 epochs.

Revised: Learning rates: maxout networks with maxout initialization: 0 . 009 , maxout networks with max-pooling initialization: 0 . 009 , maxout networks with c = 0 .

Original: Learning rates: maxout networks: 0 . 009 ,  ReLU networsk: 0 .009 .
Revised: 1 : 0 . 009 , maxout networks with c = 2 : 8 × 10 − 6 , ReLU networks: 0 .

Original: • Fashion MNIST.
Revised: • Fashion MNIST.

Original: Networks are trained for 800 epochs.
Revised: Networks are trained for 800 epochs.

Original: Learning rate is halved everyepochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks: 0 . 005 , ReLU networks: 0 .
Revised: Learning rates: maxout networks with maxout initialization: 0 .

Revised: 004 , maxout networks with max-pooling initialization: 0 . 006 , maxout networks with c = 0 . 1 : 0 .

Revised: 4 , maxout networks with c = 2 : 5 × 10 − 6 , ReLU networks: 0 . 01 .

Original: • CIFAR-10.
Revised: • CIFAR-10.

Original: Networks are trained for 1000 epochs.
Revised: Networks are trained for 1000 epochs.

Original: Learning rate is halved every 140 epochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks: 0 . 004 ,  ReLU networks: 0 . 
Revised: Learning rates: maxout networks with maxout initialization: 0 . 004 , maxout networks with max-pooling initialization: 0 . 005 , maxout networks with c = 0 . 5 , maxout networks with c = 2 : 8 × 10 − 8 , ReLU networks: 0 . 009 .

Original: • CIFAR-100.
Revised: • CIFAR-100.

Original: Networks are trained for 1000 epochs.
Revised: Networks are trained for 1000 epochs.

Original: Learning rate is halved everyepochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks: 0 . 002 ,  ReLU networks: 0 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . 002 , maxout networks with max-pooling initialization: maxout networks with c = 0 . 1 : 0 . 002 , maxout networks with c = 2 : 8 × 10 − 5 , ReLU networks: 0 .

Revised: [Table]

Original: • SVHN.
Revised: • SVHN.

Original: Networks are trained for 150 epochs.
Revised: Networks are trained for 150 epochs.

Original: Learning rate is halved every 10 epochs.
Revised: The learning rate is halved every 10 epochs.

Original: Learning rates: maxout networks: 0 . , ReLU networks: 0 .
Revised: networks

Original: Adam We use Adam optimizer Kingma & Ba (2015) with default TensorFlow parameters β 1 = 0 .
Revised: Adam We use Adam optimizer Kingma & Ba (2015) with default TensorFlow parameters β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: We search for the optimal value of the initial learning rate for all network types except for maxout networks with the max-pooling initialization, which use the same learning rate as the maxout networks with maxout initialization because we observed that their optimal learning rates are usually the same.

Original: Specific dataset settings are the following.
Revised: Specific dataset settings are the following.

Original: • MNIST (fully-connected networks).
Revised: • MNIST (fully-connected networks).

Original: Networks are trained for 600 epochs.
Revised: Networks are trained for 600 epochs.

Original: Learning rate is halved every 50 epochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks with maxout and max-pooling initializations: 0 . 0004 , maxout networks with c = 2 : 0 . 0005 , ReLU networks: 0 . 0006 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . 0008 , maxout networks with c = 2 : 0 . 0007 , ReLU networks: 0 . 0008 .

Original: • MNIST (convolutional networks).
Revised: • MNIST (convolutional networks).

Original: Networks are trained for 1000 epochs.
Revised: Networks are trained for 800 epochs.

Original: Learning rate is halved every 100 epochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks with maxout and max-pooling initializations: 0 . 00007 , maxout networks with c = 2 : 0 . 00006 , ReLU networks: 0 . 00008 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . 0001 , maxout networks with max-pooling initialization: 0 . 00006 , maxout networks with c = 2 : 0 . 00004 , ReLU networks: 0 . 00009 .

Original: • Fashion MNIST.
Revised: • Fashion MNIST.

Original: Networks are trained for 1000 epochs.
Revised: Networks are trained for 1000 epochs.

Original: Learning rate is halved everyepochs.
Revised: The learning rate is halved every 100 epochs.

Original: Learning rates: maxout networks with maxout and max-pooling initializations: 0 . 0001 , maxout networks with c = 2 : 0 . 00004 , ReLU networks: 0 . 0001 .
Revised: Learning rates: maxout networks with maxout initialization: 0 . 00007 , maxout networks with max-pooling initialization: 0 . 00008 , maxout networks with c = 2 : 0 . 00005 , ReLU networks: 0 . 0002 .

Original: • CIFAR-10.
Revised: • CIFAR-10.

Original: Networks are trained for 1000 epochs.
Revised: Networks are trained for 1000 epochs.

Original: Learning rate is halved every 100 epochs.
Revised: The learning rate is halved everyepochs.

Original: Learning rates: maxout networks with maxout and max-pooling initializations: 0 . 
Revised: Learning rates: maxout networks with maxout initialization: 0 . 00009 , maxout networks with max-pooling initialization:

Original: 00009 , maxout networks with c = 2 : 0 . 00005 , ReLU networks: 0 . 0001 .
Revised: 00009 , maxout networks with c = 2 : 0 . 00005 , ReLU networks: 0 . 0001 .

Original: • CIFAR-100.
Revised: • CIFAR-100.

Original: Networks are trained for 1400 epochs.
Revised: Networks are trained for 1000 epochs.

Original: Learning rate is halved everyepochs.
Revised: The learning rate is halved everyepochs.

Original: Learning rates: maxout networks with maxout and max-pooling initializations: 0 . 00008 , maxout networks with  c = 2 : 0 . 00005 , ReLU networks: 0 . 00007 . 
Revised: Learning rates: maxout networks with maxout initialization: 0 . 00008 , maxout networks with max-pooling initialization: 0 . 00009 , maxout networks with c = 2 : 0 . 00005 , ReLU networks: 0 . 00009 . I. A BLATION ANALYSIS

Revised: Tables 5 and 6 show the results of the additional experiments that use SGD with Nesterov momentum for more values of c and K = 5 .

Revised: From this, we see that the recommended value of c from Sectionclosely matches the empirical optimum value of c .

Revised: Note that here we have fixed the learning rate across choices of c .

Revised: More specifically, the following learning rates were used for the

Revised: experiments with different datasets.

Revised: MNIST with fully-connected networks: 0 . 002 ; Iris: 0 . 01 ;

Revised: MNIST with convolutional networks: 0 . 009 ; CIFAR-10: 0 .

Revised: I. B ATCH NORMALIZATION

Revised: Tables 7 and 8 report test accuracy for maxout networks with batch normalization trained using

Revised: SGD with Nesterov momentum for various values of c .

Revised: The implementation of the experiments is similar to that described in Section I.1, except for the following differences: The networks use batch normalization after each layer with activations; The width of the last fully connected layer is 100 , and all other layers of the convolutional networks are 8 times narrower; The learning rate is fixed at 0 .

Revised: 01 for all experiments.

Revised: We use the default batch normalization parameters from TensorFlow.

Revised: Specifically, the momentum equals 0 . 99 and ϵ = 0 .

Revised: We observe that our initialization strategy is still beneficial when training with batch normalization.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: I. C OMPARISON OF MAXOUT AND R E LU NETWORKS IN TERMS OF THE NUMBER OF PARAMETERS

Revised: We should point out that what is a fair comparison is not as straightforward as matching the parameter count.

Revised: In particular, wider networks have the advantage of having a higher dimensional representation.

Revised: A fully connected network will not necessarily perform as well as a convolutional network with the same number of parameters, and a deep and narrow network will not necessarily perform as well as a wider and shallower network with the same number of parameters.

Revised: Nevertheless, to add more details to the results

Revised: [Table]

Revised: we perform experiments using ReLU networks that have as many parameters as maxout networks.

Revised: See Tables 9 and 10 for results.

Revised: We modify network architectures described in Section I.1 for these experiments in the following way.

Revised: In the first experiment, we use fully connected ReLU networks 5 times wider than maxout networks.

Revised: For convolutional networks, however, the resulting CNNs with ReLU activations would be extremely wide, so we only made it 2 times wider.

Revised: In our setup, a 5 times wider CNN network would need to be trained for longer than 24 hours, which is difficult in our experiment environment.

Revised: Maxout networks only required a much shorter time of around 10 hours, which indicates possible benefits in some cases.

Revised: In the second experiment, we consider ReLU networks that are 5 times deeper than maxout networks.

Revised: More specifically.

Revised: they have the following architecture: [25 × fc 64 , 25 × fc 32 , 25 × fc 16 , 25 × fc 8 , out ] .

Revised: As expected, wider networks do better.

Revised: On the other hand, deeper ReLU networks of the same width do much worse than maxout networks.

Revised: [Table]

Revised: [Table]