Original: P ROVABLE B ENEFIT OF O RTHOGONAL I NITIALIZATION IN O PTIMIZING D EEP L INEAR N ETWORKS
Revised: P ROVABLE B ENEFIT OF O RTHOGONAL I NITIALIZATION IN O PTIMIZING D EEP L INEAR N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance.
Revised: The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance.

Original: Yet despite signiﬁcant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes.
Revised: Yet despite signiﬁcant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes.

Original: In this work, we analyze the effect of initialization in deep linear networks and provide for the ﬁrst time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.
Revised: In this work, we analyze the effect of initialization in deep linear networks and provide for the ﬁrst time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.

Original: We show that for deep networks, the width needed for efﬁcient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efﬁcient convergence with Gaussian initializations scales linearly in the depth.
Revised: We show that for deep networks, the width needed for efﬁcient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efﬁcient convergence with Gaussian initializations scales linearly in the depth.

Original: Our results demonstrate how the beneﬁts of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry .
Revised: Our results demonstrate how the beneﬁts of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Through their myriad successful applications across a wide range of disciplines, it is now well established that deep neural networks possess an unprecedented ability to model complex real-world datasets, and in many cases they can do so with minimal overﬁtting.
Revised: Through their myriad successful applications across a wide range of disciplines, it is now well established that deep neural networks possess an unprecedented ability to model complex real-world datasets, and in many cases they can do so with minimal overﬁtting.

Original: Indeed, the list of practical achievements of deep learning has grown at an astonishing rate, and includes models capable of human-level performance in tasks such as image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and machine translation (Wu et al., 2016).
Revised: Indeed, the list of practical achievements of deep learning has grown at an astonishing rate, and includes models capable of human-level performance in tasks such as image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and machine translation (Wu et al., 2016).

Original: Yet to each of these deep learning triumphs corresponds a large engineering effort to produce such a high-performing model.
Revised: Yet to each of these deep learning triumphs corresponds a large engineering effort to produce such a high-performing model.

Original: Part of the practical difﬁculty in designing good models stems from a proliferation of hyperparameters and a poor understanding of the general guidelines for their selection.
Revised: Part of the practical difﬁculty in designing good models stems from a proliferation of hyperparameters and a poor understanding of the general guidelines for their selection.

Original: Given a candidate network architecture, some of the most impactful hyperparameters are those governing the choice of the model’s initial weights.
Revised: Given a candidate network architecture, some of the most impactful hyperparameters are those governing the choice of the model’s initial weights.

Original: Although considerable study has been devoted to the selection of initial weights, relatively little has been proved about how these choices affect important quantities such as rate of convergence of gradient descent.
Revised: Although considerable study has been devoted to the selection of initial weights, relatively little has been proved about how these choices affect important quantities such as rate of convergence of gradient descent.

Original: In this work, we examine the effect of initialization on the rate of convergence of gradient descent in deep linear networks.
Revised: In this work, we examine the effect of initialization on the rate of convergence of gradient descent in deep linear networks.

Original: We provide for the ﬁrst time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.
Revised: We provide for the ﬁrst time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.

Original: In particular, we show that for deep networks, the width needed for efﬁcient convergence for orthogonal initializations is independent of the depth, whereas the width needed for efﬁcient convergence of Gaussian networks scales linearly in the depth.
Revised: In particular, we show that for deep networks, the width needed for efﬁcient convergence for orthogonal initializations is independent of the depth, whereas the width needed for efﬁcient convergence of Gaussian networks scales linearly in the depth.

Original: Orthogonal weight initializations have been the subject of a signiﬁcant amount of prior theoretical and empirical investigation.
Revised: Orthogonal weight initializations have been the subject of a signiﬁcant amount of prior theoretical and empirical investigation.

Original: For example, in a line of work focusing on dynamical isometry , it was found that orthogonal weights can speed up convergence for deep linear networks (Saxe et al., 2014; Advani & Saxe, 2017) and for deep non-linear networks (Pennington et al., 2018; Xiao et al., 2018;
Revised: For example, in a line of work focusing on dynamical isometry , it was found that orthogonal weights can speed up convergence for deep linear networks (Saxe et al., 2014; Advani & Saxe, 2017) and for deep non-linear networks (Pennington et al., 2018; Xiao et al., 2018;

Original: Gilboa et al., 2019; Chen et al., 2018; Pennington et al., 2017; Tarnowski et al., 2019; Ling & Qiu, 2019) when they operate in the linear regime.
Revised: Gilboa et al., 2019; Chen et al., 2018; Pennington et al., 2017; Tarnowski et al., 2019; Ling & Qiu, 2019) when they operate in the linear regime.

Original: In the context of recurrent neural networks, orthogonality can help improve the system’s stability.
Revised: In the context of recurrent neural networks, orthogonality can help improve the system’s stability.

Original: A main limitation of prior work is that it has focused almost exclusively on model’s properties at initialization.
Revised: A main limitation of prior work is that it has focused almost exclusively on model’s properties at initialization.

Original: In contrast, our analysis focuses on the beneﬁt of orthogonal initialization on the entire training process, thereby establishing a provable beneﬁt for optimization.
Revised: In contrast, our analysis focuses on the beneﬁt of orthogonal initialization on the entire training process, thereby establishing a provable beneﬁt for optimization.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: After reviewing related work in Section 2 and establishing some preliminaries in Section 3, we present our main positive result on efﬁcient convergence from orthogonal initialization in Section 4.
Revised: After reviewing related work in Section 2 and establishing some preliminaries in Section 3, we present our main positive result on efﬁcient convergence from orthogonal initialization in Section 4.

Original: In Section 5, we show that Gaussian initialization leads to exponentially long convergence time if the width is too small compared with the depth.
Revised: In Section 5, we show that Gaussian initialization leads to exponentially long convergence time if the width is too small compared with the depth.

Revised: In Section 6, we perform experiments to support our theoretical results.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep linear networks.
Revised: Deep linear networks.

Original: Despite the simplicity of their input-output maps, deep linear networks deﬁne high-dimensional non-convex optimization landscapes whose properties closely reﬂect those of their non-linear counterparts.
Revised: Despite the simplicity of their input-output maps, deep linear networks deﬁne high-dimensional non-convex optimization landscapes whose properties closely reﬂect those of their non-linear counterparts.

Original: For this reason, deep linear networks have been the subject of extensive theoretical analysis.
Revised: For this reason, deep linear networks have been the subject of extensive theoretical analysis.

Original: A line of work (Kawaguchi, 2016; Hardt & Ma, 2016; Lu & Kawaguchi, 2017; Yun et al., 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018) studied the landscape properties of deep linear networks.
Revised: A line of work (Kawaguchi, 2016; Hardt & Ma, 2016; Lu & Kawaguchi, 2017; Yun et al., 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018) studied the landscape properties of deep linear networks.

Original: Although it was established that all local minima are global under certain assumptions, these properties alone are still not sufﬁcient to guarantee global convergence or to provide a concrete rate of convergence for gradient-based optimization algorithms.
Revised: Although it was established that all local minima are global under certain assumptions, these properties alone are still not sufﬁcient to guarantee global convergence or to provide a concrete rate of convergence for gradient-based optimization algorithms.

Original: Another line of work directly analyzed the trajectory taken by gradient descent and established conditions that guarantee convergence to global minimum (Bartlett et al., 2018; Arora et al., 2018;
Revised: Another line of work directly analyzed the trajectory taken by gradient descent and established conditions that guarantee convergence to global minimum (Bartlett et al., 2018; Arora et al., 2018;

Original: Du & Hu, 2019).
Revised: Du & Hu, 2019).

Original: Most relevant to our work is the result of Du & Hu (2019), which shows that if the width of hidden layers is larger than the depth, gradient descent with Gaussian initialization can efﬁciently converge to a global minimum.
Revised: Most relevant to our work is the result of Du & Hu (2019), which shows that if the width of hidden layers is larger than the depth, gradient descent with Gaussian initialization can efﬁciently converge to a global minimum.

Original: Our result establishes that for Gaussian initialization, this linear dependence between width and depth is necessary, while for orthogonal initialization, the width can be independent of depth.
Revised: Our result establishes that for Gaussian initialization, this linear dependence between width and depth is necessary, while for orthogonal initialization, the width can be independent of depth.

Original: Our negative result for Gaussian initialization also signiﬁcantly generalizes the result of Shamir (2018), who proved a similar negative result for 1 -dimensional linear networks.
Revised: Our negative result for Gaussian initialization also signiﬁcantly generalizes the result of Shamir (2018), who proved a similar negative result for 1 -dimensional linear networks.

Original: Orthogonal weight initializations.
Revised: Orthogonal weight initializations.

Original: Orthogonal weight initializations have also found signiﬁcant success in non-linear networks.
Revised: Orthogonal weight initializations have also found signiﬁcant success in non-linear networks.

Original: In the context of feedforward models, the spectral properties of a network’s input-output Jacobian have been empirically linked to convergence speed (Saxe et al., 2014; Pennington et al., 2017; 2018; Xiao et al., 2018).
Revised: In the context of feedforward models, the spectral properties of a network’s input-output Jacobian have been empirically linked to convergence speed (Saxe et al., 2014; Pennington et al., 2017; 2018; Xiao et al., 2018).

Original: It was found that when this spectrum concentrates around 1 at initialization, a property dubbed dynamical isometry , convergence times improved by orders of magnitude.
Revised: It was found that when this spectrum concentrates around 1 at initialization, a property dubbed dynamical isometry , convergence times improved by orders of magnitude.

Original: The conditions for attaining dynamical isometry in the inﬁnitewidth limit were established by Pennington et al.
Revised: The conditions for attaining dynamical isometry in the inﬁnitewidth limit were established by Pennington et al.

Original: and basically require that input-output map to be approximately linear and for the weight matrices to be orthogonal.
Revised: and basically require that input-output map to be approximately linear and for the weight matrices to be orthogonal.

Original: Therefore the training time beneﬁts of dynamical isometry are likely rooted in the beneﬁts of orthogonality for deep linear networks, which we establish in this work.
Revised: Therefore the training time beneﬁts of dynamical isometry are likely rooted in the beneﬁts of orthogonality for deep linear networks, which we establish in this work.

Original: Orthogonal matrices are also frequently used in the context of recurrent neural networks, for which the stability of the state-to-state transition operator is determined by the spectrum of its Jacobian (Haber & Ruthotto, 2017; Laurent & von Brecht, 2016).
Revised: Orthogonal matrices are also frequently used in the context of recurrent neural networks, for which the stability of the state-to-state transition operator is determined by the spectrum of its Jacobian (Haber & Ruthotto, 2017; Laurent & von Brecht, 2016).

Original: Orthogonal matrices can improve the conditioning, leading to an ability to learn over long time horizons (Le et al., 2015; Henaff et al., 2016;
Revised: Orthogonal matrices can improve the conditioning, leading to an ability to learn over long time horizons (Le et al., 2015; Henaff et al., 2016;

Original: Chen et al., 2018; Gilboa et al., 2019).
Revised: Chen et al., 2018; Gilboa et al., 2019).

Original: While the beneﬁts of orthogonality can be quite large at initialization, little is known about whether or in what contexts these beneﬁts persist during training, a scenario that has lead to the development of efﬁcient methods of constraining the optimization to the orthogonal group (Wisdom et al., 2016; Vorontsov et al., 2017; Mhammedi et al., 2017).
Revised: While the beneﬁts of orthogonality can be quite large at initialization, little is known about whether or in what contexts these beneﬁts persist during training, a scenario that has lead to the development of efﬁcient methods of constraining the optimization to the orthogonal group (Wisdom et al., 2016; Vorontsov et al., 2017; Mhammedi et al., 2017).

Original: Although we do not study the recurrent setting in this work, an extension of our analysis might help determine when orthogonality is beneﬁcial in that setting.
Revised: Although we do not study the recurrent setting in this work, an extension of our analysis might help determine when orthogonality is beneﬁcial in that setting.

Original:  P RELIMINARIES 3. N OTATION
Revised:  P RELIMINARIES 3. N OTATION

Original: Let [ n ] = { 1 , 2 , . . . , n } .
Revised: Let [ n ] = { 1 , 2 , . . . , n } .

Original: Denote by I d the d × d identity matrix, and by I an identity matrix when its dimension is clear from context.
Revised: Denote by I d the d × d identity matrix, and by I an identity matrix when its dimension is clear from context.

Original: Denote by N ( µ, σ 2 ) the Gaussian distribution with mean µ and

Original: Denote by (cid:107)·(cid:107) the (cid:96) 2 norm of a vector or the spectral norm of a matrix.
Revised: Denote by (cid:107)·(cid:107) the (cid:96) 2 norm of a vector or the spectral norm of a matrix.

Original: Denote by (cid:107)·(cid:107) F the Frobenius norm of a matrix.
Revised: Denote by (cid:107)·(cid:107) F the Frobenius norm of a matrix.

Original: For a symmetric matrix A , let λ max ( A ) and λ min ( A ) be its maximum and minimum eigenvalues, and let λ i ( A ) be its i -th largest eigenvalue.
Revised: For a symmetric matrix A , let λ max ( A ) and λ min ( A ) be its maximum and minimum eigenvalues, and let λ i ( A ) be its i -th largest eigenvalue.

Original: For a matrix B ∈ R m × n , let σ i ( B ) be its i -th largest singular value ( i = 1 , 2 , . .
Revised: For a matrix B ∈ R m × n , let σ i ( B ) be its i -th largest singular value ( i = 1 , 2 , . .

Original: , min { m, n } ), and let σ max ( B ) =
Revised: , min { m, n } ), and let σ max ( B ) =

Original: σ 1 ( B ) , σ min ( B ) =
Revised: σ 1 ( B ) , σ min ( B ) =

Original: σ min { m,n } ( B ) .
Revised: σ min { m,n } ( B ) .

Original: Denote by vec ( A ) be the vectorization of a matrix A in column-ﬁrst order.
Revised: Denote by vec ( A ) be the vectorization of a matrix A in column-ﬁrst order.

Original: The Kronecker product between two matrices A ∈ R m 1 × n 1 and B ∈ R m 2 × n 2 is deﬁned as
Revised: The Kronecker product between two matrices A ∈ R m 1 × n 1 and B ∈ R m 2 × n 2 is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where a i,j is the element in the ( i, j ) -th entry of A .
Revised: where a i,j is the element in the ( i, j ) -th entry of A .

Original: We use the standard O ( · ) , Ω( · ) and Θ( · ) notation to hide universal constant factors.
Revised: We use the standard O ( · ) , Ω( · ) and Θ( · ) notation to hide universal constant factors.

Original: We also use C to represent a sufﬁciently large universal constant whose speciﬁc value can differ from line to line.
Revised: We also use C to represent a sufﬁciently large universal constant whose speciﬁc value can differ from line to line.

Original: 3. P ROBLEM S ETUP
Revised: 3. P ROBLEM S ETUP

Original: Suppose that there are n training examples { ( x k , y k ) } nk =1 ⊂ R d x × R d y .
Revised: Suppose that there are n training examples { ( x k , y k ) } nk =1 ⊂ R d x × R d y .

Original: Denote by X = ( x 1 , . . .
Revised: Denote by X = ( x 1 , . . .

Original: , x n ) ∈ R
Revised: , x n ) ∈ R

Original: d x × n the input data matrix and by Y =
Revised: d x × n the input data matrix and by Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) ∈ R d y × n the target matrix.
Revised: , y n ) ∈ R d y × n the target matrix.

Original: Consider an L -layer linear neural network with weight matrices W 1 , . .
Revised: Consider an L -layer linear neural network with weight matrices W 1 , . .

Original: , W L , which given an input
Revised: , W L , which given an input

Original: [Equation]
Revised: [Equation]

Original: where W i ∈ R d i × d i − 1 ( i = 1 , . .
Revised: where W i ∈ R d i × d i − 1 ( i = 1 , . .

Original: . , L ) , d 0 = d x , d L = d y , and α is a normalization constant which will be speciﬁed later according to the initialization scheme.
Revised: . , L ) , d 0 = d x , d L = d y , and α is a normalization constant which will be speciﬁed later according to the initialization scheme.

Original: We study the problem of training the deep linear network by minimizing the (cid:96) 2 loss over training data:
Revised: We study the problem of training the deep linear network by minimizing the (cid:96) 2 loss over training data:

Original: [Equation]
Revised: [Equation]

Original: The algorithm we consider to minimize the objective (2) is gradient descent with random initialization, which ﬁrst randomly samples the initial weight matrices { W i (0) }
Revised: The algorithm we consider to minimize the objective (2) is gradient descent with random initialization, which ﬁrst randomly samples the initial weight matrices { W i (0) }

Original: Li =1 from a certain distribution, and then updates the weights using gradient descent: for time t = 0 , 1 , 2 , . . . ,
Revised: Li =1 from a certain distribution, and then updates the weights using gradient descent: for time t = 0 , 1 , 2 , . . . ,

Original: [Equation]
Revised: [Equation]

Original: where η > 0 is the learning rate.
Revised: where η > 0 is the learning rate.

Original: For convenience, we denote W j : i =
Revised: For convenience, we denote W j : i =

Original: W j W j − 1 · · · W i (1 ≤ i ≤ j ≤ L ) and W i − 1: i
Revised: W j W j − 1 · · · W i (1 ≤ i ≤ j ≤ L ) and W i − 1: i

Original: = I ( i ∈
Revised: = I ( i ∈

Original: [ L ]) .
Revised: [ L ]) .

Original: The time index t is used on any variable that depends on W 1 , . .
Revised: The time index t is used on any variable that depends on W 1 , . .

Original: , W L to represent its value at time t , e.g., W j : i ( t ) =
Revised: , W L to represent its value at time t , e.g., W j : i ( t ) =

Original: W j ( t ) · · · W i ( t ) , (cid:96) ( t ) =
Revised: W j ( t ) · · · W i ( t ) , (cid:96) ( t ) =

Original: (cid:96) ( W 1 ( t ) , . . .
Revised: (cid:96) ( W 1 ( t ) , . . .

Original: , W L ( t )) , etc.
Revised: , W L ( t )) , etc.

Original:  E FFICIENT C ONVERGENCE USING O RTHOGONAL I NITIALIZATION
Revised:  E FFICIENT C ONVERGENCE USING O RTHOGONAL I NITIALIZATION

Original: In this section we present our main positive result for orthogonal initialization.
Revised: In this section we present our main positive result for orthogonal initialization.

Original: We show that orthogonal initialization enables efﬁcient convergence of gradient descent to a global minimum provided that the hidden width is not too small.
Revised: We show that orthogonal initialization enables efﬁcient convergence of gradient descent to a global minimum provided that the hidden width is not too small.

Original: In order to properly deﬁne orthogonal weights, we let the widths of all hidden layers be equal:
Revised: In order to properly deﬁne orthogonal weights, we let the widths of all hidden layers be equal:

Original: d 1 = d 2 = · · · = d L − 1 = m , and let m ≥ max { d x , d y } .
Revised: d 1 = d 2 = · · · = d L − 1 = m , and let m ≥ max { d x , d y } .

Original: Note that all intermediate matrices
Revised: Note that all intermediate matrices

Original: W 2 , . .
Revised: W 2 , . .

Original: , W L − 1 are m × m square matrices, and W 1 ∈ R m × d x , W L ∈ R d y × m .
Revised: , W L − 1 are m × m square matrices, and W 1 ∈ R m × d x , W L ∈ R d y × m .

Original: We sample each initial weight matrix W i (0) independently from a uniform distribution over scaled orthogonal matrices satisfying
Revised: We sample each initial weight matrix W i (0) independently from a uniform distribution over scaled orthogonal matrices satisfying

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In accordance with such initialization, the scaling factor α in (1) is set as α = 1 √ m L − 1 d y , which ensures E (cid:104) (cid:107) f
Revised: In accordance with such initialization, the scaling factor α in (1) is set as α = 1 √ m L − 1 d y , which ensures E (cid:104) (cid:107) f

Original: ( x ; W L (0) , . . .
Revised: ( x ; W L (0) , . . .

Original: , W 1 (0))
Revised: , W 1 (0))

Original: (cid:107) 2 (cid:105) =
Revised: (cid:107) 2 (cid:105) =

Original: (cid:107) x (cid:107) 2 for any x ∈ R d x . 1 The same scaling factor was adopted in Du & Hu (2019), which preserves the expectation of the squared (cid:96) 2 norm of any input.
Revised: (cid:107) x (cid:107) 2 for any x ∈ R d x . 1 The same scaling factor was adopted in Du & Hu (2019), which preserves the expectation of the squared (cid:96) 2 norm of any input.

Original: Let W ∗ ∈ arg min W ∈ R dy × dx (cid:107) WX − Y (cid:107) F and (cid:96) ∗ = 12 (cid:107) W ∗
Revised: Let W ∗ ∈ arg min W ∈ R dy × dx (cid:107) WX − Y (cid:107) F and (cid:96) ∗ = 12 (cid:107) W ∗

Original: X − Y (cid:107) 2 F .
Revised: X − Y (cid:107) 2 F .

Original: Then (cid:96) ∗ is the minimum value for the objective (2).
Revised: Then (cid:96) ∗ is the minimum value for the objective (2).

Original: Denote r = rank( X ) , κ = λ max ( X (cid:62) X )
Revised: Denote r = rank( X ) , κ = λ max ( X (cid:62) X )

Original: λ r
Revised: λ r

Original: ( X (cid:62) X ) , and
Revised: ( X (cid:62) X ) , and

Original: ˜ r
Revised: ˜ r

Original: = (cid:107) X (cid:107) 2 F (cid:107) X (cid:107) 2 . 

Revised: = (cid:107) X (cid:107) 2 F (cid:107) X (cid:107) 2 .

Original: Our main theorem in this section is the following:
Revised: Our main theorem in this section is the following:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Suppose
Revised: Suppose

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: for some δ ∈ (0 , 1) and a sufﬁciently large universal constant C > 0 .
Revised: for some δ ∈ (0 , 1) and a sufﬁciently large universal constant C > 0 .

Original: Set the learning rate η ≤ d y 2 L (cid:107) X (cid:107) 2 .
Revised: Set the learning rate η ≤ d y 2 L (cid:107) X (cid:107) 2 .

Original: Then with probability at least 1 − δ over the random initialization, we have
Revised: Then with probability at least 1 − δ over the random initialization, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) ( t ) is the objective value at iteration t .
Revised: where (cid:96) ( t ) is the objective value at iteration t .

Original: Notably, in Theorem 4.1, the width m need not depend on the depth L .
Revised: Notably, in Theorem 4.1, the width m need not depend on the depth L .

Original: This is in sharp contrast with the result of Du & Hu (2019) for Gaussian initialization, which requires m ≥ ˜Ω( Lrκ 3 d y ) .
Revised: This is in sharp contrast with the result of Du & Hu (2019) for Gaussian initialization, which requires m ≥ ˜Ω( Lrκ 3 d y ) .

Original: It turns out that a near-linear dependence between m and L is necessary for Gaussian initialization to have efﬁcient convergence, as we will show in Section 5.
Revised: It turns out that a near-linear dependence between m and L is necessary for Gaussian initialization to have efﬁcient convergence, as we will show in Section 5.

Original: Therefore the requirement in Du & Hu (2019) is nearly tight in terms of the dependence on L .
Revised: Therefore the requirement in Du & Hu (2019) is nearly tight in terms of the dependence on L .

Original: These results together rigorously establish the beneﬁt of orthogonal initialization in optimizing very deep linear networks.
Revised: These results together rigorously establish the beneﬁt of orthogonal initialization in optimizing very deep linear networks.

Original: If we set the learning rate optimally according to Theorem 4.1 to η = Θ( d y L (cid:107) X (cid:107) 2 ) , we obtain that (cid:96) ( t ) − (cid:96) ∗ decreases by a ratio of 1 − Θ( κ − 1 ) after every iteration.
Revised: If we set the learning rate optimally according to Theorem 4.1 to η = Θ( d y L (cid:107) X (cid:107) 2 ) , we obtain that (cid:96) ( t ) − (cid:96) ∗ decreases by a ratio of 1 − Θ( κ − 1 ) after every iteration.

Original: This matches the convergence rate of gradient descent on the ( 1 -layer) linear regression problem
Revised: This matches the convergence rate of gradient descent on the ( 1 -layer) linear regression problem

Original: [Equation]
Revised: [Equation]

Original: 4. P ROOF OF T HEOREM 4. 
Revised: 4. P ROOF OF T HEOREM 4.

Original: The proof uses the high-level framework from Du & Hu (2019), which tracks the evolution of the network’s output during optimization.
Revised: The proof uses the high-level framework from Du & Hu (2019), which tracks the evolution of the network’s output during optimization.

Original: This evolution is closely related to a time-varying positive semideﬁnite (PSD) matrix (deﬁned in (7)), and the proof relies on carefully upper and lower bounding the eigenvalues of this matrix throughout training, which in turn implies the desired convergence result.
Revised: This evolution is closely related to a time-varying positive semideﬁnite (PSD) matrix (deﬁned in (7)), and the proof relies on carefully upper and lower bounding the eigenvalues of this matrix throughout training, which in turn implies the desired convergence result.

Original: First, we can make the following simplifying assumption without loss of generality .
Revised: First, we can make the following simplifying assumption without loss of generality .

Original: See Appendix
Revised: See Appendix

Original: B in Du & Hu (2019) for justiﬁcation.
Revised: B in Du & Hu (2019) for justiﬁcation.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: (Without loss of generality) X ∈ R d x × r , rank( X ) = r , Y = W ∗ X , and (cid:96) ∗ = 0 .
Revised: (Without loss of generality) X ∈ R d x × r , rank( X ) = r , Y = W ∗ X , and (cid:96) ∗ = 0 .

Original: Now we brieﬂy review Du & Hu (2019)’s framework.
Revised: Now we brieﬂy review Du & Hu (2019)’s framework.

Original: The key idea is to look at the network’s output, deﬁned as
Revised: The key idea is to look at the network’s output, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We also write U ( t ) =
Revised: We also write U ( t ) =

Original: αW L :1 ( t ) X as the output at time t .
Revised: αW L :1 ( t ) X as the output at time t .

Original: Note that (cid:96) ( t ) =
Revised: Note that (cid:96) ( t ) =

Original: 12 (cid:107) U ( t ) − Y (cid:107) 2 F .
Revised: 12 (cid:107) U ( t ) − Y (cid:107) 2 F .

Original: According to the gradient descent update rule, we write
Revised: According to the gradient descent update rule, we write

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: stable
Revised: stable

Original: , which is where E ( t ) contains all the high-order terms (i.e., those with η 2 or higher).
Revised: , which is where E ( t ) contains all the high-order terms (i.e., those with η 2 or higher).

Original: With this deﬁnition, the evolution of U ( t ) can be written as the following equation:
Revised: With this deﬁnition, the evolution of U ( t ) can be written as the following equation:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Notice that P ( t ) is always PSD since it is the sum of L PSD matrices.
Revised: Notice that P ( t ) is always PSD since it is the sum of L PSD matrices.

Original: Therefore, in order to establish convergence, we only need to (i) show that the higher-order term E ( t ) is small and (ii) prove upper and lower bounds on P ( t ) ’s eigenvalues.
Revised: Therefore, in order to establish convergence, we only need to (i) show that the higher-order term E ( t ) is small and (ii) prove upper and lower bounds on P ( t ) ’s eigenvalues.

Original: For the second task, it sufﬁces to control the singular values of W i − 1:1 ( t ) and W L : i +1 ( t ) ( i ∈
Revised: For the second task, it sufﬁces to control the singular values of W i − 1:1 ( t ) and W L : i +1 ( t ) ( i ∈

Original: [ L ] ).
Revised: [ L ] ).

Original: 3 Under orthogonal initialization, these matrices are perfectly isometric at initialization, and we will show that they stay close to isometry during training, thus enabling efﬁcient convergence.
Revised: 3 Under orthogonal initialization, these matrices are perfectly isometric at initialization, and we will show that they stay close to isometry during training, thus enabling efﬁcient convergence.

Original: The following lemma summarizes some properties at initialization.
Revised: The following lemma summarizes some properties at initialization.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: At initialization, we have
Revised: At initialization, we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, with probability at least 1 − δ , the loss at initialization satisﬁes
Revised: Furthermore, with probability at least 1 − δ , the loss at initialization satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: The spectral property (8) follows directly from (4).
Revised: The spectral property (8) follows directly from (4).

Original: To prove (9), we essentially need to upper bound the magnitude of the network’s initial output.
Revised: To prove (9), we essentially need to upper bound the magnitude of the network’s initial output.

Original: This turns out to be equivalent to studying the magnitude of the projection of a vector onto a random lowdimensional subspace, which we can bound using standard concentration inequalities.
Revised: This turns out to be equivalent to studying the magnitude of the projection of a vector onto a random lowdimensional subspace, which we can bound using standard concentration inequalities.

Original: The details are given in Appendix A.1.
Revised: The details are given in Appendix A.1.

Original: Now we proceed to prove Theorem 4.1.
Revised: Now we proceed to prove Theorem 4.1.

Original: We deﬁne B = O (cid:16) 1 + log( r/δ )
Revised: We deﬁne B = O (cid:16) 1 + log( r/δ )

Original: d y + (cid:107) W ∗ (cid:107) 2 (cid:17) (cid:107) X (cid:107) 2 F which is the upper bound on (cid:96) (0) from (9).
Revised: d y + (cid:107) W ∗ (cid:107) 2 (cid:17) (cid:107) X (cid:107) 2 F which is the upper bound on (cid:96) (0) from (9).

Original: Conditioned on (9) being satisﬁed, we will use induction on t to prove the following three properties A ( t ) , B ( t ) and C ( t ) for all t = 0 , 1 , . . . :
Revised: Conditioned on (9) being satisﬁed, we will use induction on t to prove the following three properties A ( t ) , B ( t ) and C ( t ) for all t = 0 , 1 , . . . :

Original: • A ( t ) : (cid:96) ( t ) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d
Revised: • A ( t ) : (cid:96) ( t ) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d

Original: y (cid:1) t (cid:96) (0) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d
Revised: y (cid:1) t (cid:96) (0) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d

Original: y (cid:1) t B .
Revised: y (cid:1) t B .

Original: • B ( t ) : σ max ( W j : i ( t )) ≤ 1 .
Revised: • B ( t ) : σ max ( W j : i ( t )) ≤ 1 .

Original: 1 m j −
Revised: 1 m j −

Original: i +12 , σ min ( W j : i ( t )) ≥ 0 .
Revised: i +12 , σ min ( W j : i ( t )) ≥ 0 .

Original: 9 m j −
Revised: 9 m j −

Original: i +1 2 , ∀ 1 ≤ i ≤ j ≤ L, ( i, j ) (cid:54) =
Revised: i +1 2 , ∀ 1 ≤ i ≤ j ≤ L, ( i, j ) (cid:54) =

Original: (1 , L ) .
Revised: (1 , L ) .

Original: • C ( t ) : (cid:107) W
Revised: • C ( t ) : (cid:107) W

Original: i
Revised: i

Original: ( t )
Revised: ( t )

Original: − W i (0) (cid:107) F ≤
Revised: − W i (0) (cid:107) F ≤

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: 2min ( X ) , ∀ 1 ≤ i ≤ L .
Revised: 2min ( X ) , ∀ 1 ≤ i ≤ L .

Original: A (0) and B (0) are true according to Lemma 4.2, and C (0) is trivially true.
Revised: A (0) and B (0) are true according to Lemma 4.2, and C (0) is trivially true.

Original: In order to prove A ( t ) ,
Revised: In order to prove A ( t ) ,

Original: B ( t ) and C ( t ) for all t , we will prove the following claims for all t ≥ 0 :
Revised: B ( t ) and C ( t ) for all t , we will prove the following claims for all t ≥ 0 :

Original: Claim 4.3.
Revised: Claim 4.3.

Original: A (0) , . . .
Revised: A (0) , . . .

Original: , A ( t ) , B (0) , . . .
Revised: , A ( t ) , B (0) , . . .

Original: , B ( t ) =
Revised: , B ( t ) =

Original: ⇒ C
Revised: ⇒ C

Original: ( t + 1) .
Revised: ( t + 1) .

Original: Claim 4.4.
Revised: Claim 4.4.

Original: C ( t ) =
Revised: C ( t ) =

Original: ⇒ B ( t ) .
Revised: ⇒ B ( t ) .

Original: Claim 4.5.
Revised: Claim 4.5.

Original: A ( t ) , B ( t ) =
Revised: A ( t ) , B ( t ) =

Original: ⇒ A ( t + 1) .
Revised: ⇒ A ( t + 1) .

Original: The proofs of these claims are given in Appendix A. Notice that we ﬁnish the proof of Theorem 4.
Revised: The proofs of these claims are given in Appendix A. Notice that we ﬁnish the proof of Theorem 4.

Original: once we prove A ( t ) for all t ≥ 0 .
Revised: once we prove A ( t ) for all t ≥ 0 .

Original:  E XPONENTIAL C URSE OF G AUSSIAN I NITIALIZATION
Revised:  E XPONENTIAL C URSE OF G AUSSIAN I NITIALIZATION

Original: In this section, we show that gradient descent with Gaussian random initialization necessarily suffers from a running time that scales exponentially with the depth of the network, unless the width becomes nearly linear in the depth.
Revised: In this section, we show that gradient descent with Gaussian random initialization necessarily suffers from a running time that scales exponentially with the depth of the network, unless the width becomes nearly linear in the depth.

Original: Since we mostly focus on the dependence of width and running time on depth, we will assume the depth L to be very large.
Revised: Since we mostly focus on the dependence of width and running time on depth, we will assume the depth L to be very large.

Original: Recall that we want to minimize the objective (cid:96) ( W 1 , . . .
Revised: Recall that we want to minimize the objective (cid:96) ( W 1 , . . .

Original: , W L ) =
Revised: , W L ) =

Original: 12 (cid:107) αW L :1 X − Y (cid:107) 2 F by gradient descent.
Revised: 12 (cid:107) αW L :1 X − Y (cid:107) 2 F by gradient descent.

Original: We assume Y = W ∗ X for some W ∗ ∈ R d y × d x , so that the optimal objective value is 0 .
Revised: We assume Y = W ∗ X for some W ∗ ∈ R d y × d x , so that the optimal objective value is 0 .

Original: For convenience, we assume (cid:107) X (cid:107) F = Θ(1) and (cid:107) Y (cid:107) F = Θ(1) .
Revised: For convenience, we assume (cid:107) X (cid:107) F = Θ(1) and (cid:107) Y (cid:107) F = Θ(1) .

Original: Suppose that at layer i ∈
Revised: Suppose that at layer i ∈

Original: [ L ] , every entry of W i (0) is sampled from N (0 , σ 2 i ) , and all weights in the network are independent.
Revised: [ L ] , every entry of W i (0) is sampled from N (0 , σ 2 i ) , and all weights in the network are independent.

Original: We set the scaling factor α such that the initial output of the network does not blow up exponentially (in expectation):
Revised: We set the scaling factor α such that the initial output of the network does not blow up exponentially (in expectation):

Original: [Equation]
Revised: [Equation]

Original: Note that E (cid:104) (cid:107) f ( x ; W 1 (0) , . .
Revised: Note that E (cid:104) (cid:107) f ( x ; W 1 (0) , . .

Original: , W L (0))
Revised: , W L (0))

Original: (cid:107) 2 (cid:105) =
Revised: (cid:107) 2 (cid:105) =

Original: α 2 (cid:81) L i =1 ( d i σ 2 i ) (cid:107) x (cid:107) 2 .
Revised: α 2 (cid:81) L i =1 ( d i σ 2 i ) (cid:107) x (cid:107) 2 .

Original: Thus (10) means
Revised: Thus (10) means

Original: [Equation]
Revised: [Equation]

Original: We also assume that the magnitude of initialization at each layer cannot vanish with depth:
Revised: We also assume that the magnitude of initialization at each layer cannot vanish with depth:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the assumptions (10) and (11) are just sanity checks to rule out the obvious pathological cases – they are easily satisﬁed by all the commonly used initialization schemes in practice.
Revised: Note that the assumptions (10) and (11) are just sanity checks to rule out the obvious pathological cases – they are easily satisﬁed by all the commonly used initialization schemes in practice.

Original: Now we formally state our main theorem in this section.
Revised: Now we formally state our main theorem in this section.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Suppose max { d 0 , d 1 , . . .
Revised: Suppose max { d 0 , d 1 , . . .

Original: , d L } ≤ O ( L 1 − γ ) for some universal constant 0 < γ ≤ 1 .
Revised: , d L } ≤ O ( L 1 − γ ) for some universal constant 0 < γ ≤ 1 .

Original: Then there exists a universal constant c > 0 such that, if gradient descent is run with learning rate η ≤ e cL γ , then with probability at least 0 .
Revised: Then there exists a universal constant c > 0 such that, if gradient descent is run with learning rate η ≤ e cL γ , then with probability at least 0 .

Original: 9 over the random initialization, for the ﬁrst e Ω( L γ ) iterations, the objective value is stuck between 0 .
Revised: 9 over the random initialization, for the ﬁrst e Ω( L γ ) iterations, the objective value is stuck between 0 .

Original: 4 (cid:107) Y (cid:107) 2 F and 0 .
Revised: 4 (cid:107) Y (cid:107) 2 F and 0 .

Original: 6 (cid:107) Y (cid:107) 2 F .
Revised: 6 (cid:107) Y (cid:107) 2 F .

Original: Theorem 5.1 establishes that efﬁcient convergence from Gaussian initialization is impossible for large depth unless the width becomes nearly linear in depth.
Revised: Theorem 5.1 establishes that efﬁcient convergence from Gaussian initialization is impossible for large depth unless the width becomes nearly linear in depth.

Original: This nearly linear dependence is the best we can hope for, since Du & Hu (2019) proved a positive result when the width is larger than linear in depth.
Revised: This nearly linear dependence is the best we can hope for, since Du & Hu (2019) proved a positive result when the width is larger than linear in depth.

Original: Therefore, a phase transition from untrainable to trainable happens at the point when the width and depth has a nearly linear relation.
Revised: Therefore, a phase transition from untrainable to trainable happens at the point when the width and depth has a nearly linear relation.

Original: Furthermore, Theorem 5.1 generalizes the result of
Revised: Furthermore, Theorem 5.1 generalizes the result of

Original: Shamir (2018), which only treats the special case of d 0 = · · · = d L = 1 .
Revised: Shamir (2018), which only treats the special case of d 0 = · · · = d L = 1 .

Original: 5. P ROOF OF T HEOREM 5. 
Revised: 5. P ROOF OF T HEOREM 5.

Original: For convenience, we deﬁne a scaled version of W i : let A i
Revised: For convenience, we deﬁne a scaled version of W i : let A i

Original: =
Revised: =

Original: W i / ( √ d i σ i ) and
Revised: W i / ( √ d i σ i ) and

Original: β = α (cid:81) Li =1 ( √ d i σ i ) .
Revised: β = α (cid:81) Li =1 ( √ d i σ i ) .

Original: Then we know β ≤ L O (1) and αW L :1 =
Revised: Then we know β ≤ L O (1) and αW L :1 =

Original: βA
Revised: βA

Original: L :1 , where A j : i =
Revised: L :1 , where A j : i =

Original: A j · · · A i .
Revised: A j · · · A i .

Original: We ﬁrst give a simple upper bound on (cid:107)
Revised: We ﬁrst give a simple upper bound on (cid:107)

Original: A j : i (0) (cid:107) for all 1 ≤ i ≤ j ≤ L .
Revised: A j : i (0) (cid:107) for all 1 ≤ i ≤ j ≤ L .

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: With probability at least 1 − δ , we have (cid:107) A j : i (0) (cid:107) ≤ O (cid:16) L 3 δ (cid:17) for all 1 ≤ i ≤ j ≤ L .
Revised: With probability at least 1 − δ , we have (cid:107) A j : i (0) (cid:107) ≤ O (cid:16) L 3 δ (cid:17) for all 1 ≤ i ≤ j ≤ L .

Original: The proof of Lemma 5.2 is given in Appendix B.1.
Revised: The proof of Lemma 5.2 is given in Appendix B.1.

Original: It simply uses Markov inequality and union bound.
Revised: It simply uses Markov inequality and union bound.

Original: Furthermore, a key property at initialization is that if j − i is large enough, (cid:107) A j : i (0) (cid:107) will become exponentially small.
Revised: Furthermore, a key property at initialization is that if j − i is large enough, (cid:107) A j : i (0) (cid:107) will become exponentially small.

Original: Lemma 5.3.
Revised: Lemma 5.3.

Original: With probability at least 1 − e − Ω( L γ ) , for all 1 ≤ i ≤ j ≤ L such that j − i ≥ L 10 , we have (cid:107)
Revised: With probability at least 1 − e − Ω( L γ ) , for all 1 ≤ i ≤ j ≤ L such that j − i ≥ L 10 , we have (cid:107)

Original: A j : i (0) (cid:107) ≤ e − Ω( L γ ) .
Revised: A j : i (0) (cid:107) ≤ e − Ω( L γ ) .

Original: Proof.
Revised: Proof.

Original: We ﬁrst consider a ﬁxed pair ( i, j ) such that j −
Revised: We ﬁrst consider a ﬁxed pair ( i, j ) such that j −

Original: i ≥ L 10 .
Revised: i ≥ L 10 .

Original: In order to bound (cid:107) A j : i (0) (cid:107) , we ﬁrst take an arbitrary unit vector v ∈ R d i − 1 and bound (cid:107)
Revised: In order to bound (cid:107) A j : i (0) (cid:107) , we ﬁrst take an arbitrary unit vector v ∈ R d i − 1 and bound (cid:107)

Original: A j : i (0) v (cid:107) .
Revised: A j : i (0) v (cid:107) .

Original: We can write (cid:107)
Revised: We can write (cid:107)

Original: A j : i (0) v (cid:107) 

Revised: A j : i (0) v (cid:107)

Original: = (cid:81)
Revised: = (cid:81)

Original: jk =
Revised: jk =

Original: i Z k , where Z k = (cid:107)
Revised: i Z k , where Z k = (cid:107)

Original: A k : i (0) v (cid:107) 2 (cid:107)
Revised: A k : i (0) v (cid:107) 2 (cid:107)

Original: A k − 1: i (0) v (cid:107) 2 .
Revised: A k − 1: i (0) v (cid:107) 2 .

Original: Note that for any nonzero v (cid:48)
Revised: Note that for any nonzero v (cid:48)

Original: ∈ R
Revised: ∈ R

Original: d k − 1 independent of A k (0) , the distribution of d k · (cid:107)
Revised: d k − 1 independent of A k (0) , the distribution of d k · (cid:107)

Original: A k (0) v (cid:48)
Revised: A k (0) v (cid:48)

Original: (cid:107) 2 (cid:107) v (cid:48)
Revised: (cid:107) 2 (cid:107) v (cid:48)

Original: (cid:107)is χ 2 d k .
Revised: (cid:107)is χ 2 d k .

Original: Therefore, Z i , . . .
Revised: Therefore, Z i , . . .

Original: , Z j are independent, and d k Z k ∼ χ 2 d k ( k
Revised: , Z j are independent, and d k Z k ∼ χ 2 d k ( k

Original: = i, i + 1 , . . .
Revised: = i, i + 1 , . . .

Original: , j ).
Revised: , j ).

Original: Recall the expression for the moments of chi-squared random variables:
Revised: Recall the expression for the moments of chi-squared random variables:

Original: E (cid:2)
Revised: E (cid:2)

Original: Z λk (cid:3) = 2 λ Γ( d k / 2+ λ ) d λk Γ( d k / 2) ( ∀ λ > 0 ).
Revised: Z λk (cid:3) = 2 λ Γ( d k / 2+ λ ) d λk Γ( d k / 2) ( ∀ λ > 0 ).

Original: Taking λ = 12 and using the bound Γ( a + 12 ) Γ( a ) ≤ √ a − 0 .
Revised: Taking λ = 12 and using the bound Γ( a + 12 ) Γ( a ) ≤ √ a − 0 .

Original: 1 ( ∀ a ≥ 12 ) (Qi & Luo, 2012), we get E (cid:2) √ Z k (cid:3) ≤ (cid:113) 2( d k / 2 − 0 .
Revised: 1 ( ∀ a ≥ 12 ) (Qi & Luo, 2012), we get E (cid:2) √ Z k (cid:3) ≤ (cid:113) 2( d k / 2 − 0 .

Original: 1) d k = (cid:113) 1 − 0 .
Revised: 1) d k = (cid:113) 1 − 0 .

Original: 2 d k ≤ 1 − 0 .
Revised: 2 d k ≤ 1 − 0 .

Original: 1 d k .
Revised: 1 d k .

Original: Therefore we have
Revised: Therefore we have

Original: E (cid:34)(cid:114)(cid:89)
Revised: E (cid:34)(cid:114)(cid:89)

Original: j k = i Z k (cid:35) ≤ (cid:89)
Revised: j k = i Z k (cid:35) ≤ (cid:89)

Original: j k = i (cid:18) 1 − 0 .d
Revised: j k = i (cid:18) 1 − 0 .d

Original: k (cid:19) ≤ (cid:18) 1 − 0 .
Revised: k (cid:19) ≤ (cid:18) 1 − 0 .

Original: O ( L 1 − γ ) (cid:19)
Revised: O ( L 1 − γ ) (cid:19)

Original: j − i + ≤ (cid:0) 1 − Ω( L γ − 1 ) (cid:1) L 10 =
Revised: j − i + ≤ (cid:0) 1 − Ω( L γ − 1 ) (cid:1) L 10 =

Original: e − Ω( L γ ) .
Revised: e − Ω( L γ ) .

Original: Choose a sufﬁciently small constant c (cid:48) > 0 .
Revised: Choose a sufﬁciently small constant c (cid:48) > 0 .

Original: By Markov inequality we have
Revised: By Markov inequality we have

Original: Pr (cid:20)(cid:113)(cid:81) jk
Revised: Pr (cid:20)(cid:113)(cid:81) jk

Original: = i
Revised: = i

Original: Z k > e − c (cid:48)
Revised: Z k > e − c (cid:48)

Original: L γ (cid:21) ≤ e c (cid:48)
Revised: L γ (cid:21) ≤ e c (cid:48)

Original: L γ E (cid:20)(cid:113)(cid:81) jk =
Revised: L γ E (cid:20)(cid:113)(cid:81) jk =

Original: i
Revised: i

Original: Z k (cid:21) ≤ e c (cid:48)
Revised: Z k (cid:21) ≤ e c (cid:48)

Original: L γ e − Ω( L γ ) =
Revised: L γ e − Ω( L γ ) =

Original: e − Ω( L γ ) .
Revised: e − Ω( L γ ) .

Original: Therefore we have shown that for any ﬁxed unit vector v ∈ R d i − 1 , with probability at least 1 − e − Ω
Revised: Therefore we have shown that for any ﬁxed unit vector v ∈ R d i − 1 , with probability at least 1 − e − Ω

Original: ( L γ ) we have (cid:107) A j : i (0) v (cid:107) ≤ e − Ω
Revised: ( L γ ) we have (cid:107) A j : i (0) v (cid:107) ≤ e − Ω

Original: ( L γ ) .
Revised: ( L γ ) .

Original: Next, we use this to bound (cid:107)
Revised: Next, we use this to bound (cid:107)

Original: A j : i (0) (cid:107) via an (cid:15) -net argument.
Revised: A j : i (0) (cid:107) via an (cid:15) -net argument.

Original: We partition the index set
Revised: We partition the index set

Original: [ d i − 1 ] into [ d i − 1 ]
Revised: [ d i − 1 ] into [ d i − 1 ]

Original: = S 1 ∪ S 2 ∪ · · · ∪ S q such that | S l | ≤ L γ/ 2 ( ∀ l ∈ [ q ]) and
Revised: = S 1 ∪ S 2 ∪ · · · ∪ S q such that | S l | ≤ L γ/ 2 ( ∀ l ∈ [ q ]) and

Original: q = O
Revised: q = O

Original: ( d i − 1 L γ/ 2 ) .
Revised: ( d i − 1 L γ/ 2 ) .

Original: For each l ∈ [ q ] , let N l be a 12 -net for all the unit vectors in R d i − 1 with support in S l .
Revised: For each l ∈ [ q ] , let N l be a 12 -net for all the unit vectors in R d i − 1 with support in S l .

Original: Note that we can choose N l such that |N l | =
Revised: Note that we can choose N l such that |N l | =

Original: e O ( | S l | )
Revised: e O ( | S l | )

Original: =
Revised: =

Original: e O ( L γ/ 2 ) .
Revised: e O ( L γ/ 2 ) .

Original: Taking a union bound over ∪ ql =1 N l , we know that (cid:107)
Revised: Taking a union bound over ∪ ql =1 N l , we know that (cid:107)

Original: A j : i (0) v (cid:107) ≤ e − Ω
Revised: A j : i (0) v (cid:107) ≤ e − Ω

Original: ( L γ ) (cid:107) v (cid:107) simultaneously for all v ∈ ∪ ql =1 N l with probability at least 1 − (
Revised: ( L γ ) (cid:107) v (cid:107) simultaneously for all v ∈ ∪ ql =1 N l with probability at least 1 − (

Original: [Equation]
Revised: [Equation]

Original: Now, for any u ∈ R d i − 1 , we write it as u = (cid:80) ql =1 a l u l where a l is a scalar and u l is a unit vector supported on S l .
Revised: Now, for any u ∈ R d i − 1 , we write it as u = (cid:80) ql =1 a l u l where a l is a scalar and u l is a unit vector supported on S l .

Original: By the deﬁnition of 12 -net, for each l ∈ [ q ] there exists v l ∈ N l such that (cid:107) v l − u l (cid:107) ≤ 12 .
Revised: By the deﬁnition of 12 -net, for each l ∈ [ q ] there exists v l ∈ N l such that (cid:107) v l − u l (cid:107) ≤ 12 .

Original: We know that (cid:107)
Revised: We know that (cid:107)

Original: A j : i (0) v l (cid:107) ≤ e − Ω
Revised: A j : i (0) v l (cid:107) ≤ e − Ω

Original: ( L γ ) (cid:107) v l (cid:107) for all l ∈ [ q ] .
Revised: ( L γ ) (cid:107) v l (cid:107) for all l ∈ [ q ] .

Original: Let v = (cid:80) ql
Revised: Let v = (cid:80) ql

Original: =1 a l v l .
Revised: =1 a l v l .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:107) v (cid:107) ≤ 32 (cid:107) u (cid:107) .
Revised: which implies (cid:107) v (cid:107) ≤ 32 (cid:107) u (cid:107) .

Original: Therefore we have
Revised: Therefore we have

Original: [Equation]
Revised: [Equation]

Original: The above inequality is valid for any u ∈ R d i − 1 .
Revised: The above inequality is valid for any u ∈ R d i − 1 .

Original: Thus we can take the unit vector u that maximizes (cid:107)
Revised: Thus we can take the unit vector u that maximizes (cid:107)

Original: A j : i (0) u (cid:107) .
Revised: A j : i (0) u (cid:107) .

Original: This gives us (cid:107) A j : i (0) (cid:107) ≤ e − Ω( L γ ) + 12 (cid:107)
Revised: This gives us (cid:107) A j : i (0) (cid:107) ≤ e − Ω( L γ ) + 12 (cid:107)

Original: A j : i (0) (cid:107) , which implies (cid:107)
Revised: A j : i (0) (cid:107) , which implies (cid:107)

Original: A j : i (0) (cid:107) ≤ e − Ω( L γ ) .
Revised: A j : i (0) (cid:107) ≤ e − Ω( L γ ) .

Original: Finally, we take a union bound over all possible ( i, j ) .
Revised: Finally, we take a union bound over all possible ( i, j ) .

Original: The failure probaility is at most L 2 e − Ω( L γ ) =
Revised: The failure probaility is at most L 2 e − Ω( L γ ) =

Original: e − Ω( L γ ) .
Revised: e − Ω( L γ ) .

Original: The following lemma shows that the properties in Lemmas 5.2 and 5.3 are still to some extent preserved after applying small perturbations on all the weight matrices.
Revised: The following lemma shows that the properties in Lemmas 5.2 and 5.3 are still to some extent preserved after applying small perturbations on all the weight matrices.

Original: Lemma 5.4.
Revised: Lemma 5.4.

Original: Suppose that the initial weights satisfy (cid:107) A j : i (0) (cid:107) ≤
Revised: Suppose that the initial weights satisfy (cid:107) A j : i (0) (cid:107) ≤

Original: O ( L 3 ) for all 1 ≤ i ≤ j ≤ L , and (cid:107)
Revised: O ( L 3 ) for all 1 ≤ i ≤ j ≤ L , and (cid:107)

Original: A j : i (0) (cid:107)
Revised: A j : i (0) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: 1 L γ if j − i ≥ L 10 , where c 1 > 0 is a universal constant.
Revised: 1 L γ if j − i ≥ L 10 , where c 1 > 0 is a universal constant.

Original: Then for another set of matrices A 1 , . . .
Revised: Then for another set of matrices A 1 , . . .

Original: , A L satisfying (cid:107) A
Revised: , A L satisfying (cid:107) A

Original: i − A
Revised: i − A

Original: i (0) (cid:107) ≤ e − 0 .
Revised: i (0) (cid:107) ≤ e − 0 .

Original: 6 c 1 L γ for all
Revised: 6 c 1 L γ for all

Original: i ∈
Revised: i ∈

Original: [ L ] , we must have
Revised: [ L ] , we must have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It sufﬁces to show that the difference A j : i − A j : i (0) is tiny.
Revised: It sufﬁces to show that the difference A j : i − A j : i (0) is tiny.

Original: Let ∆ i
Revised: Let ∆ i

Original: =
Revised: =

Original: A i − A
Revised: A i − A

Original: i (0) .
Revised: i (0) .

Original: We have A j : i =
Revised: We have A j : i =

Original: ( A j (0) + ∆ j ) · · · ( A i +1 (0)
Revised: ( A j (0) + ∆ j ) · · · ( A i +1 (0)

Original: + ∆ i +1 )
Revised: + ∆ i +1 )

Original: ( A i (0) + ∆ i ) .
Revised: ( A i (0) + ∆ i ) .

Original: Expanding this product, except for the one term corresponding to A j : i (0) , every other term has the form A j :( k s +1)
Revised: Expanding this product, except for the one term corresponding to A j : i (0) , every other term has the form A j :( k s +1)

Original: (0) · ∆ k s · A ( k s − 1):( k s − 1 +1) (0) ·
Revised: (0) · ∆ k s · A ( k s − 1):( k s − 1 +1) (0) ·

Original: ∆ k s − 1 · · · ∆ k 1 · A ( k 1 − 1): i (0) , where i ≤ k 1 < · · · < k s ≤ j .
Revised: ∆ k s − 1 · · · ∆ k 1 · A ( k 1 − 1): i (0) , where i ≤ k 1 < · · · < k s ≤ j .

Original: By assumption, each ∆ k has spectral norm e − 
Revised: By assumption, each ∆ k has spectral norm e − 0 .

Original: [Table]

Original: 6 c 1 L γ , and each A j
Revised: 6 c 1 L γ , and each A j

Original: [Table]
Revised: [Table]

Original: i (cid:48) (0) has spectral norm
Revised: i (cid:48) (0) has spectral norm

Original: O ( L 3 ) , so we have (cid:13)(cid:13)
Revised: O ( L 3 ) , so we have (cid:13)(cid:13)

Original: A j :( k s +1) (0) ·
Revised: A j :( k s +1) (0) ·

Original: ∆ k s · A ( k s − 1):( k s − 1 +1) (0) · ∆
Revised: ∆ k s · A ( k s − 1):( k s − 1 +1) (0) · ∆

Original: [Table]
Revised: [Table]

Original:  · · · ∆
Revised: s − 1 · · · ∆

Original: [Table]
Revised: [Table]

Revised: A (

Original: [Table]
Revised: [Table]

Original: 0) (cid:13)(cid:13) ≤ (cid:0) e
Revised: (0) (cid:13)(cid:13) ≤ (cid:0) e

Original: [Table]
Revised: [Table]

Original: s (cid:0) O
Revised: s (cid:0) O

Original: ( L 3 ) (cid:1) s +1 .
Revised: ( L 3 ) (cid:1) s +1 .

Original: Therefore we have (cid:107)
Revised: Therefore we have (cid:107)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: s
Revised: s

Original: [Table]
Revised: [Table]

Original: ≤
Revised: ≤

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:107) A j : i (cid:107) ≤
Revised: which implies (cid:107) A j : i (cid:107) ≤

Original: O ( L 3 ) for all 1 ≤ i ≤ j ≤ L .
Revised: O ( L 3 ) for all 1 ≤ i ≤ j ≤ L .

Original: The proof of the second part of the lemma is postponed to Appendix B.2.
Revised: The proof of the second part of the lemma is postponed to Appendix B.2.

Original: As a consequence of Lemma 5.4, we can control the objective value and the gradient at any point sufﬁciently close to the random initialization.
Revised: As a consequence of Lemma 5.4, we can control the objective value and the gradient at any point sufﬁciently close to the random initialization.

Original: Lemma 5.5.
Revised: Lemma 5.5.

Original: For a set of weight matrices W 1 , . . .
Revised: For a set of weight matrices W 1 , . . .

Original: , W L with A i
Revised: , W L with A i

Original: =
Revised: =

Original: W i / ( √ d i σ i ) that satisfy (12) , the objective and the gradient satisfy
Revised: W i / ( √ d i σ i ) that satisfy (12) , the objective and the gradient satisfy

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma 5.5 is given in Appendix B.3.
Revised: The proof of Lemma 5.5 is given in Appendix B.3.

Original: Finally, we can ﬁnish the proof of Theorem 5.1 using the above lemmas.
Revised: Finally, we can ﬁnish the proof of Theorem 5.1 using the above lemmas.

Original: Proof of Theorem 5.1.
Revised: Proof of Theorem 5.1.

Original: From Lemmas 5.2 and 5.3, we know that with probability at least 0 .
Revised: From Lemmas 5.2 and 5.3, we know that with probability at least 0 .

Original: 9 , we have (i) (cid:107)
Revised: 9 , we have (i) (cid:107)

Original: A j : i (0) (cid:107) ≤ O ( L 3 ) for all 1 ≤ i ≤ j ≤ L , and (ii) (cid:107)
Revised: A j : i (0) (cid:107) ≤ O ( L 3 ) for all 1 ≤ i ≤ j ≤ L , and (ii) (cid:107)

Original: A j : i (0) (cid:107) ≤ e − c 1 L γ if ( i, j ) further satisﬁes j − i ≥ L 10 .
Revised: A j : i (0) (cid:107) ≤ e − c 1 L γ if ( i, j ) further satisﬁes j − i ≥ L 10 .

Original: Here c 1 > 0 is a universal constant.
Revised: Here c 1 > 0 is a universal constant.

Original: From now on we are conditioned on these properties being satisﬁed.
Revised: From now on we are conditioned on these properties being satisﬁed.

Original: We suppose that the learning rate η is at most e 0 .
Revised: We suppose that the learning rate η is at most e 0 .

Original: 2 c 1 L γ .
Revised: 2 c 1 L γ .

Original: We say that a set of weight matrices W 1 , . . .
Revised: We say that a set of weight matrices W 1 , . . .

Original: , W L are in the “initial neighborhood” if (cid:107)
Revised: , W L are in the “initial neighborhood” if (cid:107)

Original: A i − A
Revised: A i − A

Original: i (0) (cid:107) ≤ e − 0 .
Revised: i (0) (cid:107) ≤ e − 0 .

Original: 6 c 1 L γ for all
Revised: 6 c 1 L γ for all

Original: i ∈
Revised: i ∈

Original: [ L ] .
Revised: [ L ] .

Original: From Lemmas 5.4 and 5.5 we know that in the “initial neighborhood” the objective value is always between 0 .
Revised: From Lemmas 5.4 and 5.5 we know that in the “initial neighborhood” the objective value is always between 0 .

Original: 4 (cid:107) Y (cid:107) 2 F and 0 .
Revised: 4 (cid:107) Y (cid:107) 2 F and 0 .

Original: 6 (cid:107) Y (cid:107) 2 F .
Revised: 6 (cid:107) Y (cid:107) 2 F .

Original: Therefore we have to escape the “initial neighborhood” in order to get the objective value out of this interval.
Revised: Therefore we have to escape the “initial neighborhood” in order to get the objective value out of this interval.

Original: Now we calculate how many iterations are necessary to escape the “initial neighborhood.”
Revised: Now we calculate how many iterations are necessary to escape the “initial neighborhood.”

Original: According to Lemma 5.5, inside the “initial neighborhood” each W i can move at most η ( √ d i σ i ) − 1 e − 0 .
Revised: According to Lemma 5.5, inside the “initial neighborhood” each W i can move at most η ( √ d i σ i ) − 1 e − 0 .

Original: 9 c 1 L γ in one iteration by deﬁnition of the gradient descent algorithm.
Revised: 9 c 1 L γ in one iteration by deﬁnition of the gradient descent algorithm.

Original: In order to leave the “initial neighborhood,” some W i must satisfy (cid:107) W i − W i (0) (cid:107) =[Equation]
Revised: In order to leave the “initial neighborhood,” some W i must satisfy (cid:107) W i − W i (0) (cid:107) =

Original:  d [Equation] (cid:107) 
Revised: √ d i σ i (cid:107) A

Original: [Equation]

Original: i − 
Revised: i − A

Revised: i (0) (cid:107) > √ d i σ

Original: [Equation]
Revised: [Equation]

Original: 0) (cid:107) > √ d
Revised: e

Original: [Equation]
Revised: [Equation]

Original:  L γ .
Revised: 6 c 1 L γ .

Original: In order to move this amount, the number of iterations has to be at least√
Revised: In order to move this amount, the number of iterations has to be at least

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.

Original:  C ONCLUSION
Revised:  E XPERIMENTS

Revised: In this section, we provide empirical evidence to support the results in Sections 4 and 5.

Revised: To study how depth and width affect convergence speed of gradient descent under orthogonal and Gaussian initialization schemes, we train a family of linear networks with their widths ranging from 10 to

Revised: [Figure]

Revised: [Figure]

Revised: 1000 and depths from 1 to 700, on a ﬁxed synthetic dataset ( X, Y ) .

Revised: 4 Each network is trained using gradient descent staring from both Gaussian and orthogonal initializations.

Revised: In Figure 1, We lay out the logarithm of the relative training loss (cid:96) ( t ) (cid:96) (0) , using heap-maps, at steps t = 1258 and t = 10000 .

Revised: In each heat-map, each point represents the relative training loss of one experiment; the darker the color, the smaller the loss.

Revised: Figure 1 clearly demonstrates a sharp transition from untrainable to trainable (i.e., from red to black) when we increase the width of the network:

Revised: • for Gaussian initialization, this transition occurs across a contour characterized by a linear relation between width and depth; • for orthogonal initialization, the transition occurs at a width that is approximately independent of the depth.

Revised: These observations excellently verify our theory developed in Sections 4 and 5.

Revised: To have a closer look into the training dynamics, we also plot “relative loss v.s. training time” for a variety of depth-width conﬁgurations.

Revised: See Figure 2.

Revised: There again we can clearly see that orthogonal initialization enables fast training at small width (independent of depth), and that the required width for Gaussian initialization depends on depth.

Revised:  C ONCLUSION

Original: In this work, we studied the effect of the initialization parameter values of deep linear neural networks on the convergence time of gradient descent.
Revised: In this work, we studied the effect of the initialization parameter values of deep linear neural networks on the convergence time of gradient descent.

Original: We found that when the initial weights are iid
Revised: We found that when the initial weights are iid

Original: Gaussian, the convergence time grows exponentially in the depth unless the width is at least as large  as the depth.
Revised: Gaussian, the convergence time grows exponentially in the depth unless the width is at least as large [Figure] as the depth.

Original: In contrast, when the initial weight matrices are drawn from the orthogonal group, the width needed to guarantee efﬁcient convergence is in fact independent of the depth.
Revised: In contrast, when the initial weight matrices are drawn from the orthogonal group, the width needed to guarantee efﬁcient convergence is in fact independent of the depth.

Original: These results establish for the ﬁrst time a concrete proof that orthogonal initialization is superior to Gaussian initialization in terms of convergence time. 
Revised: These results establish for the ﬁrst time a concrete proof that orthogonal initialization is superior to Gaussian initialization in terms of convergence time. [Figure]

Original: A P ROOFS FOR S ECTIONA. P ROOF OF L EMMA 4. 
Revised: A P ROOFS FOR S ECTIONA. P ROOF OF L EMMA 4.

Original: Proof of Lemma 4.2.
Revised: Proof of Lemma 4.2.

Original: We only need to prove (9).
Revised: We only need to prove (9).

Original: We ﬁrst upper bound the magnitude of the network’s initial output on any given input x ∈ R d x .
Revised: We ﬁrst upper bound the magnitude of the network’s initial output on any given input x ∈ R d x .

Original: Let z = 1 √
Revised: Let z = 1 √

Original: m L − 1 W L − 1:1 (0) · x ∈ R m .
Revised: m L − 1 W L − 1:1 (0) · x ∈ R m .

Original: Then we have (cid:107) z (cid:107) =
Revised: Then we have (cid:107) z (cid:107) =

Original: (cid:107) x (cid:107) , and f ( x ; W 1 (0) , . . .
Revised: (cid:107) x (cid:107) , and f ( x ; W 1 (0) , . . .

Original: , W L (0))
Revised: , W L (0))

Original: = 1 √ d y W L (0) · z = (cid:113) md y · 1 √ m W L (0) · z .
Revised: = 1 √ d y W L (0) · z = (cid:113) md y · 1 √ m W L (0) · z .

Original: Note that 1 √ m W L (0) ·
Revised: Note that 1 √ m W L (0) ·

Original: z is the (signed) projection of z onto a random subspace in R m of dimension d y .
Revised: z is the (signed) projection of z onto a random subspace in R m of dimension d y .

Original: Therefore (cid:13)(cid:13)(cid:13) 1 √ m W L (0) · z (cid:13)(cid:13)(cid:13) 2 / (cid:107) z (cid:107) 2 has the same distribution as g 21 + ··· + g 2 dy g 21 + ··· + g 2 m , where g 1 , . .
Revised: Therefore (cid:13)(cid:13)(cid:13) 1 √ m W L (0) · z (cid:13)(cid:13)(cid:13) 2 / (cid:107) z (cid:107) 2 has the same distribution as g 21 + ··· + g 2 dy g 21 + ··· + g 2 m , where g 1 , . .

Original: , g m are i.i.d.
Revised: , g m are i.i.d.

Original: samples from N (0 , 1) .
Revised: samples from N (0 , 1) .

Original: By the standard tail bounds for chi-squared distributions we have
Revised: By the standard tail bounds for chi-squared distributions we have

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Let δ (cid:48)
Revised: Let δ (cid:48)

Original: =
Revised: =

Original: δ 2 r .
Revised: δ 2 r .

Original: Note that m > C · log( r/δ ) .
Revised: Note that m > C · log( r/δ ) .

Original: We know that with probability at least 1 − δr we have (cid:13)(cid:13)(cid:13)(cid:13)
Revised: We know that with probability at least 1 − δr we have (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Finally, taking a union bound
Revised: Finally, taking a union bound

Original: [Equation]
Revised: [Equation]

Original: we know that with probability at least 1 − δ , the inequality (13) holds for every x ∈ { x 1 , . . .
Revised: we know that with probability at least 1 − δ , the inequality (13) holds for every x ∈ { x 1 , . . .

Original: , x r }
Revised: , x r }

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: ≤ O
Revised: ≤ O

Original: [Equation]
Revised: [Equation]

Original: k = (cid:107) x k (cid:107) 2 + r (cid:88) k
Revised: k = (cid:107) x k (cid:107) 2 + r (cid:88) k

Original: = (cid:107) y k (cid:107)
Revised: = (cid:107) y k (cid:107)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: O (cid:18) 1 + log( r/δ )
Revised: O (cid:18) 1 + log( r/δ )

Original: d y (cid:19) (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107) 2 F ≤ O (cid:18)
Revised: d y (cid:19) (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107) 2 F ≤ O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: + log
Revised: + log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF C LAIM 4. 
Revised: A. P ROOF OF C LAIM 4.

Original: Proof of Claim 4.3.
Revised: Proof of Claim 4.3.

Original: Let γ = 

Revised: Let γ =

Original: Lσ 2min ( X ) /d
Revised: Lσ 2min ( X ) /d

Original: y .
Revised: y .

Original: From A (0) , . . .
Revised: From A (0) , . . .

Original: , A ( t ) we have (cid:96) ( s ) ≤ (1 − ηγ )
Revised: , A ( t ) we have (cid:96) ( s ) ≤ (1 − ηγ )

Original: s B for all 0 ≤ s ≤ t .
Revised: s B for all 0 ≤ s ≤ t .

Original: The gradient of the objective function (2) is ∂(cid:96)∂W i = αW (cid:62) L : i +1 ( U − Y )
Revised: The gradient of the objective function (2) is ∂(cid:96)∂W i = αW (cid:62) L : i +1 ( U − Y )

Original: ( W i − 1:1 X ) (cid:62) .
Revised: ( W i − 1:1 X ) (cid:62) .

Original: Thus we can bound the gradient norm as follows for all 0 ≤ s ≤ t and all i ∈
Revised: Thus we can bound the gradient norm as follows for all 0 ≤ s ≤ t and all i ∈

Original: [ L ] :
Revised: [ L ] :

Original: [Equation]
Revised: [Equation]

Original: where we have used B ( s ) .
Revised: where we have used B ( s ) .

Original: Then for all i ∈
Revised: Then for all i ∈

Original: [ L ] we have:
Revised: [ L ] we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This proves C ( t + 1) .
Revised: This proves C ( t + 1) .

Original: A. P ROOF OF C LAIM 4. 
Revised: A. P ROOF OF C LAIM 4.

Original: Proof of Claim 4.4.
Revised: Proof of Claim 4.4.

Original: Let R =
Revised: Let R =

Original: [Equation]
Revised: [Equation]

Original: Lσ 2min ( X ) and ∆ i =
Revised: Lσ 2min ( X ) and ∆ i =

Original: W i ( t ) − W i (0) ( i ∈
Revised: W i ( t ) − W i (0) ( i ∈

Original: [ L ]) .
Revised: [ L ]) .

Original: Then C ( t ) means (cid:107) ∆ i (cid:107) F ≤ R ( ∀ i ∈
Revised: Then C ( t ) means (cid:107) ∆ i (cid:107) F ≤ R ( ∀ i ∈

Original: [ L ]) .
Revised: [ L ]) .

Original: For 1 ≤ i ≤ j ≤ L , we have
Revised: For 1 ≤ i ≤ j ≤ L , we have

Original: [Equation]
Revised: [Equation]

Original: Expanding this product, each term except W j : i (0) has the form:
Revised: Expanding this product, each term except W j : i (0) has the form:

Original: [Equation]
Revised: [Equation]

Original: ) where i ≤ k 1 < · · · < k s ≤ j are locations where terms like ∆ k l are taken out.
Revised: ) where i ≤ k 1 < · · · < k s ≤ j are locations where terms like ∆ k l are taken out.

Original: Note that every factor in (15) of the form W j (cid:48) : i (cid:48) (0) satisﬁes
Revised: Note that every factor in (15) of the form W j (cid:48) : i (cid:48) (0) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: according to (8).
Revised: according to (8).

Original: Thus, we can bound the sum of all terms of the form (15) as
Revised: Thus, we can bound the sum of all terms of the form (15) as

Original: [Equation]
Revised: [Equation]

Original: Here the last step uses m > C ( LR ) 2 which is implied by (5).
Revised: Here the last step uses m > C ( LR ) 2 which is implied by (5).

Original: Combined with (8), this proves
Revised: Combined with (8), this proves

Original: B ( t ) .
Revised: B ( t ) .

Original: A. P ROOF OF C LAIM 4. 
Revised: A. P ROOF OF C LAIM 4.

Original: Proof of Claim 4.5.
Revised: Proof of Claim 4.5.

Original: Recall that we have the dynamics (6) for U ( t ) .
Revised: Recall that we have the dynamics (6) for U ( t ) .

Original: In order to establish convergence from (6) we need to prove upper and lower bounds on the eigenvalues of P ( t ) , as well as show that the high-order term E ( t ) is small.
Revised: In order to establish convergence from (6) we need to prove upper and lower bounds on the eigenvalues of P ( t ) , as well as show that the high-order term E ( t ) is small.

Original: We will prove these using B ( t ) .
Revised: We will prove these using B ( t ) .

Original: Using the deﬁnition (7) and property B ( t ) , we have
Revised: Using the deﬁnition (7) and property B ( t ) , we have

Original: [Equation]
Revised: [Equation]

Original: In the lower bound above, we make use of the following relation on dimensions: m ≥ d x ≥ r , which enables the inequality λ min (cid:16) ( W i − 1:1 ( t ) X ) (cid:62) ( W i − 1:1 ( t ) X ) (cid:17) =
Revised: In the lower bound above, we make use of the following relation on dimensions: m ≥ d x ≥ r , which enables the inequality λ min (cid:16) ( W i − 1:1 ( t ) X ) (cid:62) ( W i − 1:1 ( t ) X ) (cid:17) =

Original: σ 2 min ( W i − 1:1 ( t ) X ) ≥ σ 2min ( W i − 1:1 ( t )) · σ 2min ( X ) .
Revised: σ 2 min ( W i − 1:1 ( t ) X ) ≥ σ 2min ( W i − 1:1 ( t )) · σ 2min ( X ) .

Original: Next, we will prove the following bound on the high-order term E ( t ) :
Revised: Next, we will prove the following bound on the high-order term E ( t ) :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Same as (14), we have (cid:13)(cid:13)(cid:13) ∂(cid:96)∂W i ( t ) (cid:13)(cid:13)(cid:13) F ≤
Revised: Same as (14), we have (cid:13)(cid:13)(cid:13) ∂(cid:96)∂W i ( t ) (cid:13)(cid:13)(cid:13) F ≤

Original: [Table]
Revised: [Table]

Original: √ (cid:96) ( t ) (cid:107) X (cid:107) √ d y ( ∀ i ∈
Revised: √ (cid:96) ( t ) (cid:107) X (cid:107) √ d y ( ∀ i ∈

Original: [ L ] ).
Revised: [ L ] ).

Original: Then we have
Revised: Then we have

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: m d y
Revised: m d y

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: From
Revised: From

Original: [Table]
Revised: [Table]

Original: , we have
Revised: , we have

Original: [Table]
Revised: [Table]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: t ) (cid:107) X (cid:107)
Revised: t ) (cid:107) X (cid:107)

Original: [Equation]
Revised: [Equation]

Original: md
Revised: md

Original: [Equation]
Revised: [Equation]

Original: √ d y · (cid:96)
Revised: √ d y · (cid:96)

Original: [Table]
Revised: [Table]

Original: t ) √ m
Revised: t ) √ m

Original: [Table]
Revised: [Table]

Original: X
Revised: X

Original: [Table]
Revised: [Table]

Original: Note that m > C · d y B
Revised: Note that m > C · d y B

Original: [Table]
Revised: [Table]

Original: X
Revised: X

Original: [Table]
Revised: [Table]

Original: 2 ≥ C · d y (cid:96)
Revised: 2 ≥ C · d y (cid:96)

Original: [Table]
Revised: [Table]

Original: t ) (cid:107) X (cid:107) 2 .
Revised: t ) (cid:107) X (cid:107) 2 .

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: md
Revised: md

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to show that the above bound is at most 16 ηλ min ( P t ) (cid:107) U ( t ) − Y (cid:107) F = 16 ηλ min ( P t ) (cid:112) 2 (cid:96) ( t ) .
Revised: It sufﬁces to show that the above bound is at most 16 ηλ min ( P t ) (cid:107) U ( t ) − Y (cid:107) F = 16 ηλ min ( P t ) (cid:112) 2 (cid:96) ( t ) .

Original: Since λ min ( P t ) ≥ 35 Lσ 2min ( X ) /d
Revised: Since λ min ( P t ) ≥ 35 Lσ 2min ( X ) /d

Original: y , it sufﬁces to have
Revised: y , it sufﬁces to have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , which is true since m > C · d y B (cid:107) X (cid:107) 2 σ 4min ( X ) ≥ C · d y (cid:96) ( t ) (cid:107) X (cid:107) 2 σ 4min ( X ) .
Revised: , which is true since m > C · d y B (cid:107) X (cid:107) 2 σ 4min ( X ) ≥ C · d y (cid:96) ( t ) (cid:107) X (cid:107) 2 σ 4min ( X ) .

Original: Finally, from (6) and η ≤ d y 2 L
Revised: Finally, from (6) and η ≤ d y 2 L

Original: [Equation]
Revised: [Equation]

Original: X (cid:107) 2 ≤
Revised: X (cid:107) 2 ≤

Original: [Equation]
Revised: [Equation]

Original: max ( P t ) we have
Revised: max ( P t ) we have

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:107) U ( t ) − Y (cid:107) F .
Revised: (cid:107) U ( t ) − Y (cid:107) F .

Original: Therefore (cid:96) ( t +1) ≤ (cid:0) 1 − 12 ηLσ 2min
Revised: Therefore (cid:96) ( t +1) ≤ (cid:0) 1 − 12 ηLσ 2min

Original: ( X ) /d
Revised: ( X ) /d

Original: y (cid:1) 2 (cid:96) ( t ) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d
Revised: y (cid:1) 2 (cid:96) ( t ) ≤ (cid:0) 1 − 12 ηLσ 2min ( X ) /d

Original: y (cid:1) (cid:96) ( t ) .
Revised: y (cid:1) (cid:96) ( t ) .

Original: Combined with
Revised: Combined with

Original: A ( t ) , this proves A ( t + 1) .
Revised: A ( t ) , this proves A ( t + 1) .

Original: B P ROOFS FOR S ECTIONB. P ROOF OF L EMMA 5. 
Revised: B P ROOFS FOR S ECTIONB. P ROOF OF L EMMA 5.

Original: Proof of Lemma 5.2.
Revised: Proof of Lemma 5.2.

Original: Notice that for any 1 ≤ i ≤ j ≤ L we have E (cid:104) (cid:107)
Revised: Notice that for any 1 ≤ i ≤ j ≤ L we have E (cid:104) (cid:107)

Original: A j : i (0) (cid:107) 2 F (cid:105) =
Revised: A j : i (0) (cid:107) 2 F (cid:105) =

Original: d i − 1 .
Revised: d i − 1 .

Original: Then by Markov inequality we have Pr (cid:104) (cid:107)
Revised: Then by Markov inequality we have Pr (cid:104) (cid:107)

Original: A j : i (0) (cid:107) 2 F ≥
Revised: A j : i (0) (cid:107) 2 F ≥

Original: d
Revised: d

Original: i − 1 δ/L 2 (cid:105) ≤ δ/L 2 .
Revised: i − 1 δ/L 2 (cid:105) ≤ δ/L 2 .

Original: Taking a union bound, we know that with probability at least 1 − δ , for all 1 ≤ i ≤ j ≤ L simultaneously we have (cid:107) A j : i (0) (cid:107) ≤ (cid:107)
Revised: Taking a union bound, we know that with probability at least 1 − δ , for all 1 ≤ i ≤ j ≤ L simultaneously we have (cid:107) A j : i (0) (cid:107) ≤ (cid:107)

Original: A j : i (0) (cid:107) F ≤ d i − 1 δ/L 2 ≤ O ( L 3 /δ )
Revised: A j : i (0) (cid:107) F ≤ d i − 1 δ/L 2 ≤ O ( L 3 /δ )

Original: (note that d i − 1 ≤
Revised: (note that d i − 1 ≤

Original: O ( L 1 − γ )
Revised: O ( L 1 − γ )

Original: =
Revised: =

Original: O ( L ) ).
Revised: O ( L ) ).

Original: B. P ROOF OF L EMMA 5. 
Revised: B. P ROOF OF L EMMA 5.

Original: Proof of Lemma 5.4 (continued).
Revised: Proof of Lemma 5.4 (continued).

Original: For the second part of the lemma ( j − i ≥ L 4 ), we need to bound the terms of the form A j : k +1 (0) ·
Revised: For the second part of the lemma ( j − i ≥ L 4 ), we need to bound the terms of the form A j : k +1 (0) ·

Original: ∆ k · A k − 1: i (0) more carefully.
Revised: ∆ k · A k − 1: i (0) more carefully.

Original: In fact, if j − i ≥ L 4 , then max { j − k − 1 , k − 1 − i } ≥ L 10 , which by assumption means either A j : k +1 (0) or A k − 1: i (0) has spectral norm bounded by e − c 1 L γ .
Revised: In fact, if j − i ≥ L 4 , then max { j − k − 1 , k − 1 − i } ≥ L 10 , which by assumption means either A j : k +1 (0) or A k − 1: i (0) has spectral norm bounded by e − c 1 L γ .

Original: This implies (cid:107) A j : k +1 (0) ·
Revised: This implies (cid:107) A j : k +1 (0) ·

Original: ∆ k · A k − 1: i (0) (cid:107) ≤ e − c 1 L γ e − 0 .
Revised: ∆ k · A k − 1: i (0) (cid:107) ≤ e − c 1 L γ e − 0 .

Original: 6 c 1 L γ ·
Revised: 6 c 1 L γ ·

Original: O ( L 3 )
Revised: O ( L 3 )

Original: =
Revised: =

Original: e − 1 .
Revised: e − 1 .

Original: 6 c 1 L γ · O ( L 3 ) .
Revised: 6 c 1 L γ · O ( L 3 ) .

Original: Therefore we have (cid:107)
Revised: Therefore we have (cid:107)

Original: A j : i − A j : i (0) (cid:107) ≤ ( j − i + 1) e − 1 .
Revised: A j : i − A j : i (0) (cid:107) ≤ ( j − i + 1) e − 1 .

Original: 6 c 1 L γ · O ( L 3 ) + j − i +1 (cid:88) s = (cid:18)
Revised: 6 c 1 L γ · O ( L 3 ) + j − i +1 (cid:88) s = (cid:18)

Original: j − i +s (cid:19) (cid:16) e − 0 .
Revised: j − i +s (cid:19) (cid:16) e − 0 .

Original: 6 c 1 L γ (cid:17) s (cid:0) O
Revised: 6 c 1 L γ (cid:17) s (cid:0) O

Original: ( L 3 ) (cid:1) s +
Revised: ( L 3 ) (cid:1) s +

Original: ≤ e − c 1 L γ + ∞ (cid:88) s =
Revised: ≤ e − c 1 L γ + ∞ (cid:88) s =

Original: L s (cid:16) e − 0 .
Revised: L s (cid:16) e − 0 .

Original: 6 c 1 L γ (cid:17) s (cid:0) O
Revised: 6 c 1 L γ (cid:17) s (cid:0) O

Original: ( L 3 ) (cid:1) s +1 ≤ e − c 1 L γ + ∞ (cid:88)
Revised: ( L 3 ) (cid:1) s +1 ≤ e − c 1 L γ + ∞ (cid:88)

Original: s = (cid:16)
Revised: s = (cid:16)

Original: e −
Revised: e −

Original: [Table]
Revised: [Table]

Original: 5 c 1 L
Revised: 5 c 1 L

Original: [Table]
Revised: [Table]

Original: (cid:17) s =
Revised: (cid:17) s =

Original: O (cid:16) e − c 1 L γ (cid:17) .
Revised: O (cid:16) e − c 1 L γ (cid:17) .

Original: This implies (cid:107)
Revised: This implies (cid:107)

Original: A j : i (cid:107) ≤ O (cid:0)
Revised: A j : i (cid:107) ≤ O (cid:0)

Original: e − c 1 L γ (cid:1) .
Revised: e − c 1 L γ (cid:1) .

Original: B. P ROOF OF L EMMA 5. 
Revised: B. P ROOF OF L EMMA 5.

Original: Proof of Lemma 5.5.
Revised: Proof of Lemma 5.5.

Original: We can bound the network’s output as
Revised: We can bound the network’s output as

Original: [Equation]
Revised: [Equation]

Original: Thus the objective value (cid:96) ( W 1 , . . .
Revised: Thus the objective value (cid:96) ( W 1 , . . .

Original: , W L ) =
Revised: , W L ) =

Original: 12 (cid:107) αW L :1 (0) X − Y (cid:107) 2 F must be extremely close to 12 (cid:107) Y (cid:107) 2 F for large L , so 0 .
Revised: 12 (cid:107) αW L :1 (0) X − Y (cid:107) 2 F must be extremely close to 12 (cid:107) Y (cid:107) 2 F for large L , so 0 .

Original: 4 (cid:107) Y (cid:107) 2 F < (cid:96) ( W 1 , . .
Revised: 4 (cid:107) Y (cid:107) 2 F < (cid:96) ( W 1 , . .

Original: , W L ) < 0 .
Revised: , W L ) < 0 .

Original: 6 (cid:107) Y (cid:107) 2 F .
Revised: 6 (cid:107) Y (cid:107) 2 F .

Original: As for the gradient, for any i ∈ [ L ] we have
Revised: As for the gradient, for any i ∈ [ L ] we have

Original: [Equation]
Revised: [Equation]

Original: Using (12)
Revised: Using (12)

Original: [Equation]
Revised: [Equation]

Original: and noting that either L − i − 

Revised: and noting that either L − i −

Original: or i − 1 is greater than L
Revised: or i − 1 is greater than L

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]